Files
gstack/docs/cso-scanners.md
T
Garry Tan 857466ff8b v1.91.25.0 feat(cso): private two-architecture release scoring + self-checking runtime release gates (#3036)
* v1.91.23.0 feat(cso): private two-architecture release scoring and self-checking runtime release gates

* Keep the Linux-only native-evidence and requalification tests out of the Windows lane

* Skip the two Linux-release-only checks on Windows in-file instead of growing the shard list

* fix(cso): opaque producer inputs and receipts; requalification triggers independent of cwd

Producer inputs are {schemaVersion: 2, cellRef, skill, source, execution} and
receipts bind cellRef + inputHash; collection rebuilds the cell from the
matrix, so a cell VM never learns its case or variant. requalificationTriggers
resolves Bun's cwd-relative metafile inputs against the repository root.

* test(evals): bound the review judge's thinking; guarded browser cleanup is not a probe

review/SKILL.md workflow: the ~64k-token bundle let default-effort thinking
spend all 8192 output tokens without an answer in CI. Medium effort with a
64k streaming cap kept 24/24 measured samples under 5k tokens and 63 s with
the same gating scores (as ship already does).

qa-only-no-fix: the fixture prompt routes browser cleanup through the deadline
guard, but the checkpoint detector treated every guarded run as a probe that
needs a checkpoint. A child that only closes the tab with the supplied browse
binary, after the last probe, is cleanup; anything else stays a probe. The
captured CI transcript (run 37252022510) now passes and new scenarios pin
the negative controls.

* test(evals): keep the review judge at default effort; only lift its output cap

Truncation was the defect: thinking and the answer share max_tokens, and the
~64k-token review bundle once spent all 8192. The registration keeps default
effort and response format and gets ship's 64k streaming cap; the guard test
pins exactly that.
2026-10-04 21:47:21 -07:00

8.2 KiB
Raw Blame History

Running CSO scanners

gstack-cso scan RUN SCANNER [REQUEST.json] collects candidate evidence through the trusted helper when a matching qualified scanner catalog profile exists. Recognized IDs are gitleaks, osv, semgrep, zizmor, trivy, and schemathesis. Without a qualified profile, the command records a specific prerequisite; static assessment and importing existing CodeQL or other SARIF with gstack-cso import-sarif RUN REPORT.sarif remain available. The returned immutable artifact can be read later with gstack-cso scanner-outcome RUN ARTIFACT_ID; callers never need to open private state files directly.

The source-controlled catalog in scanner-images/catalog.json is empty until every release gate produces real evidence. Without a reviewed profile, the command records a specific not_assessed prerequisite. Installing a scanner on the host does not enable execution. Each profile binds a Linux platform, immutable image digest, fixed entrypoint, absolute scanner executable, version-probe hash, required adapter capabilities, helper ABI, and the exact isolation-policy hash. The helper validates that catalog when it starts and has no executable fallback. Setup preloads the current platform's qualified scanner digests through the trusted installation helper using anonymous public-registry access. A failed preload or a bounded per-image/setup deadline is a nonfatal prerequisite and never causes an audit to pull. The default is 30 seconds per declared image; GSTACK_CSO_IMAGE_PULL_TIMEOUT_SECONDS=120 selects a longer allowance (accepted range: 5–300 seconds) while the complete preload remains capped at one hour. Doctor is read-only and reports each scanner unavailable unless its exact catalog digest is already present locally.

All scanner processes use Docker containment with a watchdog, shared admission limits, read-only source, bounded output, and an isolated network namespace. Static scanners have no network access, including when the audit omits --offline. Semgrep rules and OSV/Trivy databases must be baked into the reviewed image. Their content hashes, scope, and freshness accompany the result. Database refresh is a separately reviewed image acquisition/release operation; scans never download missing assets or switch to unrestricted registry networking.

REQUEST.json can select an exact scanner profile. Schemathesis additionally requires comprehensive mode and an api object; gstack-cso schema describes its fields. The API harness supplies a bounded OpenAPI 3.0/3.1 JSON document, 1–20 unique path operation IDs, a qualified application runtime, an absolute start command, a numeric loopback port, a legitimate HTTP control, and the source files defining the security boundary. Internal schema references are allowed. Server overrides, remote references/examples, hooks, callbacks, and webhooks are rejected. The application must boot and pass the control before the scanner starts. Application and scanner share only the admitted loopback namespace. An API failure is a candidate, never a reproduced vulnerability or certified repair. Sources needing dependency preparation report that prerequisite until a prepared offline closure is available to the API runner.

Each execution probes the scanner version before scanning and validates the retained source both before and after collection. Unknown output, failed diagnostics, a timeout, changed source, or incomplete cleanup cannot become an empty clean result. The helper retains valid partial candidates when the adapter can distinguish them from failed coverage. Scanner output is redacted in place: each located sensitive span becomes a <REDACTED-...> marker, so an advisory URL or version string that resembles a phone number or IP address does not discard the whole report. Output that cannot be redacted, including a span split across stdout and stderr, is withheld.

Scanner and SARIF collection hold the run's mutation lock and reject finished or interrupted reports. Every outcome is written exclusively under a unique scanner-outcomes/ name before its coverage record enters the report. The returned artifact path is relative to the run directory. Model submissions cannot replace scanner:* coverage or certify the imported candidates. The free scanner tests use injected runners and catalogs to verify dispatch and failure handling; they do not claim real image qualification. The opt-in Docker qualification test runs the production adapter, image, watchdog, and isolation policy. It emits the version-output hash only after a representative scan completes without a coverage gap.

The reviewed release inputs in build-inputs.json cover all six scanners on both native platforms. The inputs require immutable base and SBOM-generator image digests, exact source commits and versions, signer workflow identities and digests, and reviewed canonical SLSA/SPDX statement-set digests. The workflow cryptographically re-verifies each image with those repository, source, signer, and predicate constraints before use. Semgrep additionally requires a reviewed local rules bundle. OSV and Trivy require complete offline database bundles with recorded content hashes, freshness, and ecosystem coverage. The release workflow does not invent or silently replace them. None of the upstream scanner images publish GitHub artifact attestations, so the reviewed bases come from garrytan/gstack-cso-scanner-bases. That repository re-publishes each pinned upstream digest with the offline assets added and signs provenance and an SPDX SBOM with its own workflow. Each tagged release attaches a pending build-inputs.json candidate; a reviewer independently re-verifies the identities and hashes before the state here changes to reviewed.

cso-scanner-images.yml lets a dispatched branch run read-only input and contract validation. Image publishing, native amd64/arm64 qualification, attestation verification, embedded-asset hash checks, containment, real-adapter execution, and catalog-proposal generation run only from protected main through the cso-scanner-release environment. Before enabling qualification or promotion, configure the repository's cso-scanner-release GitHub environment with required maintainer reviewers and deployments restricted to protected main. GitHub otherwise creates a referenced missing environment without protection rules. Promotion requires an explicit dispatch from main and approval through that configured environment. It opens a pull request containing only the proposed source catalog, so code review remains the final authorization step. Pushes and pull requests made with the workflow's GITHUB_TOKEN start no workflows, so after pushing the promotion branch the job dispatches free-tests.yml on it (actions: write); that run reports the required free-tests check on the branch head. Opening the PR also requires the repository setting Allow GitHub Actions to create and approve pull requests (Settings → Actions → General → Workflow permissions); without it gh pr create fails after the push and the dispatch, and a maintainer opens the PR from the pushed branch. previousRevision records the prior compatible catalog for rollback and must still equal the current main catalog when promotion runs.

Scanner flags follow the primary documentation linked in scanners.ts. Offline qualification must exercise the exact chosen version: OSV's maintainers documented a cache-location regression in 2.5.0, which demonstrates why a version string alone is insufficient. OSV issue #2983. Trivy uses an explicit memory scan cache alongside read-only baked databases and a baked misconfiguration checks bundle. Schemathesis 4.26 and later repeat fuzzing until --max-time is spent, so the adapter sets that budget to three quarters of the scanner deadline and treats a max_time stop as complete only when every selected operation was exercised. Trivy filesystem reference.