mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 22:45:02 +01:00
Garry Tan
857466ff8b
* v1.91.23.0 feat(cso): private two-architecture release scoring and self-checking runtime release gates
* Keep the Linux-only native-evidence and requalification tests out of the Windows lane
* Skip the two Linux-release-only checks on Windows in-file instead of growing the shard list
* fix(cso): opaque producer inputs and receipts; requalification triggers independent of cwd
Producer inputs are {schemaVersion: 2, cellRef, skill, source, execution} and
receipts bind cellRef + inputHash; collection rebuilds the cell from the
matrix, so a cell VM never learns its case or variant. requalificationTriggers
resolves Bun's cwd-relative metafile inputs against the repository root.
* test(evals): bound the review judge's thinking; guarded browser cleanup is not a probe
review/SKILL.md workflow: the ~64k-token bundle let default-effort thinking
spend all 8192 output tokens without an answer in CI. Medium effort with a
64k streaming cap kept 24/24 measured samples under 5k tokens and 63 s with
the same gating scores (as ship already does).
qa-only-no-fix: the fixture prompt routes browser cleanup through the deadline
guard, but the checkpoint detector treated every guarded run as a probe that
needs a checkpoint. A child that only closes the tab with the supplied browse
binary, after the last probe, is cleanup; anything else stays a probe. The
captured CI transcript (run 37252022510) now passes and new scenarios pin
the negative controls.
* test(evals): keep the review judge at default effort; only lift its output cap
Truncation was the defect: thinking and the answer share max_tokens, and the
~64k-token review bundle once spent all 8192. The registration keeps default
effort and response format and gets ship's 64k streaming cap; the guard test
pins exactly that.
124 lines
8.2 KiB
Markdown
124 lines
8.2 KiB
Markdown
# Running CSO scanners
|
||
|
||
`gstack-cso scan RUN SCANNER [REQUEST.json]` collects candidate evidence through
|
||
the trusted helper when a matching qualified scanner catalog profile exists.
|
||
Recognized IDs are `gitleaks`, `osv`, `semgrep`, `zizmor`, `trivy`, and
|
||
`schemathesis`. Without a qualified profile, the command records a specific
|
||
prerequisite; static assessment and importing existing CodeQL or other SARIF
|
||
with `gstack-cso import-sarif RUN REPORT.sarif` remain available.
|
||
The returned immutable artifact can be read later with
|
||
`gstack-cso scanner-outcome RUN ARTIFACT_ID`; callers never need to open private
|
||
state files directly.
|
||
|
||
The source-controlled catalog in
|
||
[`scanner-images/catalog.json`](../lib/cso/scanner-images/catalog.json) is empty
|
||
until every release gate produces real evidence. Without a reviewed profile,
|
||
the command records a specific `not_assessed` prerequisite. Installing a
|
||
scanner on the host does not enable execution. Each profile binds a Linux
|
||
platform, immutable image digest, fixed entrypoint, absolute scanner executable,
|
||
version-probe hash, required adapter capabilities, helper ABI, and the exact
|
||
isolation-policy hash. The helper validates that catalog when it starts and has
|
||
no executable fallback.
|
||
Setup preloads the current platform's qualified scanner digests through the
|
||
trusted installation helper using anonymous public-registry access. A failed
|
||
preload or a bounded per-image/setup deadline is a nonfatal prerequisite and
|
||
never causes an audit to pull. The default is 30 seconds per declared image;
|
||
`GSTACK_CSO_IMAGE_PULL_TIMEOUT_SECONDS=120` selects a longer allowance (accepted
|
||
range: 5–300 seconds) while the complete preload remains capped at one hour. Doctor is
|
||
read-only and reports each scanner unavailable unless its exact catalog digest
|
||
is already present locally.
|
||
|
||
All scanner processes use Docker containment with a watchdog, shared admission
|
||
limits, read-only source, bounded output, and an isolated network namespace.
|
||
Static scanners have no network access, including when the audit omits
|
||
`--offline`. Semgrep rules and OSV/Trivy databases must be baked into the reviewed
|
||
image. Their content hashes, scope, and freshness accompany the result. Database
|
||
refresh is a separately reviewed image acquisition/release operation; scans
|
||
never download missing assets or switch to unrestricted registry networking.
|
||
|
||
`REQUEST.json` can select an exact scanner `profile`. Schemathesis additionally
|
||
requires comprehensive mode and an `api` object; `gstack-cso schema` describes
|
||
its fields. The API harness supplies a bounded OpenAPI 3.0/3.1 JSON document,
|
||
1–20 unique path operation IDs, a qualified application runtime, an absolute
|
||
start command, a numeric loopback port, a legitimate HTTP control, and the
|
||
source files defining the security boundary. Internal schema references are
|
||
allowed. Server overrides, remote references/examples, hooks, callbacks, and
|
||
webhooks are rejected. The application must boot and pass the control before
|
||
the scanner starts. Application and scanner share only the admitted loopback
|
||
namespace. An API failure is a candidate, never a reproduced vulnerability or
|
||
certified repair. Sources needing dependency preparation report that prerequisite
|
||
until a prepared offline closure is available to the API runner.
|
||
|
||
Each execution probes the scanner version before scanning and validates the
|
||
retained source both before and after collection. Unknown output, failed
|
||
diagnostics, a timeout, changed source, or incomplete cleanup cannot become an
|
||
empty clean result. The helper retains valid partial candidates when the
|
||
adapter can distinguish them from failed coverage. Scanner output is redacted
|
||
in place: each located sensitive span becomes a `<REDACTED-...>` marker, so an
|
||
advisory URL or version string that resembles a phone number or IP address does
|
||
not discard the whole report. Output that cannot be redacted, including a span
|
||
split across stdout and stderr, is withheld.
|
||
|
||
Scanner and SARIF collection hold the run's mutation lock and reject finished
|
||
or interrupted reports. Every outcome is written exclusively under a unique
|
||
`scanner-outcomes/` name before its coverage record enters the report. The
|
||
returned `artifact` path is relative to the run directory. Model submissions
|
||
cannot replace `scanner:*` coverage or certify the imported candidates. The
|
||
free scanner tests use injected runners and catalogs to verify dispatch and
|
||
failure handling; they do not claim real image qualification. The opt-in Docker
|
||
qualification test runs the production adapter, image, watchdog, and isolation
|
||
policy. It emits the version-output hash only after a representative scan
|
||
completes without a coverage gap.
|
||
|
||
The reviewed release inputs in
|
||
[`build-inputs.json`](../lib/cso/scanner-images/build-inputs.json) cover all six
|
||
scanners on both native platforms. The inputs require immutable base and SBOM-generator image digests, exact source
|
||
commits and versions, signer workflow identities and digests, and reviewed
|
||
canonical SLSA/SPDX statement-set digests. The workflow cryptographically
|
||
re-verifies each image with those repository, source, signer, and predicate
|
||
constraints before use. Semgrep additionally requires a reviewed local rules bundle. OSV and
|
||
Trivy require complete offline database bundles with recorded content hashes,
|
||
freshness, and ecosystem coverage. The release workflow does not invent or
|
||
silently replace them. None of the upstream scanner images publish GitHub
|
||
artifact attestations, so the reviewed bases come from
|
||
[`garrytan/gstack-cso-scanner-bases`](https://github.com/garrytan/gstack-cso-scanner-bases).
|
||
That repository re-publishes each pinned upstream digest with the offline assets
|
||
added and signs provenance and an SPDX SBOM with its own workflow. Each tagged
|
||
release attaches a `pending` `build-inputs.json` candidate; a reviewer
|
||
independently re-verifies the identities and hashes before the state here
|
||
changes to `reviewed`.
|
||
|
||
[`cso-scanner-images.yml`](../.github/workflows/cso-scanner-images.yml) lets a
|
||
dispatched branch run read-only input and contract validation. Image publishing,
|
||
native amd64/arm64 qualification, attestation verification, embedded-asset hash
|
||
checks, containment, real-adapter execution, and catalog-proposal generation run
|
||
only from protected `main` through the `cso-scanner-release` environment. Before
|
||
enabling qualification or promotion, configure the repository's
|
||
`cso-scanner-release` GitHub environment with required maintainer reviewers and
|
||
deployments restricted to protected `main`. GitHub otherwise creates a referenced
|
||
missing environment without protection rules. Promotion requires an explicit
|
||
dispatch from `main` and approval through that configured environment. It opens a pull request
|
||
containing only the proposed source catalog, so code review remains the final
|
||
authorization step. Pushes and pull requests made with the workflow's
|
||
`GITHUB_TOKEN` start no workflows, so after pushing the promotion branch the job
|
||
dispatches [`free-tests.yml`](../.github/workflows/free-tests.yml) on it
|
||
(`actions: write`); that run reports the required `free-tests` check on the
|
||
branch head. Opening the PR also requires the repository setting **Allow GitHub
|
||
Actions to create and approve pull requests** (Settings → Actions → General →
|
||
Workflow permissions); without it `gh pr create` fails after the push and the
|
||
dispatch, and a maintainer opens the PR from the pushed branch.
|
||
`previousRevision` records the prior compatible catalog for
|
||
rollback and must still equal the current `main` catalog when promotion runs.
|
||
|
||
Scanner flags follow the primary documentation linked in
|
||
[`scanners.ts`](../lib/cso/scanners.ts). Offline qualification must exercise
|
||
the exact chosen version: OSV's maintainers documented a cache-location
|
||
regression in 2.5.0, which demonstrates why a version string alone is insufficient.
|
||
[OSV issue #2983](https://github.com/google/osv-scanner/issues/2983).
|
||
Trivy uses an explicit memory scan cache alongside read-only baked databases
|
||
and a baked misconfiguration checks bundle. Schemathesis 4.26 and later repeat
|
||
fuzzing until `--max-time` is spent, so the adapter sets that budget to three
|
||
quarters of the scanner deadline and treats a `max_time` stop as complete only
|
||
when every selected operation was exercised.
|
||
[Trivy filesystem reference](https://trivy.dev/docs/v0.68/guide/references/configuration/cli/trivy_filesystem/).
|