mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-22 12:50:50 +02:00
v1.87.0.0 feat: add verified CSO audits and replayable repair bundles (#2852)
* feat(cso): add verified audits and replayable repair bundles * fix(cso): harden qualification and setup boundaries * fix(cso): assemble security canaries at runtime * fix(cso): bound release proof and maintenance work Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(cso): require complete evaluation reports Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(cso): replay expired snapshots from supplied source Co-Authored-By: OpenAI Codex <noreply@openai.com> * test(cso): synchronize DNS cancellation assertion Co-Authored-By: OpenAI Codex <noreply@openai.com> * chore(ship): exempt repository owner from liveness proof Co-Authored-By: OpenAI Codex <noreply@openai.com> * test(cso): make recheck retention overlap deterministic Co-Authored-By: OpenAI Codex <noreply@openai.com> * chore: bump version and changelog (v1.85.0.0) Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(cso): pass native release gates Co-Authored-By: OpenAI Codex <noreply@openai.com> * chore: move release to v1.86.0.0 Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(cso): resolve rechecks by finding Co-Authored-By: OpenAI Codex <noreply@openai.com> * chore: move release to v1.87.0.0 Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(cso): pass macOS and Windows release gates Normalize BSD wc output, compare Windows paths by filesystem identity, preserve portable snapshot race coverage, and narrow POSIX-only Windows fixtures. Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix(cso): harden native verification gates * fix(cso): refine Windows native diagnostics * test(cso): isolate Windows Git startup failure * test(cso): stabilize Windows native diagnostics * fix(cso): support hardened Git on Windows * fix(cso): close final verification gaps * test(cso): bound cold Docker fixture setup * fix(cso): restore cross-platform free-suite gates --------- Co-authored-by: OpenAI Codex <noreply@openai.com>
This commit is contained in:
Garry Tan
co-authored by
OpenAI Codex
OpenAI Codex
parent
9f81911136
commit
4a3c6a8a3c
Vendored
+254
@@ -0,0 +1,254 @@
|
||||
# CSO vulnerable/fixed evaluation corpus
|
||||
|
||||
This corpus contains **40 immutable source pairs**: ten each for Node, Bun,
|
||||
Python, and Rails. `manifest.json` pins both source hashes, expected root cause,
|
||||
location, severity, and evaluation eligibility. `materialize.ts` deterministically
|
||||
creates either member of each pair. Changing source requires an explicit corpus
|
||||
manifest update; the loader rejects silent drift.
|
||||
|
||||
The ten families are SQL injection, command injection, path traversal, SSRF,
|
||||
object authorization, tenant isolation, HTML injection, open redirects, mass
|
||||
assignment, and resource exhaustion. Every app exposes `/action` and `/health` on
|
||||
loopback port 8000. The SSRF fixtures additionally create a disposable loopback
|
||||
status service on port 8001. Exhaustion assertions request 250 small records to
|
||||
prove missing admission; they do not attempt to exhaust the evaluator.
|
||||
|
||||
Node uses standard modules, including `node:sqlite`; Bun uses `bun:sqlite`; Python
|
||||
uses its standard library. Rails fixtures contain an actual Rails application
|
||||
and controller, with native SQLite and Puma dependencies. Their common lock was
|
||||
generated by Ruby 3.2.8 / Bundler 2.6.7 from public registry metadata only. The
|
||||
exact command, lock hash, and 68 public archive hashes are recorded in
|
||||
`rails-lock-provenance.json`. No fixture application was run to generate this
|
||||
lock. The recorded archive hashes must be checked by runtime qualification; a
|
||||
successful resolver run does not establish cold-start support.
|
||||
|
||||
Authentication is an explicit fixture precondition: the test adapter establishes
|
||||
member-1 in tenant-a. The assessment scope is the selected endpoint's behavior.
|
||||
This prevents an intentionally constant test identity from being mistaken for a
|
||||
production authentication design. Application README files state legitimate
|
||||
behavior; they never include attack payloads or expected finding labels.
|
||||
|
||||
## Preparing matched evaluations
|
||||
|
||||
From the repository root, export the v2 generated skill tree and build a
|
||||
canonical portable payload for each version before pinning an exact model ID:
|
||||
|
||||
```sh
|
||||
mkdir -p .context/cso-v2-source
|
||||
git archive origin/main cso/SKILL.md cso/sections | \
|
||||
tar -x -C .context/cso-v2-source --strip-components=1
|
||||
bun scripts/cso-eval.ts payload --version v2 \
|
||||
--skill-dir .context/cso-v2-source --output .context/cso-v2.payload.md
|
||||
bun scripts/cso-eval.ts payload --version v3 \
|
||||
--skill-dir cso --output .context/cso-v3.payload.md
|
||||
bun scripts/cso-eval.ts matrix \
|
||||
--model EXACT_MODEL_ID --host codex \
|
||||
--v2-payload .context/cso-v2.payload.md \
|
||||
--v3-payload .context/cso-v3.payload.md \
|
||||
--output .context/cso-eval-matrix.json
|
||||
bun scripts/cso-eval.ts materialize node-sql-injection vulnerable .context/cso-eval-app
|
||||
```
|
||||
|
||||
The full matrix contains 960 cells: 40 pairs × two source variants × two modes ×
|
||||
two skill versions × three repetitions. Each matched pair uses identical source,
|
||||
model, host, and per-mode wall-clock budget. Daily runs have 600 seconds;
|
||||
comprehensive runs have 1800 seconds. Each portable payload embeds the exact
|
||||
generated `SKILL.md`, `sections/manifest.json`, and every generated section
|
||||
listed by that manifest. The payload builder rejects missing or unlisted
|
||||
generated sections and a mismatched major version. The matrix hashes the whole
|
||||
payload, so a change to any section byte changes the pinned skill identity. The
|
||||
commands above make no model calls and execute no fixture application.
|
||||
|
||||
Prepare consumable one-cell jobs and compile the generic producer runner:
|
||||
|
||||
```sh
|
||||
bun build --compile \
|
||||
--no-compile-autoload-dotenv \
|
||||
--no-compile-autoload-bunfig \
|
||||
--no-compile-autoload-tsconfig \
|
||||
--no-compile-autoload-package-json \
|
||||
scripts/cso-eval-producer.ts --outfile .context/cso-eval-producer
|
||||
bun scripts/cso-eval.ts prepare .context/cso-eval-matrix.json \
|
||||
--v2-payload .context/cso-v2.payload.md \
|
||||
--v3-payload .context/cso-v3.payload.md \
|
||||
--output .context/cso-eval-jobs
|
||||
```
|
||||
|
||||
### Trusted five-artifact producer unit
|
||||
|
||||
The `.context/cso-eval-producer` command above creates a preparation artifact;
|
||||
ordinary `bun run build` does not distribute the private producer. For a paid
|
||||
run, start from one clean checkout and build a five-artifact staging unit in one
|
||||
session:
|
||||
|
||||
```sh
|
||||
stage="$(mktemp -d)"
|
||||
bun run build:cso
|
||||
bun build --compile \
|
||||
--no-compile-autoload-dotenv \
|
||||
--no-compile-autoload-bunfig \
|
||||
--no-compile-autoload-tsconfig \
|
||||
--no-compile-autoload-package-json \
|
||||
scripts/cso-eval-producer.ts --outfile "$stage/cso-eval-producer"
|
||||
install -m 0555 bin/gstack-cso-launcher bin/gstack-cso-core \
|
||||
bin/gstack-cso-watchdog "$stage/"
|
||||
install -m 0444 bin/.gstack-cso-generation "$stage/.gstack-cso-generation"
|
||||
(cd "$stage" && (sha256sum cso-eval-producer gstack-cso-launcher \
|
||||
gstack-cso-core gstack-cso-watchdog .gstack-cso-generation 2>/dev/null || \
|
||||
shasum -a 256 cso-eval-producer gstack-cso-launcher gstack-cso-core \
|
||||
gstack-cso-watchdog .gstack-cso-generation))
|
||||
sudo install -d -o root -g root -m 0755 /opt/gstack-cso-producer
|
||||
sudo install -o root -g root -m 0555 "$stage/cso-eval-producer" \
|
||||
"$stage/gstack-cso-launcher" "$stage/gstack-cso-core" \
|
||||
"$stage/gstack-cso-watchdog" /opt/gstack-cso-producer/
|
||||
sudo install -o root -g root -m 0444 "$stage/.gstack-cso-generation" \
|
||||
/opt/gstack-cso-producer/.gstack-cso-generation
|
||||
```
|
||||
|
||||
Run `/opt/gstack-cso-producer/cso-eval-producer` as an unprivileged account.
|
||||
Its startup check rejects missing, linked, writable, or non-root-owned members
|
||||
and a writable or linked ancestor directory. Every receipt records the SHA-256
|
||||
and byte length of all five artifacts in one installation identity; collection
|
||||
rejects a batch if those identities differ. Preserve the printed hashes with
|
||||
the release evidence and move or replace the complete unit together.
|
||||
|
||||
`prepare` creates a trusted `schedule.json` and 960 independent directories.
|
||||
It initializes each generated application as a one-commit Git repository. Each
|
||||
job has exactly one source variant and one versioned, complete instruction
|
||||
payload. Root-skill pointers resolve only to sections embedded in that payload;
|
||||
the producer must never load CSO instructions from `~/.claude`, another host
|
||||
install, this source checkout, or the other version's payload. The generic runner
|
||||
is compiled separately and contains no corpus generator, fixed alternative,
|
||||
case oracle, or schedule. These preparation commands still make zero model calls.
|
||||
|
||||
Run each job on a fresh producer filesystem that contains only that job, the
|
||||
compiled runner, the authenticated provider CLI, and the reviewed executable
|
||||
gstack helper/runtime files. Do not install or copy any CSO `SKILL.md`, carved
|
||||
section, generated skill tree, schedule, this repository checkout,
|
||||
sibling jobs, fixed alternatives, or evaluator code to that filesystem. The
|
||||
producer runner reads its opaque control file into memory and deletes it before
|
||||
the agent process starts, so the case identifier and vulnerable/fixed label are
|
||||
not agent inputs. Copy the selected hash-named directory as the literal path
|
||||
`/producer/job`; `/producer` must contain only `job`. Put the runner in a system
|
||||
tool directory and use a separate receipt mount. The runner rejects the full
|
||||
prepared batch layout, extra files beside `job`, a reused job with prior state,
|
||||
and receipt paths under `/producer`. The dedicated VM/container must ensure
|
||||
other host paths do not contain evaluator inputs; a local directory layout does
|
||||
not constrain a tool-using agent's absolute filesystem access.
|
||||
|
||||
The receipt must be written outside the isolated producer root:
|
||||
|
||||
```sh
|
||||
mkdir -m 700 /receipts
|
||||
CSO_EVAL_PAID=1 /opt/gstack-cso-producer/cso-eval-producer run \
|
||||
/producer/job/producer-input.json /receipts/CELL_ID.json --execute-paid
|
||||
```
|
||||
|
||||
Both the environment variable and flag are required because this is the only
|
||||
command in the workflow that makes a paid model call. It reuses the repository's
|
||||
Claude, Codex, and Gemini provider adapters and starts a fresh CLI process for
|
||||
each cell. The requested model, normalized effective model, identity source,
|
||||
timeout, output, token counts, pricing-table cost estimate, tool-call count, and
|
||||
latency are bound into a hashed receipt. `provider_reported` means the CLI
|
||||
resolved a different concrete ID; `requested_pin` means the adapter returned
|
||||
the exact requested ID or had to fall back to that exact CLI pin. Use a concrete
|
||||
model ID rather than an alias. The current adapters return only completed runs,
|
||||
so first-useful-result latency is explicitly unmeasured rather than copied from
|
||||
total latency. If a job is interrupted after its input is consumed, restore it
|
||||
from the trusted prepared copy; never synthesize a receipt.
|
||||
|
||||
After returning receipts to the trusted evaluator, collect them against the
|
||||
matrix and schedule:
|
||||
|
||||
```sh
|
||||
bun scripts/cso-eval.ts collect .context/cso-eval-matrix.json \
|
||||
.context/cso-eval-jobs/schedule.json .context/cso-eval-receipts \
|
||||
.context/cso-eval-producer-batch.json
|
||||
```
|
||||
|
||||
Collection makes no model calls. It rejects unknown, duplicate, changed, or
|
||||
wrong-input receipts and reports scheduled/submitted/missing denominators for
|
||||
every v2/v3 and daily/comprehensive group. It rejects an effective-model mismatch
|
||||
between matched v2/v3 cells, leaving the source receipts available for diagnosis
|
||||
and a clean rerun. The collected batch stores a compact receipt index and hashes;
|
||||
raw transcripts remain in the private receipt directory for adjudication. Cost
|
||||
in this batch is clearly labeled a
|
||||
pricing-table estimate; only a host-billed amount may populate `EvalResult`'s
|
||||
host-cost field.
|
||||
|
||||
The producer host needs Git, the chosen authenticated CLI, and reviewed gstack
|
||||
executables and catalogs without their skill source tree. Its CSO instruction
|
||||
bytes come only from the cell payload, never the installed skill or installed
|
||||
carved sections. Comprehensive cells additionally need a local Docker daemon,
|
||||
the exact qualified helper/runtime catalog pair, public dependency inputs or
|
||||
verified cache hits, and the authenticated out-of-process assertion witness. The runtime
|
||||
image workflow currently emits native staging evidence with private accuracy,
|
||||
held-out repair, canary, and watchdog gates still pending. Such evidence can
|
||||
measure setup failures, but it cannot support a release-qualifying tested repair
|
||||
until those gates and catalog promotion are complete.
|
||||
|
||||
## Private oracle boundary
|
||||
|
||||
`test/helpers/cso-eval-oracles.ts` belongs to the **trusted evaluator**. It contains
|
||||
the legitimate controls, initial security assertions, alternate held-out
|
||||
assertions, and non-vacuous repair judge. It is never copied into materialized
|
||||
source. The fixed alternative, expected manifest, and this evaluator module must
|
||||
remain unavailable to producing agents throughout their sessions.
|
||||
|
||||
A separate directory alone does not restrict a host agent's file access. The
|
||||
evaluation host must enforce this boundary with scoped tools or an isolated
|
||||
producer filesystem and test it with the `held-out-oracle-visibility` gate. If it
|
||||
cannot do that, report the gate as `not_run` and do not claim held-out
|
||||
qualification. Product execution containment does not automatically isolate the
|
||||
hosted producing agent.
|
||||
|
||||
Only trusted runtime observations enter `judgeRepair`. Boot and legitimate
|
||||
controls must pass on the original; the exact security assertion must fail for
|
||||
the intended reason. The patched app must pass that assertion, all held-out
|
||||
assertions, controls, and existing tests with unchanged verification inputs.
|
||||
Independent root-cause/feature review is mandatory; boundary-replacing mocks or
|
||||
disabled functionality fail certification. A producer's `tested` or
|
||||
`runtime_tested` label is never an oracle result.
|
||||
|
||||
`collect` preserves producer claims and performance receipts; it does not turn
|
||||
them into `EvalResult`. The independent evaluator must inspect the retained CSO
|
||||
report/bundle, execute the private assertions, and write finding judgments and
|
||||
workflow outcomes. This separation is deliberate: automatically copying agent
|
||||
claims into the trusted result contract would fabricate verification.
|
||||
|
||||
## Scoring and release gates
|
||||
|
||||
```sh
|
||||
bun scripts/cso-eval.ts score .context/cso-eval-matrix.json \
|
||||
.context/cso-eval-producer-batch.json .context/trusted-cso-results.json \
|
||||
.context/cso-qualification.json
|
||||
```
|
||||
|
||||
The result contract is `EvalResult` in `scripts/cso-eval.ts`. The independent
|
||||
evaluator supplies finding judgments, the matching `producerReceiptHash`, and
|
||||
hashes of private evidence. The release scoring command requires a complete
|
||||
producer batch, checks its integrity and effective-model parity, and rejects any
|
||||
trusted result that is not bound to that cell's receipt. Recheck
|
||||
success additionally requires a distinct current-source observation and its
|
||||
source hash. Correct alternative patches need not match the reference fix's
|
||||
text. Keep actual transcripts and reports in private evaluation state; do not
|
||||
commit them beside the fixtures.
|
||||
|
||||
Each metric includes its numerator and denominator. Duplicate findings cannot
|
||||
inflate true positives. Fixed variants contribute false positives. Missing and
|
||||
setup-blocked supported scenarios remain misses in recall and workflow rates.
|
||||
Hypotheses do not become supported discoveries; v2 legacy review evidence cannot
|
||||
become v3 reproduced, runtime-tested, or tested evidence. Unknown usage remains
|
||||
unknown, and wall-clock budgets are not represented as model-spend caps.
|
||||
|
||||
The scorer checks 95% daily precision, 80% comprehensive high/critical recall,
|
||||
no aggregate high/critical recall regression against v2, all core cold starts,
|
||||
zero falsely certified runtime-tested repairs, and a correct held-out repair in
|
||||
every application stack. It also requires complete matched results, mandatory
|
||||
reports, containment/canary receipts, and enforced oracle separation. Missing
|
||||
evaluations return
|
||||
`unmeasured` or `partial`, never passing release gates.
|
||||
|
||||
The free accounting tests use clearly labeled synthetic observations to test the
|
||||
scorer. They are **not measured agent performance**. No paid comparison, fixture
|
||||
cold-start qualification, or containment qualification is claimed by this corpus.
|
||||
Reference in New Issue
Block a user