# CSO vulnerable/fixed evaluation corpus This corpus contains **40 immutable source pairs**: ten each for Node, Bun, Python, and Rails. `manifest.json` pins both source hashes, expected root cause, location, severity, and evaluation eligibility. `materialize.ts` deterministically creates either member of each pair. Changing source requires an explicit corpus manifest update; the loader rejects silent drift. The ten families are SQL injection, command injection, path traversal, SSRF, object authorization, tenant isolation, HTML injection, open redirects, mass assignment, and resource exhaustion. Every app exposes `/action` and `/health` on loopback port 8000. The SSRF fixtures additionally create a disposable loopback status service on port 8001. Exhaustion assertions request 250 small records to prove missing admission; they do not attempt to exhaust the evaluator. Node uses standard modules, including `node:sqlite`; Bun uses `bun:sqlite`; Python uses its standard library. Rails fixtures contain an actual Rails application and controller, with native SQLite and Puma dependencies. Their common lock was generated by Ruby 3.2.8 / Bundler 2.6.7 from public registry metadata only. The exact command, lock hash, and 68 public archive hashes are recorded in `rails-lock-provenance.json`. No fixture application was run to generate this lock. The recorded archive hashes must be checked by runtime qualification; a successful resolver run does not establish cold-start support. Authentication is an explicit fixture precondition: the test adapter establishes member-1 in tenant-a. The assessment scope is the selected endpoint's behavior. This prevents an intentionally constant test identity from being mistaken for a production authentication design. Application README files state legitimate behavior; they never include attack payloads or expected finding labels. ## Preparing matched evaluations From the repository root, export the v2 generated skill tree and build a canonical portable payload for each version before pinning an exact model ID: ```sh mkdir -p .context/cso-v2-source git archive origin/main cso/SKILL.md cso/sections | \ tar -x -C .context/cso-v2-source --strip-components=1 bun scripts/cso-eval.ts payload --version v2 \ --skill-dir .context/cso-v2-source --output .context/cso-v2.payload.md bun scripts/cso-eval.ts payload --version v3 \ --skill-dir cso --output .context/cso-v3.payload.md bun scripts/cso-eval.ts matrix \ --model EXACT_MODEL_ID --host codex \ --v2-payload .context/cso-v2.payload.md \ --v3-payload .context/cso-v3.payload.md \ --output .context/cso-eval-matrix.json bun scripts/cso-eval.ts materialize node-sql-injection vulnerable .context/cso-eval-app ``` The full matrix contains 960 cells: 40 pairs × two source variants × two modes × two skill versions × three repetitions. Each matched pair uses identical source, model, host, and per-mode wall-clock budget. Daily runs have 600 seconds; comprehensive runs have 1800 seconds. Each portable payload embeds the exact generated `SKILL.md`, `sections/manifest.json`, and every generated section listed by that manifest. The payload builder rejects missing or unlisted generated sections and a mismatched major version. The matrix hashes the whole payload, so a change to any section byte changes the pinned skill identity. The commands above make no model calls and execute no fixture application. Prepare consumable one-cell jobs and compile the generic producer runner: ```sh bun build --compile \ --no-compile-autoload-dotenv \ --no-compile-autoload-bunfig \ --no-compile-autoload-tsconfig \ --no-compile-autoload-package-json \ scripts/cso-eval-producer.ts --outfile .context/cso-eval-producer bun scripts/cso-eval.ts prepare .context/cso-eval-matrix.json \ --v2-payload .context/cso-v2.payload.md \ --v3-payload .context/cso-v3.payload.md \ --output .context/cso-eval-jobs ``` ### Trusted five-artifact producer unit The `.context/cso-eval-producer` command above creates a preparation artifact; ordinary `bun run build` does not distribute the private producer. For a paid run, start from one clean checkout and build a five-artifact staging unit in one session: ```sh stage="$(mktemp -d)" bun run build:cso bun build --compile \ --no-compile-autoload-dotenv \ --no-compile-autoload-bunfig \ --no-compile-autoload-tsconfig \ --no-compile-autoload-package-json \ scripts/cso-eval-producer.ts --outfile "$stage/cso-eval-producer" install -m 0555 bin/gstack-cso-launcher bin/gstack-cso-core \ bin/gstack-cso-watchdog "$stage/" install -m 0444 bin/.gstack-cso-generation "$stage/.gstack-cso-generation" (cd "$stage" && (sha256sum cso-eval-producer gstack-cso-launcher \ gstack-cso-core gstack-cso-watchdog .gstack-cso-generation 2>/dev/null || \ shasum -a 256 cso-eval-producer gstack-cso-launcher gstack-cso-core \ gstack-cso-watchdog .gstack-cso-generation)) sudo install -d -o root -g root -m 0755 /opt/gstack-cso-producer sudo install -o root -g root -m 0555 "$stage/cso-eval-producer" \ "$stage/gstack-cso-launcher" "$stage/gstack-cso-core" \ "$stage/gstack-cso-watchdog" /opt/gstack-cso-producer/ sudo install -o root -g root -m 0444 "$stage/.gstack-cso-generation" \ /opt/gstack-cso-producer/.gstack-cso-generation ``` Run `/opt/gstack-cso-producer/cso-eval-producer` as an unprivileged account. Its startup check rejects missing, linked, writable, or non-root-owned members and a writable or linked ancestor directory. Every receipt records the SHA-256 and byte length of all five artifacts in one installation identity; collection rejects a batch if those identities differ. Preserve the printed hashes with the release evidence and move or replace the complete unit together. `prepare` creates a trusted `schedule.json` and 960 independent directories. It initializes each generated application as a one-commit Git repository. Each job has exactly one source variant and one versioned, complete instruction payload. Root-skill pointers resolve only to sections embedded in that payload; the producer must never load CSO instructions from `~/.claude`, another host install, this source checkout, or the other version's payload. The generic runner is compiled separately and contains no corpus generator, fixed alternative, case oracle, or schedule. These preparation commands still make zero model calls. Run each job on a fresh producer filesystem that contains only that job, the compiled runner, the authenticated provider CLI, and the reviewed executable gstack helper/runtime files. Do not install or copy any CSO `SKILL.md`, carved section, generated skill tree, schedule, this repository checkout, sibling jobs, fixed alternatives, or evaluator code to that filesystem. The producer runner reads its opaque control file into memory and deletes it before the agent process starts, so the case identifier and vulnerable/fixed label are not agent inputs. Copy the selected hash-named directory as the literal path `/producer/job`; `/producer` must contain only `job`. Put the runner in a system tool directory and use a separate receipt mount. The runner rejects the full prepared batch layout, extra files beside `job`, a reused job with prior state, and receipt paths under `/producer`. The dedicated VM/container must ensure other host paths do not contain evaluator inputs; a local directory layout does not constrain a tool-using agent's absolute filesystem access. The receipt must be written outside the isolated producer root: ```sh mkdir -m 700 /receipts CSO_EVAL_PAID=1 /opt/gstack-cso-producer/cso-eval-producer run \ /producer/job/producer-input.json /receipts/CELL_ID.json --execute-paid ``` Both the environment variable and flag are required because this is the only command in the workflow that makes a paid model call. It reuses the repository's Claude, Codex, and Gemini provider adapters and starts a fresh CLI process for each cell. The requested model, normalized effective model, identity source, timeout, output, token counts, pricing-table cost estimate, tool-call count, and latency are bound into a hashed receipt. `provider_reported` means the CLI resolved a different concrete ID; `requested_pin` means the adapter returned the exact requested ID or had to fall back to that exact CLI pin. Use a concrete model ID rather than an alias. The current adapters return only completed runs, so first-useful-result latency is explicitly unmeasured rather than copied from total latency. If a job is interrupted after its input is consumed, restore it from the trusted prepared copy; never synthesize a receipt. After returning receipts to the trusted evaluator, collect them against the matrix and schedule: ```sh bun scripts/cso-eval.ts collect .context/cso-eval-matrix.json \ .context/cso-eval-jobs/schedule.json .context/cso-eval-receipts \ .context/cso-eval-producer-batch.json ``` Collection makes no model calls. It rejects unknown, duplicate, changed, or wrong-input receipts and reports scheduled/submitted/missing denominators for every v2/v3 and daily/comprehensive group. It rejects an effective-model mismatch between matched v2/v3 cells, leaving the source receipts available for diagnosis and a clean rerun. The collected batch stores a compact receipt index and hashes; raw transcripts remain in the private receipt directory for adjudication. Cost in this batch is clearly labeled a pricing-table estimate; only a host-billed amount may populate `EvalResult`'s host-cost field. The producer host needs Git, the chosen authenticated CLI, and reviewed gstack executables and catalogs without their skill source tree. Its CSO instruction bytes come only from the cell payload, never the installed skill or installed carved sections. Comprehensive cells additionally need a local Docker daemon, the exact qualified helper/runtime catalog pair, public dependency inputs or verified cache hits, and the authenticated out-of-process assertion witness. The runtime image workflow currently emits native staging evidence with private accuracy, held-out repair, canary, and watchdog gates still pending. Such evidence can measure setup failures, but it cannot support a release-qualifying tested repair until those gates and catalog promotion are complete. ## Private oracle boundary `test/helpers/cso-eval-oracles.ts` belongs to the **trusted evaluator**. It contains the legitimate controls, initial security assertions, alternate held-out assertions, and non-vacuous repair judge. It is never copied into materialized source. The fixed alternative, expected manifest, and this evaluator module must remain unavailable to producing agents throughout their sessions. A separate directory alone does not restrict a host agent's file access. The evaluation host must enforce this boundary with scoped tools or an isolated producer filesystem and test it with the `held-out-oracle-visibility` gate. If it cannot do that, report the gate as `not_run` and do not claim held-out qualification. Product execution containment does not automatically isolate the hosted producing agent. Only trusted runtime observations enter `judgeRepair`. Boot and legitimate controls must pass on the original; the exact security assertion must fail for the intended reason. The patched app must pass that assertion, all held-out assertions, controls, and existing tests with unchanged verification inputs. Independent root-cause/feature review is mandatory; boundary-replacing mocks or disabled functionality fail certification. A producer's `tested` or `runtime_tested` label is never an oracle result. `collect` preserves producer claims and performance receipts; it does not turn them into `EvalResult`. The independent evaluator must inspect the retained CSO report/bundle, execute the private assertions, and write finding judgments and workflow outcomes. This separation is deliberate: automatically copying agent claims into the trusted result contract would fabricate verification. ## Scoring and release gates ```sh bun scripts/cso-eval.ts score .context/cso-eval-matrix.json \ .context/cso-eval-producer-batch.json .context/trusted-cso-results.json \ .context/cso-qualification.json ``` The result contract is `EvalResult` in `scripts/cso-eval.ts`. The independent evaluator supplies finding judgments, the matching `producerReceiptHash`, and hashes of private evidence. The release scoring command requires a complete producer batch, checks its integrity and effective-model parity, and rejects any trusted result that is not bound to that cell's receipt. Recheck success additionally requires a distinct current-source observation and its source hash. Correct alternative patches need not match the reference fix's text. Keep actual transcripts and reports in private evaluation state; do not commit them beside the fixtures. Each metric includes its numerator and denominator. Duplicate findings cannot inflate true positives. Fixed variants contribute false positives. Missing and setup-blocked supported scenarios remain misses in recall and workflow rates. Hypotheses do not become supported discoveries; v2 legacy review evidence cannot become v3 reproduced, runtime-tested, or tested evidence. Unknown usage remains unknown, and wall-clock budgets are not represented as model-spend caps. The scorer checks 95% daily precision, 80% comprehensive high/critical recall, no aggregate high/critical recall regression against v2, all core cold starts, zero falsely certified runtime-tested repairs, and a correct held-out repair in every application stack. It also requires complete matched results, mandatory reports, containment/canary receipts, and enforced oracle separation. Missing evaluations return `unmeasured` or `partial`, never passing release gates. The free accounting tests use clearly labeled synthetic observations to test the scorer. They are **not measured agent performance**. No paid comparison, fixture cold-start qualification, or containment qualification is claimed by this corpus.