gstack/test/fixtures/cso-eval/README.md

# CSO vulnerable/fixed evaluation corpus

This corpus contains **40 immutable source pairs**: ten each for Node, Bun,
Python, and Rails. `manifest.json` pins both source hashes, expected root cause,
location, severity, and evaluation eligibility. `materialize.ts` deterministically
creates either member of each pair. Changing source requires an explicit corpus
manifest update; the loader rejects silent drift.

The ten families are SQL injection, command injection, path traversal, SSRF,
object authorization, tenant isolation, HTML injection, open redirects, mass
assignment, and resource exhaustion. Every app exposes `/action` and `/health` on
loopback port 8000. The SSRF fixtures additionally create a disposable loopback
status service on port 8001. Exhaustion assertions request 250 small records to
prove missing admission; they do not attempt to exhaust the evaluator.

Node uses standard modules, including `node:sqlite`; Bun uses `bun:sqlite`; Python
uses its standard library. Rails fixtures contain an actual Rails application
and controller, with native SQLite and Puma dependencies. Their common lock was
generated by Ruby 3.2.8 / Bundler 2.6.7 from public registry metadata only. The
exact command, lock hash, and 68 public archive hashes are recorded in
`rails-lock-provenance.json`. No fixture application was run to generate this
lock. The recorded archive hashes must be checked by runtime qualification; a
successful resolver run does not establish cold-start support.

Authentication is an explicit fixture precondition: the test adapter establishes
member-1 in tenant-a. The assessment scope is the selected endpoint's behavior.
This prevents an intentionally constant test identity from being mistaken for a
production authentication design. Application README files state legitimate
behavior; they never include attack payloads or expected finding labels.

## Preparing matched evaluations

From the repository root, export the v2 generated skill tree and build a
canonical portable payload for each version before pinning an exact model ID:

```sh
mkdir -p .context/cso-v2-source
git archive origin/main cso/SKILL.md cso/sections | \
  tar -x -C .context/cso-v2-source --strip-components=1
bun scripts/cso-eval.ts payload --version v2 \
  --skill-dir .context/cso-v2-source --output .context/cso-v2.payload.md
bun scripts/cso-eval.ts payload --version v3 \
  --skill-dir cso --output .context/cso-v3.payload.md
bun scripts/cso-eval.ts matrix \
  --model EXACT_MODEL_ID --host codex \
  --v2-payload .context/cso-v2.payload.md \
  --v3-payload .context/cso-v3.payload.md \
  --output .context/cso-eval-matrix.json
bun scripts/cso-eval.ts materialize node-sql-injection vulnerable .context/cso-eval-app
```

The full matrix contains 960 cells: 40 pairs × two source variants × two modes ×
two skill versions × three repetitions. Each matched pair uses identical source,
model, host, and per-mode wall-clock budget. Daily runs have 600 seconds;
comprehensive runs have 1800 seconds. Each portable payload embeds the exact
generated `SKILL.md`, `sections/manifest.json`, and every generated section
listed by that manifest. The payload builder rejects missing or unlisted
generated sections and a mismatched major version. The matrix hashes the whole
payload, so a change to any section byte changes the pinned skill identity. The
commands above make no model calls and execute no fixture application.

Prepare consumable one-cell jobs and compile the generic producer runner:

```sh
bun build --compile \
  --no-compile-autoload-dotenv \
  --no-compile-autoload-bunfig \
  --no-compile-autoload-tsconfig \
  --no-compile-autoload-package-json \
  scripts/cso-eval-producer.ts --outfile .context/cso-eval-producer
bun scripts/cso-eval.ts prepare .context/cso-eval-matrix.json \
  --v2-payload .context/cso-v2.payload.md \
  --v3-payload .context/cso-v3.payload.md \
  --output .context/cso-eval-jobs
```

### Trusted five-artifact producer unit

The `.context/cso-eval-producer` command above creates a preparation artifact;
ordinary `bun run build` does not distribute the private producer. For a paid
run, start from one clean checkout and build a five-artifact staging unit in one
session:

```sh
stage="$(mktemp -d)"
bun run build:cso
bun build --compile \
  --no-compile-autoload-dotenv \
  --no-compile-autoload-bunfig \
  --no-compile-autoload-tsconfig \
  --no-compile-autoload-package-json \
  scripts/cso-eval-producer.ts --outfile "$stage/cso-eval-producer"
install -m 0555 bin/gstack-cso-launcher bin/gstack-cso-core \
  bin/gstack-cso-watchdog "$stage/"
install -m 0444 bin/.gstack-cso-generation "$stage/.gstack-cso-generation"
(cd "$stage" && (sha256sum cso-eval-producer gstack-cso-launcher \
  gstack-cso-core gstack-cso-watchdog .gstack-cso-generation 2>/dev/null || \
  shasum -a 256 cso-eval-producer gstack-cso-launcher gstack-cso-core \
  gstack-cso-watchdog .gstack-cso-generation))
sudo install -d -o root -g root -m 0755 /opt/gstack-cso-producer
sudo install -o root -g root -m 0555 "$stage/cso-eval-producer" \
  "$stage/gstack-cso-launcher" "$stage/gstack-cso-core" \
  "$stage/gstack-cso-watchdog" /opt/gstack-cso-producer/
sudo install -o root -g root -m 0444 "$stage/.gstack-cso-generation" \
  /opt/gstack-cso-producer/.gstack-cso-generation
```

Run `/opt/gstack-cso-producer/cso-eval-producer` as an unprivileged account.
Its startup check rejects missing, linked, writable, or non-root-owned members
and a writable or linked ancestor directory. Every receipt records the SHA-256
and byte length of all five artifacts in one installation identity; collection
rejects a batch if those identities differ. Preserve the printed hashes with
the release evidence and move or replace the complete unit together.

`prepare` creates a trusted `schedule.json` and 960 independent directories.
It initializes each generated application as a one-commit Git repository. Each
job has exactly one source variant and one versioned, complete instruction
payload. Root-skill pointers resolve only to sections embedded in that payload;
the producer must never load CSO instructions from `~/.claude`, another host
install, this source checkout, or the other version's payload. The generic runner
is compiled separately and contains no corpus generator, fixed alternative,
case oracle, or schedule. These preparation commands still make zero model calls.

Run each job on a fresh producer filesystem that contains only that job, the
compiled runner, the authenticated provider CLI, and the reviewed executable
gstack helper/runtime files. Do not install or copy any CSO `SKILL.md`, carved
section, generated skill tree, schedule, this repository checkout,
sibling jobs, fixed alternatives, or evaluator code to that filesystem. The
producer runner reads its opaque control file into memory and deletes it before
the agent process starts. That file never names the case, variant, version, or
repetition: it is `{schemaVersion: 2, cellRef, skill, source, execution}`, where
`cellRef` is the matrix cell id and `execution` holds only the mode, model, host,
budget, source and skill hashes, and platform. The receipt binds `cellRef` and
`inputHash`, and `collect` rebuilds each cell from the matrix, so neither file
tells an agent with root on its cell VM which answer it is graded against. Copy the selected hash-named directory as the literal path
`/producer/job`; `/producer` must contain only `job`. Put the runner in a system
tool directory and use a separate receipt mount. The runner rejects the full
prepared batch layout, extra files beside `job`, a reused job with prior state,
and receipt paths under `/producer`. The dedicated VM/container must ensure
other host paths do not contain evaluator inputs; a local directory layout does
not constrain a tool-using agent's absolute filesystem access.

The receipt must be written outside the isolated producer root:

```sh
mkdir -m 700 /receipts
CSO_EVAL_PAID=1 /opt/gstack-cso-producer/cso-eval-producer run \
  /producer/job/producer-input.json /receipts/CELL_ID.json --execute-paid
```

Both the environment variable and flag are required because this is the only
command in the workflow that makes a paid model call. It reuses the repository's
Claude, Codex, and Gemini provider adapters and starts a fresh CLI process for
each cell. The requested model, normalized effective model, identity source,
timeout, output, token counts, pricing-table cost estimate, tool-call count, and
latency are bound into a hashed receipt. `provider_reported` means the CLI
resolved a different concrete ID; `requested_pin` means the adapter returned
the exact requested ID or had to fall back to that exact CLI pin. Use a concrete
model ID rather than an alias. The current adapters return only completed runs,
so first-useful-result latency is explicitly unmeasured rather than copied from
total latency. If a job is interrupted after its input is consumed, restore it
from the trusted prepared copy; never synthesize a receipt.

After returning receipts to the trusted evaluator, collect them against the
matrix and schedule:

```sh
bun scripts/cso-eval.ts collect .context/cso-eval-matrix.json \
  .context/cso-eval-jobs/schedule.json .context/cso-eval-receipts \
  .context/cso-eval-producer-batch.json
```

Collection makes no model calls. It rejects unknown, duplicate, changed, or
wrong-input receipts and reports scheduled/submitted/missing denominators for
every v2/v3 and daily/comprehensive group. It rejects an effective-model mismatch
between matched v2/v3 cells, leaving the source receipts available for diagnosis
and a clean rerun. The collected batch stores a compact receipt index and hashes;
raw transcripts remain in the private receipt directory for adjudication. Cost
in this batch is clearly labeled a
pricing-table estimate; only a host-billed amount may populate `EvalResult`'s
host-cost field.

The producer host needs Git, the chosen authenticated CLI, and reviewed gstack
executables and catalogs without their skill source tree. Its CSO instruction
bytes come only from the cell payload, never the installed skill or installed
carved sections. Comprehensive cells additionally need a local Docker daemon,
the exact qualified helper/runtime catalog pair, public dependency inputs or
verified cache hits, and the authenticated out-of-process assertion witness. The runtime
image workflow currently emits native staging evidence with private accuracy,
held-out repair, canary, and watchdog gates still pending. Such evidence can
measure setup failures, but it cannot support a release-qualifying tested repair
until those gates and catalog promotion are complete.

## Private oracle boundary

`test/helpers/cso-eval-oracles.ts` belongs to the **trusted evaluator**. It contains
the legitimate controls, initial security assertions, alternate held-out
assertions, and non-vacuous repair judge. It is never copied into materialized
source. The fixed alternative, expected manifest, and this evaluator module must
remain unavailable to producing agents throughout their sessions.

A separate directory alone does not restrict a host agent's file access. The
evaluation host must enforce this boundary with scoped tools or an isolated
producer filesystem and test it with the `held-out-oracle-visibility` gate. If it
cannot do that, report the gate as `not_run` and do not claim held-out
qualification. Product execution containment does not automatically isolate the
hosted producing agent.

Only trusted runtime observations enter `judgeRepair`. Boot and legitimate
controls must pass on the original; the exact security assertion must fail for
the intended reason. The patched app must pass that assertion, all held-out
assertions, controls, and existing tests with unchanged verification inputs.
Independent root-cause/feature review is mandatory; boundary-replacing mocks or
disabled functionality fail certification. A producer's `tested` or
`runtime_tested` label is never an oracle result.

`collect` preserves producer claims and performance receipts; it does not turn
them into `EvalResult`. The independent evaluator must inspect the retained CSO
report/bundle, execute the private assertions, and write finding judgments and
workflow outcomes. This separation is deliberate: automatically copying agent
claims into the trusted result contract would fabricate verification.

## Scoring and release gates

```sh
bun scripts/cso-eval.ts score .context/cso-eval-matrix.json \
  .context/cso-eval-producer-batch.json .context/trusted-cso-results.json \
  .context/cso-qualification.json
```

The result contract is `EvalResult` in `scripts/cso-eval.ts`. The independent
evaluator supplies finding judgments, the matching `producerReceiptHash`, and
hashes of private evidence. The release scoring command requires a complete
producer batch, checks its integrity and effective-model parity, and rejects any
trusted result that is not bound to that cell's receipt. Recheck
success additionally requires a distinct current-source observation and its
source hash. Correct alternative patches need not match the reference fix's
text. Keep actual transcripts and reports in private evaluation state; do not
commit them beside the fixtures.

Each metric includes its numerator and denominator. Duplicate findings cannot
inflate true positives. Fixed variants contribute false positives. Missing and
setup-blocked supported scenarios remain misses in recall and workflow rates.
Hypotheses do not become supported discoveries; v2 legacy review evidence cannot
become v3 reproduced, runtime-tested, or tested evidence. Unknown usage remains
unknown, and wall-clock budgets are not represented as model-spend caps.

The scorer checks 95% daily precision, 80% comprehensive high/critical recall,
no aggregate high/critical recall regression against v2, all core cold starts,
zero falsely certified runtime-tested repairs (claims on declared instrumentation
count in neither direction; an incidental claim needs a passing replay of its
signed bundle), and a correct held-out repair in
every application stack. It also requires complete matched results, mandatory
reports, containment/canary receipts, and enforced oracle separation. Missing
evaluations return
`unmeasured` or `partial`, never passing release gates.

## Release profile, private corpora, and two architectures

The matrix above is the `full` profile: matched v2/v3 cells, three repetitions,
and one producer installation. This public corpus is a development set, because
its oracles are public. A private evaluator scores a held-out corpus with the
same layout through the same code:

- **Injected corpus.** `defineEvalCorpus(manifest, sourceFiles)` accepts a
  manifest with this `manifest.json` layout and a deterministic source generator,
  checks every variant's source hash, and refuses the public corpus version.
  `prepareEvalJobs(..., corpus)` and `materializeCase(..., corpus)` materialize
  from it; `createEvalMatrix`, `validateMatrix`, `collectProducerReceipts`,
  `scoreCollectedEval`, and `createEvalBaseline` take its manifest. Its oracles
  use the `PrivateOracle` shape from `test/helpers/cso-eval-oracles.ts` and stay
  in the private evaluator.
- **`release` profile.** `createEvalMatrix({ profile: 'release', skillHashes: { v3 },
  platforms: { daily, comprehensive } })` builds v3-only cells with one repetition.
  Daily cells run on one platform; each comprehensive case is fixed to
  `linux/amd64` or `linux/arm64`. Every cell carries its `platform`, and the
  producer refuses a cell whose platform differs from its host.
- **Per-platform identities.** One batch may hold both architectures. Each
  platform must have exactly one producer installation and one provider binary;
  platforms must use different installations with identical embedded catalogs
  and identical provider family, policy revision, version, and arguments. All
  cells must report one effective model.
- **External v2 baseline.** A `baseline` profile matrix holds v2 comprehensive
  cells with the release platform assignment. `createEvalBaseline` turns its
  complete, fully adjudicated, receipt-bound results into an `EvalBaseline`
  record whose `baselineHash` covers the corpus hash, model, host, budget,
  platform assignment, v2 payload hash, producer batch, effective model, provider
  policy and version, and each case's outcome. A release matrix pins that hash
  when it is created; scoring then requires the exact record and the same
  effective model and provider policy. Without a pinned baseline the
  regression gate stays `unmeasured`.

A matrix, batch, or result from one corpus never validates against another:
the matrix binds the corpus hash, so a mixed-corpus batch is rejected. The
`collect`, `score`, and `baseline` commands accept `--corpus-manifest <file>`,
and `score` accepts `--baseline <file>`. The `full` profile's matrix bytes and
gates are unchanged.

The free accounting tests use clearly labeled synthetic observations to test the
scorer. They are **not measured agent performance**. No paid comparison, fixture
cold-start qualification, or containment qualification is claimed by this corpus.