Commit Graph
2 Commits
Author SHA1 Message Date
Garry Tanandcapy-ai[bot] 91ddd3bdb1 fix(evals): the Codex removed-rule arm really removes the rule from installed skills (GST-840) (#3816)
* fix(evals): Codex removed-rule arm strips the rule from the installed skills (GST-840)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(evals): copy linked directories inside a skill before stripping the rule; record the harness decision (GST-840)

/ship review found readdirSync({ recursive: true }) follows linked
subdirectories, so a rule file under a linked directory inside a real
skill directory would have been deleted and rewritten in the checkout.
Every linked directory on the way to a rule file is now copied first,
with a failing-first test row. The harness-version manifest records
codex-host-runner.ts as non-behavioral: only the manual removed arm
on the Codex host changes.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep the Codex safety-arm test off Windows, and spawn its probe with bunIsolationArgs (GST-840)

windows-free-shard 3 (run 41336347706) failed every case: native Bun on
Windows cannot open --config=/dev/null, and the test builds a symlinked
~/.codex/skills install that needs Developer Mode the runners lack.
./setup --host codex copies instead of linking on Windows, so the strip
only matters on POSIX. The probe now uses bunIsolationArgs().

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-05 00:29:18 +00:00
Garry Tanandcapy-ai[bot] 9cf105ccff v1.91.149.0 feat(evals): Codex host for gate cases, sol weekly and Fable/Astra monthly census arms (GSTA-49 TM2) (#3292)
* feat(evals): Opus 5.5 is the paid gate's model under test; judge stays Fable (GSTA-49 TM1)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* v1.91.140.0 chore: stamp TM1 (Opus 5.5 model under test) and CHANGELOG

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(evals): Codex host adapter, real Codex cost, two-arm weekly and monthly census (GSTA-49 TM2)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: TM2 CHANGELOG under [Unreleased]; re-pin codex-host-runner in the harness manifest after the main merge

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ci: retrigger PR workflows (base changed to main after open; no pull_request run fired)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* v1.91.149.0 chore: stamp TM2 (Codex host, census arms) and CHANGELOG

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-19 18:26:20 +00:00