fix(evals): the Codex removed-rule arm really removes the rule from installed skills (GST-840) (#3816)

* fix(evals): Codex removed-rule arm strips the rule from the installed skills (GST-840)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(evals): copy linked directories inside a skill before stripping the rule; record the harness decision (GST-840)

/ship review found readdirSync({ recursive: true }) follows linked
subdirectories, so a rule file under a linked directory inside a real
skill directory would have been deleted and rewritten in the checkout.
Every linked directory on the way to a rule file is now copied first,
with a failing-first test row. The harness-version manifest records
codex-host-runner.ts as non-behavioral: only the manual removed arm
on the Codex host changes.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep the Codex safety-arm test off Windows, and spawn its probe with bunIsolationArgs (GST-840)

windows-free-shard 3 (run 41336347706) failed every case: native Bun on
Windows cannot open --config=/dev/null, and the test builds a symlinked
~/.codex/skills install that needs Developer Mode the runners lack.
./setup --host codex copies instead of linking on Windows, so the strip
only matters on POSIX. The probe now uses bunIsolationArgs().

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-05 00:29:18 +00:00
1 parent fc5e32b712
commit 91ddd3bdb1
5 files changed
+133 -1

No files matched your search

@@ -1 +1 @@
{"blob":"bff225017e235f0be6ece29f04971ac04b95b544","since":6,"nonBehavioral":"Claude-host verdicts are unchanged: the Codex host (EVALS_HOST=codex, codex-host-runner) runs only when selected and records its own per-model series; the Codex runner refactor (execCodex extraction, onSpawn hook) keeps its events and verdicts and adds token cost accounting; host fields on the plan/report are absent for claude"}
{"blob":"4596f41514cf87e2fe5f1791c00ea4991cba548c","since":6,"nonBehavioral":"Only GSTACK_SAFETY_ARM=removed runs on the Codex host change (GST-840): the installed ~/.codex/skills copies lose the registered rule paragraph, as the removed-arm fixture already did, so the removed control measures a missing rule. That arm is a manual comparison arm; gate and CI panels run the default rule arm, whose installed skills, prompts and verdicts are unchanged, and Claude-host runs never load this runner"}
+2
View File
@@ -57,6 +57,8 @@ export const KNOWN_WINDOWS_INCOMPATIBLE: Array<{ file: string; reason: string }>
{ file: 'test/shared-libs-index-flags-readonly.test.ts', reason: 'its assume-unchanged replay seeds the review through test/helpers/shared-libs-path-fixture.ts, which launches bin/gstack-review-log (a bash script) directly; windows-latest refuses it as not found in $PATH (windows-free-shard 5, run 40727825859)' },
{ file: 'test/credential-rejection.test.ts', reason: 'replays run 39696775546 through the --report and --rerun-plan CLI of the paid lane, which runs only on Linux CI; on windows-latest that CLI exits 1 with EINVAL in open() (windows-free-shard run 39982914852)' },
// Field list 2, W11: the review-voice install is a POSIX launcher contract, exercised end to end on Linux and macOS.
// GST-840: the Codex host runtime links each installed skill into the checkout, as ./setup --host codex does on POSIX.
{ file: 'test/codex-host-safety-arm.test.ts', reason: 'builds a symlinked ~/.codex/skills install (a linked skill folder and a linked folder inside a real one), which needs Developer Mode that windows-latest runners lack; ./setup --host codex copies instead of linking on Windows, so the removed-arm strip it tests only matters on POSIX (windows-free-shard 3, run 41336347706)' },
{ file: 'test/capy-review-voice.test.ts', reason: 'copies the checkout with tar -C into the temp root (GNU tar reads a D: path as a remote host) and launches the bash --for review-voice installer with a colon-joined GSTACK_LAUNCH PATH' },
// C-pregate. Since fix wave PR 5 (item 10b) test/helpers/pregate-fixture.ts and restamp-fixture.ts
// runBin launch bins through lib/script-invocation.ts, so gstack-pregate runs on windows-latest