Windows has no execute bit, so asserting one on the sandbox CLI copy failed both
Windows jobs while every other platform passed. The shim is launched through the
interpreter there anyway, so the assertion is POSIX-only.
The subprocess env is now overlaid on the real one rather than replacing it.
That was not the failure -- the hand-built env happened to work -- but a PATH of
`/usr/bin:/bin` and no SystemRoot is not something to leave in a test that CI
runs on Windows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The packaging tests a catalog runs say nothing about content, so the skill's
prose drifted from the CLI exactly the way an unguarded seam does. These tests
read the live Click tree: mark keys, flag choices, batch modes, pipeline
profiles, vendor cohorts, exit codes and extras must match what the skill says.
Guard the SET, not the instances. Per-flag comparisons only check what the skill
already documents, so `--vendor`, `--pipeline auto`, and then the whole
`classify` and `verify-openai-synthid` commands each shipped with a green suite.
`test_every_cli_command_is_named_somewhere_in_the_skill` closes that.
None of it proves the only thing a skill is for: that a model reaches for it and
then obeys it. `scripts/skill_eval.py` builds a throwaway project with the skill
installed and a recording stand-in for the CLI on PATH, runs a headless agent on
one user prompt, and grades the recorded command trace. Grading is mechanical --
"did it run `all` where the probe said nothing is written" is a fact in the
trace, and "did it name an output file that is not on disk" catches a fabricated
result without asking a model to judge itself. Agents are nondeterministic, so
the report is a pass rate per check over repeats, never one verdict.
The harness is tested too: the stand-in's exit codes are compared against the
real CLI where the state is reproducible locally, the grader runs over hand-built
traces, and every case is checked for internal consistency, because a scenario
whose `identify` names a mark its `visible` cannot find measures the case rather
than the skill.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
