test: derive paid touchfiles from each eval's static closure (E)

touchfiles.test.ts now checks, per key, that the paid file's static
test/helpers and test/fixtures closure (plus fixture paths it names in string
literals) is covered, and names the file, path, chain and key to fix when it is
not. Free *.test.ts files are no longer touchfiles, so editing a free replay
test stops selecting paid evals: 950 entries removed, 653 real closure paths
added. The hand-copied inventories go: periodic-fixture-selection,
fake-impeccable-touchfiles and 45 per-file selection examples. Selection for
the sample edits (plan-eng-review template, claude-pty-runner,
plan-count-fixture, gstack-config) loses no case under either profile.
CONTRIBUTING documents the rule and its lower bound.
This commit is contained in:
garrytan committed 2026-09-29 07:00:28 +00:00
1 parent 689ef3094b
commit 7cbb5e04e4
53 files changed
+627 -1873

No files matched your search

+10
View File
@@ -393,6 +393,16 @@ Each dimension is scored 1-5. Threshold: every dimension must score **≥ 4**. T
- Tests live in `test/skill-llm-eval.test.ts`
- Calls the Anthropic API directly (not `claude -p`), so it works from anywhere including inside Claude Code
### Paid-test touchfiles
`test/helpers/touchfiles-data.ts` maps each paid case to the files whose edits select it. Free
`*.test.ts` files are never listed: editing a free test does not run paid evals. `test/touchfiles.test.ts`
derives each paid file's static `test/helpers` / `test/fixtures` import closure, plus the fixture and helper
paths it names in string literals, and fails when that closure is not covered by the case's key. When it
fails, add the named path to the named key and check selection with
`bun run scripts/test-paid-shards.ts --tier gate --profile pr --list`. The rule is a lower bound: a fixture
path the test builds at runtime is not visible to it, so add such paths to the key by hand.
### CI
A GitHub Action (`.github/workflows/skill-docs.yml`) generates all hosts on pushes to main and on PRs, then rejects tracked differences and nonignored untracked output. Generation errors also fail the job. Optional ignored host caches are not compared against Git.