feat: granular touchfiles + 2-tier E2E test system (gate/periodic)

- Shrink GLOBAL_TOUCHFILES from 9 to 3 (only truly global deps)
- Move scoped deps (gen-skill-docs, llm-judge, test-server, worktree,
  codex/gemini session runners) into individual test entries
- Add E2E_TIERS map classifying each test as gate or periodic
- Replace EVALS_FAST with EVALS_TIER env var (gate/periodic)
- Add tier validation test (E2E_TIERS keys must match E2E_TOUCHFILES)
- CI runs only gate tests; periodic tests run weekly via cron
- Add evals-periodic.yml workflow (Monday 6 AM UTC + manual)
- Remove allow_failure flags (gate tests should be reliable)
- Add test:gate and test:periodic scripts, remove test:e2e:fast
This commit is contained in:
Garry Tan
2026-03-24 08:15:53 -07:00
parent 3501f5dd03
commit c4aa31c988
7 changed files with 340 additions and 52 deletions
+1 -2
View File
@@ -10,6 +10,7 @@ concurrency:
env:
IMAGE: ghcr.io/${{ github.repository }}/ci
EVALS_TIER: gate
jobs:
# Build Docker image with pre-baked toolchain (cached — only rebuilds on Dockerfile/lockfile change)
@@ -87,10 +88,8 @@ jobs:
file: test/skill-e2e-review.test.ts
- name: e2e-workflow
file: test/skill-e2e-workflow.test.ts
allow_failure: true # /ship + /setup-browser-cookies are env-dependent
- name: e2e-routing
file: test/skill-routing-e2e.test.ts
allow_failure: true # LLM routing is non-deterministic
- name: e2e-codex
file: test/codex-e2e.test.ts
- name: e2e-gemini