fix(ci): run the ship-docsync gate E2E in the evals matrix + silent-skip tripwire

The evals.yml matrix is hand-enumerated and the Run step never exported
EVALS_TIER, so the new whole-file-gated ship-docsync E2E would have
self-skipped even with a row — a hollow green one layer deeper than the
documented rehomed-monolith incident. Add the e2e-ship-docsync row with a
row-level `tier: gate` property, exported as EVALS_TIER by the Run step
(empty = unset for every existing row: all readers are `=== '<tier>'` or
truthiness).

New free tripwire test/evals-workflow-matrix.test.ts ratchets the class:
matrix files must exist; gate-hosting files must have a row; whole-file-gated
matrix files must carry a matching row tier; and the burn-down lists enforce
their own cleanup. It enumerates the PRE-EXISTING holes found while wiring
this (8 gate-hosting files with no row; codex/gemini rows running zero tests;
the pty-plan-smoke row hollow since its files adopted describeE2ETier) —
tracked in TODOS as the CI gate-lane hollow-coverage burn-down.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-26 17:56:43 +00:00
co-authored by Claude Fable 5
parent 65b9cff299
commit 8f18682880
3 changed files with 241 additions and 0 deletions
+17
View File
@@ -135,6 +135,17 @@ jobs:
file: test/skill-e2e-coverage-audit.test.ts
- name: e2e-triage
file: test/skill-e2e-triage.test.ts
# ship-docsync is whole-file tier-gated (describeE2ETier('gate') keeps
# it out of the periodic shard census), so its row MUST set tier: gate
# — without it the self-gate skips every test and the job reports a
# hollow green (the same silent-skip class as the rehomed monolith
# above, one layer deeper). The Run step exports EVALS_TIER from this
# property; rows without it keep EVALS_TIER empty (= unset: every
# reader is `=== '<tier>'` or truthiness). Enforced by
# test/evals-workflow-matrix.test.ts.
- name: e2e-ship-docsync
file: test/skill-e2e-ship-docsync.test.ts
tier: gate
- name: e2e-routing
file: test/skill-routing-e2e.test.ts
- name: e2e-codex
@@ -337,6 +348,12 @@ jobs:
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
EVALS_CONCURRENCY: "40"
PLAYWRIGHT_BROWSERS_PATH: /opt/playwright-browsers
# Per-row tier activation for whole-file-gated suites. Empty when the
# row declares no tier — every EVALS_TIER reader treats empty as unset
# (`=== '<tier>'` comparisons and the truthiness check in
# test/helpers/e2e-helpers.ts:70), so untiered rows are byte-for-byte
# unaffected.
EVALS_TIER: ${{ matrix.suite.tier || '' }}
run: EVALS=1 bun test --retry ${{ matrix.suite.retries || 1 }} --concurrent --max-concurrency 40 ${{ matrix.suite.file }}
- name: Upload eval results