CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-16 01:45:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(ci): run the ship-docsync gate E2E in the evals matrix + silent-skip tripwire

Browse Source 
The evals.yml matrix is hand-enumerated and the Run step never exported
EVALS_TIER, so the new whole-file-gated ship-docsync E2E would have
self-skipped even with a row — a hollow green one layer deeper than the
documented rehomed-monolith incident. Add the e2e-ship-docsync row with a
row-level `tier: gate` property, exported as EVALS_TIER by the Run step
(empty = unset for every existing row: all readers are `=== '<tier>'` or
truthiness).

New free tripwire test/evals-workflow-matrix.test.ts ratchets the class:
matrix files must exist; gate-hosting files must have a row; whole-file-gated
matrix files must carry a matching row tier; and the burn-down lists enforce
their own cleanup. It enumerates the PRE-EXISTING holes found while wiring
this (8 gate-hosting files with no row; codex/gemini rows running zero tests;
the pty-plan-smoke row hollow since its files adopted describeE2ETier) —
tracked in TODOS as the CI gate-lane hollow-coverage burn-down.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-26 17:56:43 +00:00
co-authored by Claude Fable 5
parent 65b9cff299
commit 8f18682880
3 changed files with 241 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TODOS.md
+28
View File
@@ -2378,6 +2378,34 @@ pin) and `test/skill-e2e-ship-docsync.test.ts` (dispatch E2E, gate tier).
**Priority:** P3
**Depends on:** None
### CI gate-lane hollow-coverage burn-down (evals.yml matrix)
**What:** `test/evals-workflow-matrix.test.ts` (added v1.70.1.0) ratchets two
pre-existing CI coverage holes; burn them down. (1) Eight gate-hosting test
files have no `evals.yml` matrix row, so CI never runs them
(`KNOWN_MATRIX_GAPS` in the test enumerates them — notably the plan-mode and
finding-floor smokes and the AUQ format-compliance gate). (2) Four matrix rows
point at whole-file tier-gated files but set no row `tier:` property, so with
`EVALS_TIER` unexported those suites self-skip: `codex-e2e`/`gemini-e2e` run
ZERO tests and report green on every PR (vestigial rows; the periodic cron
lane owns them — consider deleting the rows), and `e2e-pty-plan-smoke` spends
~7 min on setup then skips every describe (hollow-green since the files
adopted `describeE2ETier('gate')` — set `tier: gate` on the row to reactivate,
after confirming the smokes still pass).
**Why:** "Gate tier blocks merge" is silently false for these files. Each fix
is a deliberate cost/flake decision (activating paid suites on every PR), so
they're enumerated instead of drive-by-fixed. The mechanism already exists:
per-row `tier:` property, exported as `EVALS_TIER` by the Run step.
**Context:** Found 2026-08-26 on PR #2700 while adding the `ship-docsync` row.
Fix = add/adjust the matrix row, then DELETE the corresponding burn-down entry
(the tripwire fails on stale entries, so cleanup is enforced).
**Effort:** S per file (mechanical) + one burn-in run each to confirm green
**Priority:** P2
**Depends on:** None
### Periodic paid-test shard census is one ungated file from the detach-timeout floor
**What:** The periodic tier's shard census is 67 files — one ungated slot below