CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-12 07:59:02 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat(test): e2e-gate — one tier-gate implementation, side-effect-free, with the trap pinned

Browse Source 
The EVALS/EVALS_TIER gate was copy-pasted into ~40 test files and had drifted
into six different predicates — the drift that made 'eval:bg:all runs
everything' silently false. test/helpers/e2e-gate.ts owns the semantics now:
describeE2ETier(tier) + e2eTierEnabled(tier), env read at call time, zero
side effects (the existing e2e-helpers module runs a ~30s claude ping at
import under EVALS=1, so the gate lives in its own module; purity is pinned
by tests that scan imports and comment-stripped source).

The unit matrix pins all four env combos — including EVALS=1 with EVALS_TIER
unset -> SKIP, the exact trap that made eval:bg:all a non-run. The
tier-alignment tripwire gains a second regex for the helper shape (old shape
still detected — stragglers can't hide), and the sharded paid runner's
PRE-SPAWN tier classifier learns the helper shape too: without that, every
gate-sharded run would have spawned all 28 periodic shards just to skip them,
each paying the e2e-helpers import ping (~15 min of dead wall clock in the
CI-blocking lane). Verified: gate runs exclude the 29 periodic files,
periodic excludes the 8 gate files — identical to pre-migration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-14 21:12:23 -07:00
co-authored by Claude Fable 5
parent bcfdd7da4a
commit e23a3ae3b1
5 changed files with 184 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/test-paid-shards.ts
+9 -5
View File
@@ -90,8 +90,11 @@ export interface TierClassification {
*
* Exclusion is the dangerous direction (a wrongly-skipped gate test is exactly
* the invisible-non-execution bug this runner exists to kill), so the only
* exclusion evidence accepted is an explicit whole-file `EVALS_TIER === '<other>'`
* guard. Inferring a file's tier from which E2E_TIERS names appear in its source
* exclusion evidence accepted is an explicit whole-file tier guard: either the
* raw `EVALS_TIER === '<other>'` predicate or the consolidated helper form
* `describeE2ETier('<other>')` / `e2eTierEnabled('<other>')` from
* test/helpers/e2e-gate.ts (same semantics, read from env at module load).
* Inferring a file's tier from which E2E_TIERS names appear in its source
* is guesswork that silently drops real work: short keys like 'retro' match
* unrelated strings, and LLM-judge tests are keyed off LLM_JUDGE_TOUCHFILES and
* carry no E2E_TIERS name at all. Everything without an explicit other-tier
@@ -100,10 +103,11 @@ export interface TierClassification {
export function classifyPaidTestFile(source: string, tier: PaidTier): TierClassification {
const other: PaidTier = tier === 'gate' ? 'periodic' : 'gate';
const declares = (candidate: PaidTier) =>
new RegExp(`EVALS_TIER\\s*===\\s*['"\`]${candidate}['"\`]`).test(source);
new RegExp(`EVALS_TIER\\s*===\\s*['"\`]${candidate}['"\`]`).test(source) ||
new RegExp(`\\b(?:describeE2ETier|e2eTierEnabled)\\(\\s*['"\`]${candidate}['"\`]`).test(source);
if (declares(tier)) return { included: true, reason: `declares EVALS_TIER === '${tier}'` };
if (declares(other)) return { included: false, reason: `declares EVALS_TIER === '${other}' only` };
if (declares(tier)) return { included: true, reason: `declares tier '${tier}'` };
if (declares(other)) return { included: false, reason: `declares tier '${other}' only` };
return { included: true, reason: 'no whole-file tier guard — runtime E2E_TIERS filter decides' };
}