CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-08-22 14:07:14 +02:00
Code Issues Packages Projects Releases Wiki Activity

evals: map-diff selection — a touchfiles-data edit runs only what changed

Browse Source 
Editing the eval dep-list data no longer forces the full ~$38 /
30-45min suite (measured on 21.9% of recent commits). When
touchfiles-data.ts is in the diff, selection now evaluates the BASE
version (git show -> mkdtemp -> spawnSync bun child printing the four
maps as JSON — sync because e2e-helpers selects at module scope) and
JSON-diffs per key: added entries, edited dep lists, and tier flips are
selected; keys removed from all maps are reported, never silently
dropped; a GLOBAL_TOUCHFILES edit still runs everything.

FAIL-CLOSED with named causes: missing-base-ref, git-show-failed,
import-failed, shape-mismatch each degrade to run-all and print
'selection: global — touchfiles-data changed (<cause>)' (D9 — silently
expensive beats silently wrong, but never silently). eval:select prints
'selected N of M, reason: ...' + removed tests; --base scopes the
map-diff too.

The temporary conservative GLOBAL entry for touchfiles-data.ts is gone —
its changes route through the map-diff. 23 new free tests: pure-core
fixtures, selectTests wiring incl. a poison-injection guard, and a temp
git repo exercising every fail-closed cause end-to-end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-15 08:30:09 -07:00
co-authored by Claude Fable 5
parent f23b2263bc
commit 820afa26e8
5 changed files with 646 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/eval-select.ts
+11 -4
View File
@@ -38,8 +38,11 @@ if (changedFiles.length === 0) {
process.exit(0);
}
const e2eSelection = selectTests(changedFiles, E2E_TOUCHFILES, GLOBAL_TOUCHFILES);
const llmSelection = selectTests(changedFiles, LLM_JUDGE_TOUCHFILES, GLOBAL_TOUCHFILES);
// baseRef/cwd scope the map-diff path (used when touchfiles-data.ts changed)
// to the same base this script diffed against — including a --base override.
const selectOpts = { baseRef: baseBranch, cwd: ROOT };
const e2eSelection = selectTests(changedFiles, E2E_TOUCHFILES, GLOBAL_TOUCHFILES, selectOpts);
const llmSelection = selectTests(changedFiles, LLM_JUDGE_TOUCHFILES, GLOBAL_TOUCHFILES, selectOpts);
if (jsonMode) {
console.log(JSON.stringify({
@@ -49,6 +52,7 @@ if (jsonMode) {
selected: e2eSelection.selected,
skipped: e2eSelection.skipped,
reason: e2eSelection.reason,
removed_tests: e2eSelection.removedTests ?? [],
count: `${e2eSelection.selected.length}/${Object.keys(E2E_TOUCHFILES).length}`,
},
llm_judge: {
@@ -63,7 +67,10 @@ if (jsonMode) {
console.log(`Changed files: ${changedFiles.length}`);
console.log();
console.log(`E2E (${e2eSelection.reason}): ${e2eSelection.selected.length}/${Object.keys(E2E_TOUCHFILES).length} tests`);
console.log(`E2E: selected ${e2eSelection.selected.length} of ${Object.keys(E2E_TOUCHFILES).length}, reason: ${e2eSelection.reason}`);
if (e2eSelection.removedTests && e2eSelection.removedTests.length > 0) {
console.log(` Removed from maps (reported, not selected): ${e2eSelection.removedTests.join(', ')}`);
}
if (e2eSelection.selected.length > 0 && e2eSelection.selected.length < Object.keys(E2E_TOUCHFILES).length) {
console.log(` Selected: ${e2eSelection.selected.join(', ')}`);
console.log(` Skipped: ${e2eSelection.skipped.join(', ')}`);
@@ -74,7 +81,7 @@ if (jsonMode) {
}
console.log();
console.log(`LLM-judge (${llmSelection.reason}): ${llmSelection.selected.length}/${Object.keys(LLM_JUDGE_TOUCHFILES).length} tests`);
console.log(`LLM-judge: selected ${llmSelection.selected.length} of ${Object.keys(LLM_JUDGE_TOUCHFILES).length}, reason: ${llmSelection.reason}`);
if (llmSelection.selected.length > 0 && llmSelection.selected.length < Object.keys(LLM_JUDGE_TOUCHFILES).length) {
console.log(` Selected: ${llmSelection.selected.join(', ')}`);
console.log(` Skipped: ${llmSelection.skipped.join(', ')}`);