CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-13 00:19:03 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat(evals): env-driven lazy eval dir + shard-aware store and tooling

Browse Source 
Importing eval-store no longer spawns the gstack-slug subprocess: the
module-level DEFAULT_EVAL_DIR constant is now a memoized defaultEvalDir()
resolved at collector construction. Resolution order: explicit
constructor arg, then GSTACK_EVAL_DIR, then slug detection — so the
sharded paid runner can point each shard child at its own
<evalDir>/shards/<slug>/ dir with plain env, no --preload.

Runs collected under a shards/ subdir record their slug in the eval
JSON (EvalResult.shard). findPreviousRun scans one shards/<slug>/ level
and prefers same-slug priors, so each shard baselines against its own
history instead of whichever shard flushed last. eval:list,
eval:summary, and eval:compare enumerate the same one level of shard
subdirs; eval:compare's no-arg mode also stops picking an in-progress
accumulator as the after-run.

eval-watch stays flat (documented follow-up): it tails a single dir for
live progress and gains nothing from per-shard baselines until the
runner emits a merged stream.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e1f53f7d9c7fe6b65877d843f2e25bd2e2d12ffd)
This commit is contained in:
Garry Tan
2026-08-12 15:31:47 -07:00
parent ea1c2851e5
commit 9ff2b9b162
5 changed files with 185 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/eval-summary.ts
+4 -11
View File
@@ -6,20 +6,13 @@
*/
import * as fs from 'fs';
import * as path from 'path';
import * as os from 'os';
import type { EvalResult } from '../test/helpers/eval-store';
import { getProjectEvalDir } from '../test/helpers/eval-store';
import { getProjectEvalDir, listEvalJsonFiles } from '../test/helpers/eval-store';
const EVAL_DIR = getProjectEvalDir();
let files: string[];
try {
files = fs.readdirSync(EVAL_DIR).filter(f => f.endsWith('.json'));
} catch {
console.log('No eval runs yet. Run: EVALS=1 bun run test:evals');
process.exit(0);
}
// Flat dir plus one level of shards/<slug>/
const files = listEvalJsonFiles(EVAL_DIR);
if (files.length === 0) {
console.log('No eval runs yet. Run: EVALS=1 bun run test:evals');
@@ -30,7 +23,7 @@ if (files.length === 0) {
const results: EvalResult[] = [];
for (const file of files) {
try {
results.push(JSON.parse(fs.readFileSync(path.join(EVAL_DIR, file), 'utf-8')));
results.push(JSON.parse(fs.readFileSync(file, 'utf-8')));
} catch { continue; }
}