CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-16 01:45:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: use agent capture budgets for deploy evals

Browse Source 
Multi-turn deploy and benchmark sessions were incorrectly limited to the single-call judge timeout. Use the existing capture tier and leave outer-test cleanup headroom, with a free policy regression test. Keep all behavioral assertions and frontier models unchanged.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
This commit is contained in:
Garry Tan
2026-09-09 04:41:23 +00:00
co-authored by OpenAI Codex
parent 1f678a5b81
commit eb4fd65ed3
2 changed files with 20 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/eval-budgets-policy.test.ts
+7
View File
@@ -40,6 +40,13 @@ describe('eval budget tiers', () => {
expect(Math.max(...values)).toBe(PTY_LONG_MS);
});
test('deploy workflow sessions use capture budgets, not single-call judge budgets', () => {
const source = fs.readFileSync(path.join(ROOT, 'test/skill-e2e-deploy.test.ts'), 'utf8');
expect(source).not.toContain('JUDGE_MS');
expect([...source.matchAll(/timeout:\s*CAPTURE_MS/g)]).toHaveLength(6);
expect([...source.matchAll(/\},\s*CAPTURE_LONG_MS\);/g)]).toHaveLength(6);
});
test('no paid-test timeout literal exceeds the ceiling tier', () => {
const out = spawnSync('git', ['ls-files', 'test/*.test.ts'], { cwd: ROOT, encoding: 'utf-8', timeout: 30_000 });
const files = out.stdout.split('\n').filter((f) => f && isPaidTestFile(f));