test(office-hours): attempt lifecycle fixture runs on fake timers, not wall-clock sleeps (GSTA-65)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-23 14:00:25 +00:00
1 parent b04cc35832
commit a6b80a62c6
2 files changed
+15 -3

No files matched your search

+10
View File
@@ -1,5 +1,15 @@
# Changelog
## [Unreleased]
**The office-hours attempt lifecycle test no longer races the wall clock.** Today's daily flaky-fix pass: in #3341's remote pregate, "recommendation near-success stays inside the shared work deadline" failed once with two judge starts where it expects one, then passed on the serial retry. Its fake runner and judge used real sleeps (220 ms plus 10 ms) against a mocked 300 ms deadline, so a loaded machine could push the first attempt past it. The sibling late-judge case had the same exposure from the other side.
### Tests
- **`test/office-hours-attempt.test.ts`**: the plan-format lifecycle subprocess now runs on Bun's fake timers, and the fake runner and judge advance them by their scenario durations instead of sleeping. The helper's own deadline (`Date.now()` and its `setTimeout`) moves only when the fixture says so, so near-success always finishes at 230 ms of 300 and late-judge always crosses the deadline, whatever the machine's load. With a 400 ms real stall injected into the fake runner, near-success and late-judge failed before the change and pass after it. The whole file passed 20/20 runs unloaded and 20/20 under 8 busy loops on 4 cores.
### Gate integrity
- No product code, budget, timeout, threshold or assertion changed: the mocked 300 ms capture budget, the 50 ms drain grace and the scenario durations are the same.
## [1.91.182.0] - 2026-10-23
**gstack installs on a project that pins Bun 1.4.4 again.** gbrain master moved its CI to Bun 1.4.4 (gbrain #7374, the Brotli CVE-2026-111497 fix), and gstack's Capy installer then stopped every gbrain session with `RUNTIME_PIN_CONFLICT` ("verified archives: 1.4.2 1.4.3 … pins exactly 1.4.4"), because it installs only Bun archives whose SHA-256 it carries. It now carries 1.4.4, so the same run installs the pinned Bun and ends `gstack: ok`.
+5 -3
View File
@@ -570,13 +570,14 @@ async function runFormatLifecycle(file: string, id: string, scenario: string) {
const facts = path.join(dir, 'events.jsonl');
const script = path.join(dir, 'format.test.ts');
fs.writeFileSync(script, `
import { mock, spyOn } from 'bun:test';
import { jest, mock, spyOn } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
const root = ${JSON.stringify(ROOT)}, selected = ${JSON.stringify(id)}, scenario = ${JSON.stringify(scenario)};
const event = value => fs.appendFileSync(${JSON.stringify(facts)}, JSON.stringify(value) + '\\n');
let attempt = 0;
let currentSignal;
const elapse = async ms => { jest.advanceTimersByTime(ms); };
mock.module(path.join(root, 'test/helpers/eval-budgets.ts'), () => ({ CAPTURE_MS: 300, CAPTURE_LONG_MS: 600 }));
mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({
SESSION_DRAIN_GRACE_MS: 50,
@@ -586,7 +587,7 @@ mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({
event({ kind: 'start', id, timeout: opts.timeout, maxTurns: opts.maxTurns, model: opts.model, cwd: opts.workingDirectory });
opts.signal?.addEventListener('abort', () => event({ kind: 'abort', id }), { once: true });
const timeout = scenario === 'both-timeout' || (scenario === 'recover' && id === 1);
await new Promise(resolve => setTimeout(resolve, timeout ? opts.timeout + 50 : scenario === 'late-judge' || scenario === 'near-success' ? 220 : 20));
await elapse(timeout ? opts.timeout + 50 : scenario === 'late-judge' || scenario === 'near-success' ? 220 : 20);
event({ kind: 'ready', id, fixtureExists: fs.existsSync(opts.workingDirectory) });
if (!timeout) {
const kind = selected.endsWith('-mode') || selected.endsWith('-kind');
@@ -607,7 +608,7 @@ mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({
const fakeJudgeRequest = async (body, options) => {
event({ kind: 'judge-start', id: attempt, sameSignal: options?.signal === currentSignal, model: body.model });
const id = attempt;
await new Promise(resolve => setTimeout(resolve, scenario === 'late-judge' ? 130 : 10));
await elapse(scenario === 'late-judge' ? 130 : 10);
event({ kind: 'judge-ready', id, aborted: options?.signal?.aborted ?? false });
return { stop_reason: 'end_turn', content: [{ type: 'text', text: JSON.stringify({ reason_substance: scenario === 'bad-score' ? 3 : 5, reasoning: 'fixture specific tradeoff' }) }] };
};
@@ -624,6 +625,7 @@ spyOn(EvalCollector.prototype, 'finalize').mockImplementation(function() {
event({ kind: 'finalized' });
return finalize.call(this);
});
jest.useFakeTimers();
await import(path.join(root, 'test', ${JSON.stringify(file)}));
`);
try {