mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-02 15:04:37 +01:00
v1.91.184.0 test(office-hours): attempt lifecycle fixture runs on fake timers, not wall-clock sleeps (GSTA-65) (#3356)
* test(office-hours): attempt lifecycle fixture runs on fake timers, not wall-clock sleeps (GSTA-65) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.184.0 test(office-hours): the attempt lifecycle fixture runs on fake timers (GSTA-65) (#3356) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
9c7b4c81ca
commit
8f37284a0b
5 files changed
+18
-6
No files matched your search
@@ -1,5 +1,15 @@
|
||||
# Changelog
|
||||
|
||||
## [1.91.184.0] - 2026-10-23
|
||||
|
||||
**The office-hours attempt lifecycle test no longer races the wall clock.** Today's daily flaky-fix pass: in #3341's remote pregate, "recommendation near-success stays inside the shared work deadline" failed once with two judge starts where it expects one, then passed on the serial retry. Its fake runner and judge used real sleeps (220 ms plus 10 ms) against a mocked 300 ms deadline, so a loaded machine could push the first attempt past it. The sibling late-judge case had the same exposure from the other side.
|
||||
|
||||
### Tests
|
||||
- **`test/office-hours-attempt.test.ts`**: the plan-format lifecycle subprocess now runs on Bun's fake timers, and the fake runner and judge advance them by their scenario durations instead of sleeping. The helper's own deadline (`Date.now()` and its `setTimeout`) moves only when the fixture says so, so near-success always finishes at 230 ms of 300 and late-judge always crosses the deadline, whatever the machine's load. With a 400 ms real stall injected into the fake runner, near-success and late-judge failed before the change and pass after it. The whole file passed 20/20 runs unloaded and 20/20 under 8 busy loops on 4 cores.
|
||||
|
||||
### Gate integrity
|
||||
- No product code, budget, timeout, threshold or assertion changed: the mocked 300 ms capture budget, the 50 ms drain grace and the scenario durations are the same.
|
||||
|
||||
## [1.91.183.0] - 2026-10-23
|
||||
|
||||
**A rejected API key no longer turns a gate into 131 assertion reds and 131 reruns.** Train gate run 39696775546 ran with a rejected `ANTHROPIC_API_KEY`. Every case failed on a 401 before the model ran, and the report counted them as assertion failures. Rule 16 then reran all 131 cases with the same key. A trial whose provider rejected the run's credentials is now INFRA: the gate never reruns it, the verdict counts it under INFRA, and its failure line reads `credentials rejected: Invalid API key · …`. A real assertion failure that only mentions a key keeps its class and is still rerun.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.91.183.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.91.184.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.91.183",
|
||||
"version": "1.91.184",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -570,13 +570,14 @@ async function runFormatLifecycle(file: string, id: string, scenario: string) {
|
||||
const facts = path.join(dir, 'events.jsonl');
|
||||
const script = path.join(dir, 'format.test.ts');
|
||||
fs.writeFileSync(script, `
|
||||
import { mock, spyOn } from 'bun:test';
|
||||
import { jest, mock, spyOn } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
const root = ${JSON.stringify(ROOT)}, selected = ${JSON.stringify(id)}, scenario = ${JSON.stringify(scenario)};
|
||||
const event = value => fs.appendFileSync(${JSON.stringify(facts)}, JSON.stringify(value) + '\\n');
|
||||
let attempt = 0;
|
||||
let currentSignal;
|
||||
const elapse = async ms => { jest.advanceTimersByTime(ms); };
|
||||
mock.module(path.join(root, 'test/helpers/eval-budgets.ts'), () => ({ CAPTURE_MS: 300, CAPTURE_LONG_MS: 600 }));
|
||||
mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({
|
||||
SESSION_DRAIN_GRACE_MS: 50,
|
||||
@@ -586,7 +587,7 @@ mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({
|
||||
event({ kind: 'start', id, timeout: opts.timeout, maxTurns: opts.maxTurns, model: opts.model, cwd: opts.workingDirectory });
|
||||
opts.signal?.addEventListener('abort', () => event({ kind: 'abort', id }), { once: true });
|
||||
const timeout = scenario === 'both-timeout' || (scenario === 'recover' && id === 1);
|
||||
await new Promise(resolve => setTimeout(resolve, timeout ? opts.timeout + 50 : scenario === 'late-judge' || scenario === 'near-success' ? 220 : 20));
|
||||
await elapse(timeout ? opts.timeout + 50 : scenario === 'late-judge' || scenario === 'near-success' ? 220 : 20);
|
||||
event({ kind: 'ready', id, fixtureExists: fs.existsSync(opts.workingDirectory) });
|
||||
if (!timeout) {
|
||||
const kind = selected.endsWith('-mode') || selected.endsWith('-kind');
|
||||
@@ -607,7 +608,7 @@ mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({
|
||||
const fakeJudgeRequest = async (body, options) => {
|
||||
event({ kind: 'judge-start', id: attempt, sameSignal: options?.signal === currentSignal, model: body.model });
|
||||
const id = attempt;
|
||||
await new Promise(resolve => setTimeout(resolve, scenario === 'late-judge' ? 130 : 10));
|
||||
await elapse(scenario === 'late-judge' ? 130 : 10);
|
||||
event({ kind: 'judge-ready', id, aborted: options?.signal?.aborted ?? false });
|
||||
return { stop_reason: 'end_turn', content: [{ type: 'text', text: JSON.stringify({ reason_substance: scenario === 'bad-score' ? 3 : 5, reasoning: 'fixture specific tradeoff' }) }] };
|
||||
};
|
||||
@@ -624,6 +625,7 @@ spyOn(EvalCollector.prototype, 'finalize').mockImplementation(function() {
|
||||
event({ kind: 'finalized' });
|
||||
return finalize.call(this);
|
||||
});
|
||||
jest.useFakeTimers();
|
||||
await import(path.join(root, 'test', ${JSON.stringify(file)}));
|
||||
`);
|
||||
try {
|
||||
|
||||
Reference in new issue
Block a user