diff --git a/CHANGELOG.md b/CHANGELOG.md index b61da0af8..94f9192f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## [Unreleased] -**Four flaky tests now fail only for real defects, and a judge reply that isn't valid JSON says so in its failure line (daily flaky-fix pass, GSTA-91).** `context-restore-fragment-match` failed 0/3 on main cb99df4, although every run restored the right checkpoint: to find the checkpoint whose title matches, the agent often prints all three, and the case failed whenever another checkpoint's body appeared anywhere in the transcript. `test/gstack-ff-update.test.ts` counted every `sleep 300` on the machine, so heartbeats from parallel test files changed its count (remote pregate at #3412's head: expected "1", received "3"). The `GSTACK_CAPY=1` install-admission case in `test/gstack-skill-start.test.ts` makes ten `gstack-skill-start` spawns under bun's 5 s default timeout and took 3.6–5.7 s on a 4-core machine. The setup-browser-cookies workflow judge in train-4 gate 40173759210 returned JSON with an unescaped quote in its reasoning, and the flake record read only "JSON Parse error: Expected '}'". +**Five flaky tests now fail only for real defects, and a judge reply that isn't valid JSON says so in its failure line (daily flaky-fix pass, GSTA-91).** `context-restore-fragment-match` failed 0/3 on main cb99df4, although every run restored the right checkpoint: to find the checkpoint whose title matches, the agent often prints all three, and the case failed whenever another checkpoint's body appeared anywhere in the transcript. `test/gstack-ff-update.test.ts` counted every `sleep 300` on the machine, so heartbeats from parallel test files changed its count (remote pregate at #3412's head: expected "1", received "3"). The `GSTACK_CAPY=1` install-admission case in `test/gstack-skill-start.test.ts` makes ten `gstack-skill-start` spawns under bun's 5 s default timeout and took 3.6–5.7 s on a 4-core machine. The QA-only fixture's late scenarios mocked a 1000 ms capture budget but ran each attempt's real setup inside it. The setup-browser-cookies workflow judge in train-4 gate 40173759210 returned JSON with an unescaped quote in its reasoning, and the flake record read only "JSON Parse error: Expected '}'". ### Fixed - **`test/helpers/context-restore-fragment.ts`** (new): `fragmentRestoreVerdict` reads only the agent's report (top-level assistant text and the final result), never tool output. The report must carry the payments checkpoint's body and no other checkpoint's body. `test/skill-e2e-context-skills.test.ts` uses it for the fragment-match case. @@ -12,10 +12,12 @@ - `test/context-restore-fragment-verdict.test.ts` replays the three captured false reds from main (`test/fixtures/context-restore-fragment-match-captures.json`) as passes. Controls that must fail: captures rewritten to resume alpha or omega, synthetic runs that read and resume only alpha or omega, resume summaries that mix two checkpoints, and a payments body that appears only in tool output. Restoring the old any-byte surface fails all four positive controls. Focused paid runs of the case: 0/3 on main, 3/3 after (about $0.11 a trial). - `test/llm-judge-frontier.test.ts` replays the exact captured cookie-judge reply: it fails closed with no score, one request and the new diagnosis. It fails on the previous helper. - `test/gstack-ff-update.test.ts`: the reap case runs `gstack-ff-update` with a heartbeat interval unique to the test process and counts only sleepers of that interval, starting from zero. The old count failed with a loop starting unrelated `sleep 300` processes beside it (expected "5", received "14"); the new one passes, and still fails when the bin stops reaping its sleeper. +- `test/qa-only-fixture.test.ts`: in the late scenarios the fixture clock stands still from the start of each attempt until its runner starts, so the real setup work (directory copies and five `git` calls) costs no fixture time. The runner and cleanup stay on the real clock, as the deadline-evidence check requires. Each late capture must now get exactly the 1000 ms budget. A new `late-slow-retry` scenario stalls each attempt's setup for 1100 ms, longer than the whole budget, and its attempt 2 still has to succeed. Before, a loaded machine either used up the work budget in setup (`QA-only work budget exhausted`) or Bun's 1580 ms deadline killed a setup `git` child (exit `null` at `skill-e2e-qa-workflow.test.ts:133`); this is what failed late-retry in the 2026-10-24 and 2026-10-25 remote pregates. Under 24 busy loops on 4 cores: late-retry 1/20 → 20/20, late-failure-retry 2/20 → 20/20, late-timeout 6/20 → 20/20, late-slow-retry 20/20. Reverting the clock fails all four late scenarios unloaded; reverting the deadline fails `late-slow-retry`. - `test/gstack-skill-start.test.ts`: the `GSTACK_CAPY=1` install-admission case gets an explicit 20 s timeout. Measured on a 4-core Capy machine with a temporary 60 s cap: 3.6–5.7 s through Capy's git shim (9 of 10 over 5 s), 2.6–4.9 s with the real git binary. Four of its ten spawns are full admissions at about 800 ms each. ### Gate integrity - One test budget changed, approved by GStack Minder on 2026-10-25: the install-admission case's timeout goes from bun's 5 s default to 20 s, about 3.5 times the 5.7 s worst case measured above. No assertion changed. +- The QA-only late scenarios register each attempt with its real outer budget (1000 + 80 + 5000 = 6080 ms, the value `outerMs` already asserts) instead of a shortened 1580 ms Bun deadline, which left cleanup 500 ms of real time under load. Measured child totals with the fix under 24 busy loops, 20 runs each: late-timeout 2.4–2.7 s, late-retry 4.1–4.3 s, late-failure-retry 4.7–5.0 s, late-slow-retry 6.2–6.6 s, against the parent's unchanged 15 s spawn timeout. The mocked 1000 ms capture budget, the 80 ms drain and every existing assertion are unchanged. Verdict: justified, because the deadline was a fixture artifact shorter than the budget the case declares, and the clock freeze, not the deadline, is what removes the load sensitivity. - `context-restore-fragment-match`: the failure surface moves from the whole transcript to the agent's report. Alpha or omega bytes in tool output no longer fail the case; the payments body in tool output no longer counts as loaded, so the report must show it. A run that resumes another checkpoint, or mixes two, still fails. `test/test-of-test-ratchet.test.ts` gains one BASELINE entry, the replay test, with its reason, and the new helper joins the touchfiles of the nine context-skills cases, as the derived-closure rule requires. - No threshold, trial count, case kind, rubric, judge request or prompt changed. diff --git a/test/qa-only-fixture.test.ts b/test/qa-only-fixture.test.ts index 004b5c65e..af853d920 100644 --- a/test/qa-only-fixture.test.ts +++ b/test/qa-only-fixture.test.ts @@ -19,7 +19,7 @@ test('QA-only fixture helpers and owned sections select the no-fix consumer', () }); test.each(['success', 'max-turns', 'dependencies', 'metadata', 'edit', 'source-write', 'source-delete', 'source-add', 'setup', 'runner', 'recording', 'runner-recording', 'artifact-error', - 'process-error', 'slow-setup', 'exhausted-setup', 'late-timeout', 'late-retry', 'late-failure-retry', 'unknown-timeout', + 'process-error', 'slow-setup', 'exhausted-setup', 'late-timeout', 'late-retry', 'late-slow-retry', 'late-failure-retry', 'unknown-timeout', 'cleanup', 'cleanup-retry', 'runner-cleanup', 'cleanup-recording', 'remove-error', 'server-stop', 'missing-deadline', 'unguarded-probe', 'forged-receipt', 'deadline-write', 'reset-deadline', 'preparation-missing', 'preparation-failed', 'preparation-unacknowledged', 'preparation-wrong-path', 'preparation-late', @@ -47,11 +47,12 @@ if (scenario === 'remove-error') spyOn(fs, 'rmSync').mockImplementation((target, return originalRm(target, options); }); mock.module(path.join(root, 'test/helpers/eval-store.ts'), () => ({ getProjectEvalDir: () => ${JSON.stringify(path.join(dir, 'evals'))} })); -const late = scenario.startsWith('late-'), workMs = 1000, drainMs = 80; +const late = scenario.startsWith('late-'), lateRetry = ['late-retry', 'late-slow-retry'].includes(scenario), workMs = 1000, drainMs = 80; const realNow = Date.now, controlledClock = ['slow-setup', 'exhausted-setup'].includes(scenario); const clockStart = realNow(); -let clock = clockStart; +let clock = clockStart, frozenAt; if (controlledClock) Date.now = () => clock; +else if (late) Date.now = () => frozenAt ?? realNow(); let finalized = false, outerMs; const collector = { addTest: row => { const cwd = directories.at(-1); @@ -77,8 +78,7 @@ mock.module(path.join(root, 'test/helpers/e2e-helpers.ts'), () => ({ testConcurrentIfSelected: (id, body, budget) => { if (id !== 'qa-only-no-fix') return; outerMs = budget; - const deadline = late ? (budget >= workMs + 5000 ? budget - 5000 + 500 : budget) : 5000; - test(id, body, deadline); + test(id, late ? () => { frozenAt = realNow(); return body(); } : body, late ? budget : 5000); }, copyDirSync: (source, target) => { if (scenario === 'setup') throw new Error('QA-only setup failed'); @@ -87,6 +87,7 @@ mock.module(path.join(root, 'test/helpers/e2e-helpers.ts'), () => ({ setupBrowseShims: cwd => { directories.push(cwd); fs.mkdirSync(path.join(cwd, 'browse/bin'), { recursive: true }); if (controlledClock) clock += scenario === 'slow-setup' ? 295000 : 300000; + if (scenario === 'late-slow-retry') Bun.sleepSync(workMs + 100); }, logCost() {}, createEvalCollector: () => collector, finalizeEvalCollector: async () => { finalized = true; }, recordE2E: (_collector, name, suite, result, extra) => collector.addTest({ name, suite, @@ -96,6 +97,7 @@ mock.module(path.join(root, 'test/helpers/e2e-helpers.ts'), () => ({ mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({ SESSION_DRAIN_GRACE_MS: late ? drainMs : 5000, runSkillTest: async options => { + frozenAt = undefined; const cwd = options.workingDirectory; const fact = { cwd, exists: fs.existsSync(cwd), recordedBeforeCapture: records.length, reportBefore: fs.existsSync(path.join(cwd, 'qa-reports/qa-only-report.md')), captureMs: options.timeout, @@ -145,7 +147,7 @@ mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({ fact.existsAfterDrain = fs.existsSync(cwd); return { ...result, exitReason: 'timeout' }; } - if (late && (scenario !== 'late-retry' || attempts.length === 1)) { + if (late && (!lateRetry || attempts.length === 1)) { await new Promise(resolve => setTimeout(resolve, options.timeout + drainMs)); fact.existsAfterDrain = fs.existsSync(cwd); return { ...result, exitReason: 'timeout' }; @@ -237,7 +239,7 @@ await import(path.join(root, 'test/skill-e2e-qa-workflow.test.ts')); expect(records[index].cleanupDone).toBe(true); expect(cleanups[index].budget).toBeGreaterThan(0); expect(cleanups[index].budget).toBeLessThanOrEqual(4000); - expect(records[index].passed).toBe(scenario === 'recording' || !shouldFail && !(['late-retry', 'cleanup-retry'].includes(scenario) && index === 0)); + expect(records[index].passed).toBe(scenario === 'recording' || !shouldFail && !(['late-retry', 'late-slow-retry', 'cleanup-retry'].includes(scenario) && index === 0)); if (scenario.includes('cleanup') && !(scenario === 'cleanup-retry' && index === 1)) expect(records[index].error).toContain('QA-only browser cleanup: invalid daemon identity'); } for (const [index, attempt] of attempts.entries()) { @@ -257,6 +259,7 @@ await import(path.join(root, 'test/skill-e2e-qa-workflow.test.ts')); } if (scenario.startsWith('late-')) { expect(outerMs).toBe(1000 + 80 + 5000); + for (const attempt of attempts) expect(attempt.captureMs).toBe(1000); expect(records[0]).toMatchObject({ passed: false, exit_reason: 'timeout', cost_usd: 0.25 }); expect(records[0].error).toContain('timeout'); if (count === 2) expect(directories[0]).not.toBe(directories[1]);