test(qa-only): late scenarios freeze the fixture clock during setup and register their real outer budget (GSTA-91)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 ## [Unreleased]
 
-**Four flaky tests now fail only for real defects, and a judge reply that isn't valid JSON says so in its failure line (daily flaky-fix pass, GSTA-91).** `context-restore-fragment-match` failed 0/3 on main cb99df4, although every run restored the right checkpoint: to find the checkpoint whose title matches, the agent often prints all three, and the case failed whenever another checkpoint's body appeared anywhere in the transcript. `test/gstack-ff-update.test.ts` counted every `sleep 300` on the machine, so heartbeats from parallel test files changed its count (remote pregate at #3412's head: expected "1", received "3"). The `GSTACK_CAPY=1` install-admission case in `test/gstack-skill-start.test.ts` makes ten `gstack-skill-start` spawns under bun's 5 s default timeout and took 3.6–5.7 s on a 4-core machine. The setup-browser-cookies workflow judge in train-4 gate 40173759210 returned JSON with an unescaped quote in its reasoning, and the flake record read only "JSON Parse error: Expected '}'".
+**Five flaky tests now fail only for real defects, and a judge reply that isn't valid JSON says so in its failure line (daily flaky-fix pass, GSTA-91).** `context-restore-fragment-match` failed 0/3 on main cb99df4, although every run restored the right checkpoint: to find the checkpoint whose title matches, the agent often prints all three, and the case failed whenever another checkpoint's body appeared anywhere in the transcript. `test/gstack-ff-update.test.ts` counted every `sleep 300` on the machine, so heartbeats from parallel test files changed its count (remote pregate at #3412's head: expected "1", received "3"). The `GSTACK_CAPY=1` install-admission case in `test/gstack-skill-start.test.ts` makes ten `gstack-skill-start` spawns under bun's 5 s default timeout and took 3.6–5.7 s on a 4-core machine. The QA-only fixture's late scenarios mocked a 1000 ms capture budget but ran each attempt's real setup inside it. The setup-browser-cookies workflow judge in train-4 gate 40173759210 returned JSON with an unescaped quote in its reasoning, and the flake record read only "JSON Parse error: Expected '}'".
 
 ### Fixed
 - **`test/helpers/context-restore-fragment.ts`** (new): `fragmentRestoreVerdict` reads only the agent's report (top-level assistant text and the final result), never tool output. The report must carry the payments checkpoint's body and no other checkpoint's body. `test/skill-e2e-context-skills.test.ts` uses it for the fragment-match case.
@@ -12,10 +12,12 @@
 - `test/context-restore-fragment-verdict.test.ts` replays the three captured false reds from main (`test/fixtures/context-restore-fragment-match-captures.json`) as passes. Controls that must fail: captures rewritten to resume alpha or omega, synthetic runs that read and resume only alpha or omega, resume summaries that mix two checkpoints, and a payments body that appears only in tool output. Restoring the old any-byte surface fails all four positive controls. Focused paid runs of the case: 0/3 on main, 3/3 after (about $0.11 a trial).
 - `test/llm-judge-frontier.test.ts` replays the exact captured cookie-judge reply: it fails closed with no score, one request and the new diagnosis. It fails on the previous helper.
 - `test/gstack-ff-update.test.ts`: the reap case runs `gstack-ff-update` with a heartbeat interval unique to the test process and counts only sleepers of that interval, starting from zero. The old count failed with a loop starting unrelated `sleep 300` processes beside it (expected "5", received "14"); the new one passes, and still fails when the bin stops reaping its sleeper.
+- `test/qa-only-fixture.test.ts`: in the late scenarios the fixture clock stands still from the start of each attempt until its runner starts, so the real setup work (directory copies and five `git` calls) costs no fixture time. The runner and cleanup stay on the real clock, as the deadline-evidence check requires. Each late capture must now get exactly the 1000 ms budget. A new `late-slow-retry` scenario stalls each attempt's setup for 1100 ms, longer than the whole budget, and its attempt 2 still has to succeed. Before, a loaded machine either used up the work budget in setup (`QA-only work budget exhausted`) or Bun's 1580 ms deadline killed a setup `git` child (exit `null` at `skill-e2e-qa-workflow.test.ts:133`); this is what failed late-retry in the 2026-10-24 and 2026-10-25 remote pregates. Under 24 busy loops on 4 cores: late-retry 1/20 → 20/20, late-failure-retry 2/20 → 20/20, late-timeout 6/20 → 20/20, late-slow-retry 20/20. Reverting the clock fails all four late scenarios unloaded; reverting the deadline fails `late-slow-retry`.
 - `test/gstack-skill-start.test.ts`: the `GSTACK_CAPY=1` install-admission case gets an explicit 20 s timeout. Measured on a 4-core Capy machine with a temporary 60 s cap: 3.6–5.7 s through Capy's git shim (9 of 10 over 5 s), 2.6–4.9 s with the real git binary. Four of its ten spawns are full admissions at about 800 ms each.
 
 ### Gate integrity
 - One test budget changed, approved by GStack Minder on 2026-10-25: the install-admission case's timeout goes from bun's 5 s default to 20 s, about 3.5 times the 5.7 s worst case measured above. No assertion changed.
+- The QA-only late scenarios register each attempt with its real outer budget (1000 + 80 + 5000 = 6080 ms, the value `outerMs` already asserts) instead of a shortened 1580 ms Bun deadline, which left cleanup 500 ms of real time under load. Measured child totals with the fix under 24 busy loops, 20 runs each: late-timeout 2.4–2.7 s, late-retry 4.1–4.3 s, late-failure-retry 4.7–5.0 s, late-slow-retry 6.2–6.6 s, against the parent's unchanged 15 s spawn timeout. The mocked 1000 ms capture budget, the 80 ms drain and every existing assertion are unchanged. Verdict: justified, because the deadline was a fixture artifact shorter than the budget the case declares, and the clock freeze, not the deadline, is what removes the load sensitivity.
 - `context-restore-fragment-match`: the failure surface moves from the whole transcript to the agent's report. Alpha or omega bytes in tool output no longer fail the case; the payments body in tool output no longer counts as loaded, so the report must show it. A run that resumes another checkpoint, or mixes two, still fails. `test/test-of-test-ratchet.test.ts` gains one BASELINE entry, the replay test, with its reason, and the new helper joins the touchfiles of the nine context-skills cases, as the derived-closure rule requires.
 - No threshold, trial count, case kind, rubric, judge request or prompt changed.
 

test/qa-only-fixture.test.ts

@@ -19,7 +19,7 @@ test('QA-only fixture helpers and owned sections select the no-fix consumer', ()
 });
 
 test.each(['success', 'max-turns', 'dependencies', 'metadata', 'edit', 'source-write', 'source-delete', 'source-add', 'setup', 'runner', 'recording', 'runner-recording', 'artifact-error',
-  'process-error', 'slow-setup', 'exhausted-setup', 'late-timeout', 'late-retry', 'late-failure-retry', 'unknown-timeout',
+  'process-error', 'slow-setup', 'exhausted-setup', 'late-timeout', 'late-retry', 'late-slow-retry', 'late-failure-retry', 'unknown-timeout',
   'cleanup', 'cleanup-retry', 'runner-cleanup', 'cleanup-recording', 'remove-error', 'server-stop',
   'missing-deadline', 'unguarded-probe', 'forged-receipt', 'deadline-write', 'reset-deadline',
   'preparation-missing', 'preparation-failed', 'preparation-unacknowledged', 'preparation-wrong-path', 'preparation-late',
@@ -47,11 +47,12 @@ if (scenario === 'remove-error') spyOn(fs, 'rmSync').mockImplementation((target,
   return originalRm(target, options);
 });
 mock.module(path.join(root, 'test/helpers/eval-store.ts'), () => ({ getProjectEvalDir: () => ${JSON.stringify(path.join(dir, 'evals'))} }));
-const late = scenario.startsWith('late-'), workMs = 1000, drainMs = 80;
+const late = scenario.startsWith('late-'), lateRetry = ['late-retry', 'late-slow-retry'].includes(scenario), workMs = 1000, drainMs = 80;
 const realNow = Date.now, controlledClock = ['slow-setup', 'exhausted-setup'].includes(scenario);
 const clockStart = realNow();
-let clock = clockStart;
+let clock = clockStart, frozenAt;
 if (controlledClock) Date.now = () => clock;
+else if (late) Date.now = () => frozenAt ?? realNow();
 let finalized = false, outerMs;
 const collector = { addTest: row => {
   const cwd = directories.at(-1);
@@ -77,8 +78,7 @@ mock.module(path.join(root, 'test/helpers/e2e-helpers.ts'), () => ({
   testConcurrentIfSelected: (id, body, budget) => {
     if (id !== 'qa-only-no-fix') return;
     outerMs = budget;
-    const deadline = late ? (budget >= workMs + 5000 ? budget - 5000 + 500 : budget) : 5000;
-    test(id, body, deadline);
+    test(id, late ? () => { frozenAt = realNow(); return body(); } : body, late ? budget : 5000);
   },
   copyDirSync: (source, target) => {
     if (scenario === 'setup') throw new Error('QA-only setup failed');
@@ -87,6 +87,7 @@ mock.module(path.join(root, 'test/helpers/e2e-helpers.ts'), () => ({
   setupBrowseShims: cwd => {
     directories.push(cwd); fs.mkdirSync(path.join(cwd, 'browse/bin'), { recursive: true });
     if (controlledClock) clock += scenario === 'slow-setup' ? 295000 : 300000;
+    if (scenario === 'late-slow-retry') Bun.sleepSync(workMs + 100);
   },
   logCost() {}, createEvalCollector: () => collector, finalizeEvalCollector: async () => { finalized = true; },
   recordE2E: (_collector, name, suite, result, extra) => collector.addTest({ name, suite,
@@ -96,6 +97,7 @@ mock.module(path.join(root, 'test/helpers/e2e-helpers.ts'), () => ({
 mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({
   SESSION_DRAIN_GRACE_MS: late ? drainMs : 5000,
   runSkillTest: async options => {
+    frozenAt = undefined;
     const cwd = options.workingDirectory;
     const fact = { cwd, exists: fs.existsSync(cwd), recordedBeforeCapture: records.length,
       reportBefore: fs.existsSync(path.join(cwd, 'qa-reports/qa-only-report.md')), captureMs: options.timeout,
@@ -145,7 +147,7 @@ mock.module(path.join(root, 'test/helpers/session-runner.ts'), () => ({
       fact.existsAfterDrain = fs.existsSync(cwd);
       return { ...result, exitReason: 'timeout' };
     }
-    if (late && (scenario !== 'late-retry' || attempts.length === 1)) {
+    if (late && (!lateRetry || attempts.length === 1)) {
       await new Promise(resolve => setTimeout(resolve, options.timeout + drainMs));
       fact.existsAfterDrain = fs.existsSync(cwd);
       return { ...result, exitReason: 'timeout' };
@@ -237,7 +239,7 @@ await import(path.join(root, 'test/skill-e2e-qa-workflow.test.ts'));
       expect(records[index].cleanupDone).toBe(true);
       expect(cleanups[index].budget).toBeGreaterThan(0);
       expect(cleanups[index].budget).toBeLessThanOrEqual(4000);
-      expect(records[index].passed).toBe(scenario === 'recording' || !shouldFail && !(['late-retry', 'cleanup-retry'].includes(scenario) && index === 0));
+      expect(records[index].passed).toBe(scenario === 'recording' || !shouldFail && !(['late-retry', 'late-slow-retry', 'cleanup-retry'].includes(scenario) && index === 0));
       if (scenario.includes('cleanup') && !(scenario === 'cleanup-retry' && index === 1)) expect(records[index].error).toContain('QA-only browser cleanup: invalid daemon identity');
     }
     for (const [index, attempt] of attempts.entries()) {
@@ -257,6 +259,7 @@ await import(path.join(root, 'test/skill-e2e-qa-workflow.test.ts'));
     }
     if (scenario.startsWith('late-')) {
       expect(outerMs).toBe(1000 + 80 + 5000);
+      for (const attempt of attempts) expect(attempt.captureMs).toBe(1000);
       expect(records[0]).toMatchObject({ passed: false, exit_reason: 'timeout', cost_usd: 0.25 });
       expect(records[0].error).toContain('timeout');
       if (count === 2) expect(directories[0]).not.toBe(directories[1]);