v1.93.14.0 ci: each free-suite job's timeout follows the plan's runner wall (#3850) (#3877)

* fix(free-tests): each free-suite job's timeout comes from the plan (runner wall + flaky retry + setup), not a fixed 20 minutes (#3850)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: free-suite job timeout follows the plan (#3850)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(free-tests): narrow the CI matrix in the #3850 wiring test; keep free-suite job names

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

.github/workflows/free-tests.yml

@@ -52,10 +52,16 @@ on:
 # that free-suite reads. FREE_SUITE_JOBS is k in `--ci-plan --shard-jobs k`:
 # each shard runs min(k, its parallel files) sandboxed sub-shards side by side
 # (scripts/lib/shard-jobs.ts); the exclusive host-state shard stays one child.
+# FREE_SUITE_SETUP_SECONDS is a free-suite job's time outside the "Run free
+# suite" step. --ci-plan adds it to each shard's runner wall and flaky-retry
+# wall to emit that shard's job timeout (#3850), so GitHub never kills a job
+# before the runner can name a hung file. Measured over 110 free-suite jobs in
+# PR runs 41467261350..41478988224: p50 60 s, p90 80 s, max 142 s.
 env:
   FREE_SUITE_RUNNER: ubicloud-standard-8
   FREE_SUITE_SHARDS: '10'
   FREE_SUITE_JOBS: '4'
+  FREE_SUITE_SETUP_SECONDS: '300'
 
 # Keyed on the PR number, not head_ref: a bare branch name carries no fork
 # prefix, so same-name branches from two forks would share one group and a
@@ -149,7 +155,7 @@ jobs:
           if [ "$runner" != "$FREE_SUITE_RUNNER" ] || [ "$shards" != "$FREE_SUITE_SHARDS" ] || [ "$jobs" != "$FREE_SUITE_JOBS" ]; then
             echo "- Runner-sizing experiment: $shards shards on $runner at $jobs sub-shards each (defaults $FREE_SUITE_SHARDS on $FREE_SUITE_RUNNER at $FREE_SUITE_JOBS); not a PR or train Free Tests result" >> "$GITHUB_STEP_SUMMARY"
           fi
-          matrix=$(bun run scripts/test-free-shards.ts --ci-plan "$RUNNER_TEMP/free-plan.json" --shards "$shards" --shard-jobs "$jobs")
+          matrix=$(bun run scripts/test-free-shards.ts --ci-plan "$RUNNER_TEMP/free-plan.json" --shards "$shards" --shard-jobs "$jobs" --job-setup-seconds "$FREE_SUITE_SETUP_SECONDS")
           echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
           echo "runner=$runner" >> "$GITHUB_OUTPUT"
       - name: Seed growth ratchet (60 s ceiling, shrink-only allowlist)
@@ -434,10 +440,12 @@ jobs:
         run: xvfb-run -a bun run test:quick
 
   free-suite:
+    name: free-suite (${{ matrix.shard }})
     needs: [free-plan, reuse]
     if: ${{ !cancelled() && needs.free-plan.result == 'success' && (needs.reuse.result != 'success' || needs.reuse.outputs.reused != 'true') }}
     runs-on: ${{ needs.free-plan.outputs.runner }}
-    timeout-minutes: 20
+    # Per shard, from the plan: runner wall + flaky retry + setup (#3850).
+    timeout-minutes: ${{ matrix.timeout_minutes }}
     strategy:
       fail-fast: false
       max-parallel: 20

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## [1.93.14.0] - 2026-11-02
+
+Free Tests no longer goes red with no named file when one test hangs. Each `free-suite` job's GitHub timeout used to be a fixed 20 minutes, shorter than the runner's own wall for a 10-shard plan (about 37 minutes), so GitHub killed the job before the runner could name the hung file or retry it, and no shard result was uploaded. The plan now emits each shard's job timeout as its runner wall plus the flaky-retry pass plus the job's measured setup (`--ci-plan --job-setup-seconds`), 49 minutes per shard today, and fails loudly above 90 minutes instead of truncating. See docs/TESTING_INTERNALS.md#runners-how-the-suites-execute-2026-08-overhaul. Closes #3850.
+
 ## [1.93.13.0] - 2026-11-02
 
 **gstack's own tests no longer write into a developer's real gbrain.** If you export `GBRAIN_DATABASE_URL` for your working brain and run gstack's test suite, tests that spawn bins with a temp `HOME` used to pass that URL straight through. With no gbrain config under the temp `HOME`, the real `gbrain` on your PATH then added `brain-sync-remote-*` sources and test pages to your live brain. From #3466 by @AndreIntelas.

VERSION

@@ -1 +1 @@
-1.93.13.0
+1.93.14.0

agents-digest/gstack-AGENTS.md

@@ -1,4 +1,4 @@
-# gstack digest v1.93.13.0 — regenerate/re-copy after upgrading gstack
+# gstack digest v1.93.14.0 — regenerate/re-copy after upgrading gstack
 
 Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
 for agent hosts without a full skill install. The full skills add workflows,

docs/TESTING_INTERNALS.md

@@ -206,7 +206,16 @@ GSTA-203 chose them by a 3-run A/B against 20 × `ubicloud-standard-4` at k = 1
 (the PR that set them has the table). A `workflow_dispatch` run may
 override them with `free_runner` (`ubicloud-standard-2`, `-4` or `-8`),
 `free_shards` (1–40) and `free_jobs` (1–8) for runner-sizing experiments only;
-its job summary says so, and it is never cited as a PR or train Free Tests result. Each shard prints
+its job summary says so, and it is never cited as a PR or train Free Tests result.
+Each `free-suite` job's GitHub timeout comes from the plan, not a fixed number
+(#3850): `--ci-plan --job-setup-seconds $FREE_SUITE_SETUP_SECONDS` emits every
+shard's `timeout_minutes` as the shard's runner wall (the packed wall above),
+plus the flaky-retry pass's wall (at most 5 files, so the 6-minute base), plus
+the job's setup and teardown (`ciShardJobTimeoutMinutes`). So GitHub never kills
+a hung file before the runner names it and retries it. A shard that would need
+more than 90 minutes fails the plan with `FREE_JOB_TIMEOUT_OVER_CAP`; plan more
+shards or a higher k instead. The plan step prints the emitted timeouts and adds
+them to its job summary. Each shard prints
 `RESOURCES: peak_used_mib=… min_free_disk_mib=… samples=…` from a memory and
 disk sample every 2 seconds, even when the suite fails. The public
 `TREE_MUTATING` map now classifies exclusive host-state fixtures; its sole

package.json

@@ -1,6 +1,6 @@
 {
   "name": "gstack",
-  "version": "1.93.13",
+  "version": "1.93.14",
   "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
   "license": "MIT",
   "type": "module",

scripts/test-free-shards.ts

@@ -201,6 +201,24 @@ export function wallTimeoutForPackedShard(predictedMs: number, baseMs = DEFAULT_
   // tighter — it keeps the per-file floor the runner has always guaranteed.
   return Math.max(baseMs, Math.ceil(predictedMs * 3), fileCount * PER_FILE_WALL_MS);
 }
+/** The flaky-retry pass re-runs at most this many failing files, once, serially, in the same CI step. */
+export const FLAKY_RETRY_CAP = 5;
+/** No free-suite GitHub job gets a longer timeout; a plan whose shard would need more fails instead. */
+export const MAX_CI_JOB_TIMEOUT_MINUTES = 90;
+/**
+ * The GitHub job timeout for one planned CI shard (#3850): the wall --ci-run
+ * gives the shard, plus the flaky-retry pass's wall, plus the job's measured
+ * setup and teardown. GitHub must never kill the job before the runner can
+ * name a hung file and retry it; that kill uploads no result.
+ */
+export function ciShardJobTimeoutMinutes(shard: { shard: number; files: string[]; predictedMs: number }, setupSeconds: number): number {
+  const runnerMs = wallTimeoutForPackedShard(shard.predictedMs, DEFAULT_WALL_TIMEOUT_MS, shard.files.length) + wallTimeoutForShard(FLAKY_RETRY_CAP);
+  const minutes = Math.ceil((runnerMs / 1000 + setupSeconds) / 60);
+  if (minutes > MAX_CI_JOB_TIMEOUT_MINUTES) {
+    throw new Error(`FREE_JOB_TIMEOUT_OVER_CAP: CI shard ${shard.shard} needs a ${minutes}-minute job (runner wall plus retry plus ${setupSeconds} s setup), over the ${MAX_CI_JOB_TIMEOUT_MINUTES}-minute cap; plan more shards or a higher --shard-jobs instead of a longer job`);
+  }
+  return minutes;
+}
 /**
  * Full-suite parallelism: use all available CPUs, with a floor of one and a
  * per-platform cap (maxFullSuiteJobs). Shards stay serial internally; separate
@@ -453,6 +471,16 @@ export function createFreeCiPlan(files: string[], count: number, durations: Reco
   return { ...body, id: planDigest(body) };
 }
 
+/**
+ * --ci-plan's stdout, the GitHub matrix. With --job-setup-seconds each entry
+ * carries the shard's job timeout (ciShardJobTimeoutMinutes); without it, the
+ * bare shard list the Windows lane reads.
+ */
+export function ciPlanMatrix(plan: FreeCiPlan, setupSeconds?: number): { shard: number[] } | { include: Array<{ shard: number; timeout_minutes: number }> } {
+  if (setupSeconds === undefined) return { shard: plan.shards.map(shard => shard.shard) };
+  return { include: plan.shards.map(shard => ({ shard: shard.shard, timeout_minutes: ciShardJobTimeoutMinutes(shard, setupSeconds) })) };
+}
+
 export function validateFreeCiPlan(plan: FreeCiPlan, files: string[], revision: string): void {
   const { id, version, shards } = plan;
   if (version !== 1 || plan.revision !== revision || !Array.isArray(shards) || !shards.length
@@ -575,6 +603,8 @@ type CliOptions = {
   jobs?: number;
   /** Set only by --shard-jobs (with --ci-plan); absent means k = 1 in every shard. */
   shardJobs?: number;
+  /** Set only by --job-setup-seconds (with --ci-plan): the matrix then carries each shard's job timeout. */
+  jobSetupSeconds?: number;
 };
 
 export function parseCliOptions(argv: string[]): CliOptions {
@@ -591,6 +621,7 @@ export function parseCliOptions(argv: string[]): CliOptions {
   let attributeHome = false;
   let jobs: number | undefined;
   let shardJobs: number | undefined;
+  let jobSetupSeconds: number | undefined;
   const paths: Record<'ciPlan' | 'ciRun' | 'ciVerify' | 'result' | 'results' | 'filesFrom', string | null> = {
     ciPlan: null, ciRun: null, ciVerify: null, result: null, results: null, filesFrom: null,
   };
@@ -616,6 +647,11 @@ export function parseCliOptions(argv: string[]): CliOptions {
     '--files-from': pathFlag('--files-from', 'filesFrom'),
     '--jobs': (next) => { jobs = parseJobsFlag(next()); },
     '--shard-jobs': (next) => { shardJobs = parseJobsFlag(next(), '--shard-jobs'); },
+    '--job-setup-seconds': (next) => {
+      const value = next() ?? '';
+      jobSetupSeconds = /^\d+$/.test(value) ? Number(value) : NaN;
+      if (!(jobSetupSeconds > 0 && jobSetupSeconds <= 3600)) throw new Error('--job-setup-seconds needs a whole number of seconds from 1 to 3600');
+    },
     '--shards': (next) => {
       const value = next();
       if (!value) throw new Error('Missing value for --shards');
@@ -641,8 +677,9 @@ export function parseCliOptions(argv: string[]): CliOptions {
   if (paths.ciVerify && !paths.results) throw new Error('--ci-verify requires --results');
   if (jobs !== undefined && !paths.ciRun) rejectJobsWithoutCiRun();
   if (shardJobs !== undefined && !paths.ciPlan) rejectShardJobsWithoutCiPlan();
+  if (jobSetupSeconds !== undefined && !paths.ciPlan) throw new Error('--job-setup-seconds applies only to --ci-plan');
   if (quick && (recordDurations || windowsOnly || shardIndex !== null)) throw new Error('--quick cannot change recording, Windows or shard selection');
-  return { dryRun, listOnly, recordDurations, windowsOnly, verbose, shardCount, shardIndex, wallTimeoutMs, wallTimeoutExplicit, quick, attributeHome, ...paths, ...(jobs === undefined ? {} : { jobs }), ...(shardJobs === undefined ? {} : { shardJobs }) };
+  return { dryRun, listOnly, recordDurations, windowsOnly, verbose, shardCount, shardIndex, wallTimeoutMs, wallTimeoutExplicit, quick, attributeHome, ...paths, ...(jobs === undefined ? {} : { jobs }), ...(shardJobs === undefined ? {} : { shardJobs }), ...(jobSetupSeconds === undefined ? {} : { jobSetupSeconds }) };
 }
 
 const describeLines = (lines: string[]) => `${lines.slice(0, 5).join(', ')}${lines.length > 5 ? ` (+${lines.length - 5} more)` : ''}`;
@@ -1891,7 +1928,6 @@ async function retryFailedFreeFiles(
   // Exists for syscall-supervised sandboxes (see fullSuiteJobs) where a run
   // lands 0-1 spurious browser-timing failures under an otherwise-green
   // suite. Capped so a genuinely broken tree never masquerades as flaky.
-  const RETRY_CAP = 5;
   if (
     worst !== 0
     && process.env.GSTACK_FREE_RETRY_FLAKY === '1'
@@ -1906,7 +1942,7 @@ async function retryFailedFreeFiles(
     // mask the unattributable evidence as a FLAKY-PASS.
     const allAttributed = outcomes.every((o) => hasScopedFailureAttribution(o) && (o.status === 'passed'
       || (o.failingFiles.length > 0 && o.unattributedFailures === 0)));
-    if (allAttributed && flakyFiles.length > 0 && flakyFiles.length <= RETRY_CAP) {
+    if (allAttributed && flakyFiles.length > 0 && flakyFiles.length <= FLAKY_RETRY_CAP) {
       console.log(`[test:free] flaky-retry: re-running ${flakyFiles.length} failing file(s) once, serially: ${flakyFiles.join(', ')}`);
       const retryOutcome = await runFreeShard(flakyFiles, totalShards + 1, totalShards + 1, {
         wallTimeoutMs: shardTimeout(flakyFiles.length),
@@ -1942,7 +1978,7 @@ async function retryFailedFreeFiles(
         console.error('[test:free] flaky-retry FAILED — the failures reproduce serially; not flaky.');
       }
     } else {
-      console.log(`[test:free] flaky-retry skipped: ${allAttributed ? `${flakyFiles.length} failing file(s) exceeds cap ${RETRY_CAP}` : 'failures not fully attributed'}.`);
+      console.log(`[test:free] flaky-retry skipped: ${allAttributed ? `${flakyFiles.length} failing file(s) exceeds cap ${FLAKY_RETRY_CAP}` : 'failures not fully attributed'}.`);
     }
   }
   return { exitCode: worst, retry };
@@ -1978,8 +2014,14 @@ async function main(): Promise<number> {
       warnUnseededFreeFiles(ciFiles, durations);
       const plan = createFreeCiPlan(ciFiles, options.shardCount, durations, revision, options.shardJobs);
       validateFreeCiPlan(plan, ciFiles, revision);
+      const matrix = ciPlanMatrix(plan, options.jobSetupSeconds);
+      if ('include' in matrix) {
+        const line = `job timeout per shard (runner wall + flaky retry + ${options.jobSetupSeconds} s setup): ${matrix.include.map(entry => `${entry.shard}=${entry.timeout_minutes}m`).join(' ')}`;
+        console.error(`[test:free] ${line}`);
+        appendStepSummary(line);
+      }
       writeJson(options.ciPlan, plan);
-      console.log(JSON.stringify({ shard: plan.shards.map(shard => shard.shard) }));
+      console.log(JSON.stringify(matrix));
       return 0;
     }
     const plan = JSON.parse(fs.readFileSync((options.ciRun ?? options.ciVerify)!, 'utf8')) as FreeCiPlan;

test/free-tests-workflow-wiring.test.ts

@@ -19,6 +19,10 @@ import * as fs from 'fs';
 import * as path from 'path';
 import * as os from 'os';
 import { spawnSync } from 'node:child_process';
+import {
+  DEFAULT_WALL_TIMEOUT_MS, FLAKY_RETRY_CAP, ciPlanMatrix, collectFreeTestFiles, createFreeCiPlan,
+  loadFreeTestDurations, wallTimeoutForPackedShard, wallTimeoutForShard,
+} from '../scripts/test-free-shards';
 
 const WORKFLOW = path.resolve(import.meta.dir, '..', '.github', 'workflows', 'free-tests.yml');
 
@@ -235,4 +239,30 @@ describe('free-tests workflow wiring', () => {
     expect(source).toMatch(/permissions:\s*\n\s*contents:\s*read/);
     expect(source).toMatch(/persist-credentials:\s*false/);
   });
+  test('every free-suite job timeout covers its shard\'s runner wall, flaky retry and setup (#3850)', () => {
+    // GitHub must never kill a free-suite job before the runner's own wall can
+    // name the hung file and its flaky-retry pass can run: that kill uploads no
+    // result and leaves red CI with no attribution. The job timeout therefore
+    // follows the real plan for the committed duration seed and the live
+    // FREE_SUITE_SHARDS / FREE_SUITE_JOBS, never a hand-picked number.
+    const workflow = Bun.YAML.parse(source) as any;
+    const setupSeconds = Number(workflow.env.FREE_SUITE_SETUP_SECONDS);
+    expect(Number.isInteger(setupSeconds) && setupSeconds > 0).toBe(true);
+    const plan = createFreeCiPlan(collectFreeTestFiles(), Number(workflow.env.FREE_SUITE_SHARDS),
+      loadFreeTestDurations() ?? {}, 'wiring', Number(workflow.env.FREE_SUITE_JOBS));
+    const declared = workflow.jobs['free-suite']['timeout-minutes'];
+    const emitted = ciPlanMatrix(plan, setupSeconds);
+    if (!('include' in emitted)) throw new Error('the CI matrix carries no per-shard timeouts');
+    const short = plan.shards.flatMap(shard => {
+      const neededMs = wallTimeoutForPackedShard(shard.predictedMs, DEFAULT_WALL_TIMEOUT_MS, shard.files.length)
+        + wallTimeoutForShard(FLAKY_RETRY_CAP) + setupSeconds * 1000;
+      const minutes = declared === '${{ matrix.timeout_minutes }}'
+        ? emitted.include.find(entry => entry.shard === shard.shard)?.timeout_minutes ?? 0
+        : Number(declared);
+      return minutes * 60_000 >= neededMs ? [] : [`shard ${shard.shard}: ${minutes} min < ${Math.ceil(neededMs / 1000)} s`];
+    });
+    expect(short).toEqual([]);
+    const planStep = workflow.jobs['free-plan'].steps.find((step: any) => step.id === 'plan');
+    expect(planStep.run).toContain('--job-setup-seconds "$FREE_SUITE_SETUP_SECONDS"');
+  });
 });

test/test-free-shards.test.ts

@@ -29,6 +29,9 @@ import {
   loadFreeTestDurations,
   packShardsByDuration,
   wallTimeoutForPackedShard,
+  ciShardJobTimeoutMinutes,
+  ciPlanMatrix,
+  MAX_CI_JOB_TIMEOUT_MINUTES,
   createFreeCiPlan,
   validateFreeCiPlan,
   verifyFreeCiResults,
@@ -1101,4 +1104,30 @@ describe('test-free-shards: duration-aware packing (full-suite LPT)', () => {
     // When the prediction is the larger bound, it still wins.
     expect(wallTimeoutForPackedShard(600_000, WALL_BASE_MS, 10)).toBe(1_800_000);
   });
+
+  test('a shard\'s job timeout is its runner wall plus the flaky-retry wall plus setup, rounded up (#3850)', () => {
+    // 750 s predicted -> 2250 s wall, + 360 s retry, + 300 s setup = 2910 s -> 49 min.
+    expect(ciShardJobTimeoutMinutes({ shard: 1, files: ['a'], predictedMs: 750_000 }, 300)).toBe(49);
+    // A tiny shard still covers the 6-minute base wall and the retry pass.
+    expect(ciShardJobTimeoutMinutes({ shard: 2, files: ['a'], predictedMs: 1_000 }, 60)).toBe(13);
+  });
+
+  test('a shard that would need a job over the cap fails the plan instead of being truncated', () => {
+    const predictedMs = MAX_CI_JOB_TIMEOUT_MINUTES * 60_000 / 3;
+    expect(() => ciShardJobTimeoutMinutes({ shard: 4, files: ['a'], predictedMs }, 300)).toThrow(/FREE_JOB_TIMEOUT_OVER_CAP: CI shard 4 needs a 101-minute job/);
+  });
+
+  test('the matrix carries per-shard timeouts only when the plan is given a setup margin', () => {
+    const plan = createFreeCiPlan(['a.test.ts', 'b.test.ts'], 2, { 'a.test.ts': 750_000, 'b.test.ts': 1_000 }, 'rev');
+    expect(ciPlanMatrix(plan)).toEqual({ shard: [1, 2] });
+    expect(ciPlanMatrix(plan, 300)).toEqual({ include: [{ shard: 1, timeout_minutes: 49 }, { shard: 2, timeout_minutes: 17 }] });
+  });
+
+  test('--job-setup-seconds is a --ci-plan flag with whole seconds from 1 to 3600', () => {
+    expect(parseCliOptions(['--ci-plan', 'p.json', '--job-setup-seconds', '300']).jobSetupSeconds).toBe(300);
+    expect(() => parseCliOptions(['--shards', '2', '--job-setup-seconds', '300'])).toThrow('--job-setup-seconds applies only to --ci-plan');
+    for (const bad of ['0', '3601', '1.5', 'x']) {
+      expect(() => parseCliOptions(['--ci-plan', 'p.json', '--job-setup-seconds', bad])).toThrow('--job-setup-seconds needs a whole number of seconds from 1 to 3600');
+    }
+  });
 });