fix(cso): bounded 900 s ceiling for preparation dependency commands

- PREPARATION_COMMAND_TIMEOUT_MS (900 s) applies only to acquisition and
  offline dependency commands; everything else keeps COMMAND_TIMEOUT_MS
  (300 s), and both are cut to the journey deadline
- prepared-tree export refusals fail with PREPARED_EXPORT_REJECTED and the
  helper's fixed reason or errno, never target paths, with no retry
- free tests pin the ceiling, its callers, and the rejection parser

CHANGELOG.md

@@ -2,9 +2,9 @@
 
 ## [1.91.28.0] - 2026-10-05
 
-**Native runtime qualification can pass for Node, Bun and Python. Rails gets past every earlier failure, but building its native gems from source still takes longer than the five-minute command limit.**
+**Native runtime qualification can pass for Node, Bun, Python and Rails.**
 
-Main run 37345275258 of `cso-runtime-images.yml` staged every image, but eight `qualify-native` rows failed. Two causes showed up in the logs, and six more were hiding behind them. All eight are fixed below. Each was reproduced on a local Docker daemon against freshly built images.
+Main run 37345275258 of `cso-runtime-images.yml` staged every image, but eight `qualify-native` rows failed. Two causes showed up in the logs, and more were hiding behind them. Every cause below is fixed, and each was reproduced on a local amd64 Docker daemon against freshly built images.
 
 - **Promotion evidence is recomputed for injected runtimes.** The cold-start test put the staged runtimes into its test catalog without re-binding `promotion.evidenceDigest`, so Bun, Python and Rails failed with `RUNTIME_PROMOTION_EVIDENCE_MISMATCH`. A shared `installStagedRuntime` helper re-binds the digest, and the test now validates the catalog it builds.
 - **npm dependency acquisition works.** npm 11 refuses to load the same file as both its user and global config. The preparation commands passed `/opt/cso/empty-config` for both, so every Node acquisition failed. The global config now points at its own empty file, `/opt/cso/empty-globalconfig`, which the Node image creates.
@@ -15,8 +15,10 @@ Main run 37345275258 of `cso-runtime-images.yml` staged every image, but eight `
 - **Gem acquisition is batched.** Each `gem fetch` process spends about 11 seconds loading the RubyGems index, so acquisition now runs one `gem fetch` per platform for all exact gems.
 - **Rails commands wait for the source copy.** `db:prepare` ran while `run-app` was still copying the prepared tree, and failed with "Could not locate Gemfile". Rails application and test containers now start through `startHeldApplication`, which waits until the copy has finished.
 - **Test fixtures.** The Python cold fixture called a method that `unittest.TestCase` does not have, and the Rails cold fixture had no `bin/rails`, so `rails server` printed the `rails new` help instead of booting.
+- **Dependency installs get a bounded longer ceiling.** The Rails fixture's lockfile lists only the `ruby` platform, so `bundle install --local` compiles nokogiri, sqlite3 and pg. At the app container's 0.85 CPU share that takes about 385 seconds, past the 300-second limit on one command. Dependency fetch and install commands in the preparation phases now get `PREPARATION_COMMAND_TIMEOUT_MS` (900 seconds). Every other command keeps 300 seconds, both stay inside the journey deadline, and a free test pins which calls can ask for the longer ceiling. The CPU share and `ISOLATION_POLICY_HASH` are unchanged.
+- **Export refusals name their reason.** A prepared-tree export the helper refuses now fails with `PREPARED_EXPORT_REJECTED`, the helper's exit status and its fixed reason or filesystem errno. Target paths are never included, and nothing retries. One unexplained refusal was seen locally in 20 exports, so a recurrence on CI will now say why.
 
-The images change (`/opt/cso/preparation` and the Node recipe), so qualification needs a fresh staging run on main. The Rails fixture's lockfile lists only the `ruby` platform, so `bundle install --local` compiles nokogiri, sqlite3 and pg. At the app container's 0.85 CPU share that takes about 385 seconds, which is longer than the 300-second limit on a single command. Rails stays red until that limit or the CPU share changes.
+The images change (`/opt/cso/preparation` and the Node recipe), so qualification needs a fresh staging run on main.
 
 ## [1.91.27.0] - 2026-10-05
 

docs/cso-release-qualification.md

@@ -107,6 +107,25 @@ copies the source; `db:prepare` starts only once that copy has finished.
 Changes to `/opt/cso/preparation` or an image recipe need a new staging run
 before native qualification can pass.
 
+Each supervised command has a 300-second ceiling (`COMMAND_TIMEOUT_MS` in
+`lib/cso/process.ts`). Only the dependency fetch and install commands of the
+acquisition and offline preparation phases get `PREPARATION_COMMAND_TIMEOUT_MS`,
+900 seconds. A Rails lockfile that lists only the `ruby` platform compiles
+nokogiri, sqlite3 and pg offline, which takes about 385 seconds at the app
+role's 0.85 CPU share. Both ceilings are cut to the time left before the
+journey's aggregate deadline, so the longer ceiling never extends a journey.
+Helper probes, exports, application starts, `db:prepare` and project tests keep
+the 300-second ceiling. A free test in `test/cso-docker-mounts.test.ts` pins
+which calls may request the longer ceiling. The CPU share is unchanged, so
+`ISOLATION_POLICY_HASH` and the profiles bound to it are unaffected.
+
+When the in-image helper refuses to export a prepared tree, preparation fails
+with error code `PREPARED_EXPORT_REJECTED` instead of the generic
+`TOOL_FAILED`. The message carries the helper's exit status and its fixed
+reason, such as `prepared tree contains a hard-linked file`, or the filesystem
+errno and syscall. It never includes target paths. Nothing retries. Treat a
+recurrence as evidence for a defect, not as a flake to rerun.
+
 For hashed Python requirements resolved by pip, the broker records the requested
 index URL, allowed contacted hosts, and verified wheel hash. Pip's internal
 `files.pythonhosted.org` response URL is opaque to that TLS boundary, so

lib/cso/contracts.ts

@@ -24,7 +24,8 @@ export type ErrorCode =
   | 'CANCELLED'
   | 'PREREQUISITE'
   | 'INCOMPATIBLE_INPUT'
-  | 'ASSERTION_FAILED';
+  | 'ASSERTION_FAILED'
+  | 'PREPARED_EXPORT_REJECTED';
 export class CsoError extends Error {
   constructor(
     public code: ErrorCode,

lib/cso/docker.ts

@@ -4,7 +4,7 @@ import { canonical, CsoError, MAX_OUTPUT, sha256 } from './contracts';
 import { spawn } from 'node:child_process';
 import { dirname } from 'node:path';
 import { GROUP_LIMITS, Role, ROLE_LIMITS, Lease, admit, markSupervised, release, total } from './admission';
-import { childEnvironment, executable, runProcess } from './process';
+import { childEnvironment, commandTimeoutMs, executable, runProcess } from './process';
 import { secureDirectory } from './state';
 export const CONTAINER_SHM_BYTES = 8 * 1024 * 1024;
 export const ISOLATION_POLICY_HASH = sha256(
@@ -403,6 +403,19 @@ export function validateSingleContainerProcessOutput(output: string): void {
       'Offline lifecycle left a background process; prepared output was withheld',
     );
 }
+/**
+ * Names why the in-image export helper refused a prepared tree. Only the
+ * helper's fixed "prepared ..." messages and filesystem errno/syscall
+ * names pass through, so target-controlled paths never reach the error.
+ */
+export function preparedExportRejection(stderr: string): string {
+  const reason = stderr.match(/^error: (prepared [A-Za-z ,-]+)$/m)?.[1];
+  if (reason) return reason;
+  const errno = stderr.match(/^\s*code: "(E[A-Z]+)",?$/m)?.[1],
+    syscall = stderr.match(/^\s*syscall: "([a-z]+)",?$/m)?.[1];
+  if (errno) return `filesystem error ${errno}${syscall ? ` during ${syscall}` : ''}`;
+  return 'unrecognized helper failure';
+}
 export function heldApplicationReady(output: string): boolean {
   const lines = output
     .split('\n')
@@ -888,7 +901,12 @@ export class DockerGroup {
   async execCapture(
     id: string,
     command: string[],
-    options: { workdir?: string; env?: Record<string, string>; redaction?: 'withhold' | 'splice' } = {},
+    options: {
+      workdir?: string;
+      env?: Record<string, string>;
+      redaction?: 'withhold' | 'splice';
+      preparationCommand?: true;
+    } = {},
   ): Promise<{ code: number; stdout: string; stderr: string }> {
     if (!command.length || !command[0].startsWith('/'))
       throw new CsoError('INVALID_SCHEMA', 'Exec needs an absolute executable');
@@ -896,7 +914,7 @@ export class DockerGroup {
       throw new CsoError('INVALID_SCHEMA', 'Exec working directory is outside the preparation contract');
     if (this.remainingOutput <= 0)
       throw new CsoError('REDACTION_FAILED', 'Reproduction-group output budget is exhausted');
-    const remaining = Math.max(1, Math.min(300_000, this.deadline - Date.now()));
+    const remaining = commandTimeoutMs(this.deadline, options.preparationCommand ? 'preparation' : 'command');
     const args = ['exec'];
     if (options.workdir) args.push('--workdir', options.workdir);
     for (const [key, value] of Object.entries(options.env ?? {})) {
@@ -911,6 +929,7 @@ export class DockerGroup {
       timeoutMs: remaining,
       maxBytes: this.remainingOutput,
       redaction: options.redaction,
+      preparationCommand: options.preparationCommand,
     });
     this.remainingOutput = Math.max(0, this.remainingOutput - r.capturedBytes);
     if (r.timedOut) throw new CsoError('DEADLINE', 'Target command exceeded the reproduction deadline');

lib/cso/preparation-docker.ts

@@ -7,7 +7,7 @@ import { spawn } from 'node:child_process';
 import { dirname, isAbsolute, join, resolve, sep } from 'node:path';
 import { atomicWriteSync } from '../fs-atomic';
 import { canonical, CsoError, sha256 } from './contracts';
-import { DockerGroup, type DockerEndpoint } from './docker';
+import { DockerGroup, preparedExportRejection, type DockerEndpoint } from './docker';
 import { secureDirectory } from './state';
 import {
   admittedPreparationRuntime,
@@ -1235,7 +1235,12 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner
     group: DockerGroup,
     id: string,
     command: string[],
-    options: { workdir?: string; env?: Record<string, string> } = {},
+    options: {
+      workdir?: string;
+      env?: Record<string, string>;
+      preparationCommand?: true;
+      redaction?: 'splice';
+    } = {},
   ) {
     const forbidden = new Set([
       'BUN_OPTIONS',
@@ -1262,7 +1267,11 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner
         .map(([key, value]) => `${key}=${value}`),
       ...command,
     ];
-    return group.execCapture(id, argv, { workdir: options.workdir });
+    return group.execCapture(id, argv, {
+      workdir: options.workdir,
+      preparationCommand: options.preparationCommand,
+      redaction: options.redaction,
+    });
   }
 
   async acquire(request: PreparationAcquireRequest): Promise<AcquisitionReceipt> {
@@ -1364,6 +1373,7 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner
           result = await this.execClean(group, container, [command.executable, ...command.args], {
             workdir: command.cwd,
             env: { ...command.env, ...proxy },
+            preparationCommand: true,
           });
         broker.assertClean();
         commandResults.push(commandReceipt(command, index, result.code));
@@ -1580,6 +1590,7 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner
           result = await this.execClean(group, app, [command.executable, ...command.args], {
             workdir: command.cwd,
             env: command.env,
+            preparationCommand: true,
           });
         commands.push(commandReceipt(command, index, result.code));
         if (result.code !== 0)
@@ -1589,15 +1600,23 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner
       if (finalized.code !== 0) fail('TOOL_FAILED', 'Offline preparation scratch cleanup failed');
       await group.assertOnlyInitProcess(app);
       const containerExport = `/work/.gstack-cso-export-${randomBytes(12).toString('hex')}`;
-      const exported = await this.execClean(group, app, [
-        '/opt/cso/preparation',
-        'export-prepared',
-        '/work',
-        containerExport,
-        String(request.limits.writableBytes),
-      ]);
+      const exported = await this.execClean(
+        group,
+        app,
+        [
+          '/opt/cso/preparation',
+          'export-prepared',
+          '/work',
+          containerExport,
+          String(request.limits.writableBytes),
+        ],
+        { redaction: 'splice' },
+      );
       if (exported.code !== 0)
-        fail('TOOL_FAILED', 'Qualified prepared-tree export rejected offline application output');
+        fail(
+          'PREPARED_EXPORT_REJECTED',
+          `Qualified prepared-tree export rejected offline application output (exit ${exported.code}: ${preparedExportRejection(exported.stderr)})`,
+        );
       await group.assertOnlyInitProcess(app);
       await group.pause(app);
       const inertExport = secureDirectory(join(executionCopies, 'prepared-export')),

lib/cso/process.ts

@@ -496,6 +496,26 @@ function hardenGit(file: string, args: string[]): { args: string[]; configs?: Gi
     ],
   };
 }
+/** Ceiling for one supervised child command. */
+export const COMMAND_TIMEOUT_MS = 300_000;
+/**
+ * Ceiling for one dependency fetch or install command inside a preparation
+ * container. Rails lockfiles that pin only the `ruby` platform compile native
+ * gems offline, which takes about 385 s at the app role's CPU share. Every
+ * other command keeps COMMAND_TIMEOUT_MS, and both stay inside the caller's
+ * aggregate deadline.
+ */
+export const PREPARATION_COMMAND_TIMEOUT_MS = 900_000;
+export function commandTimeoutMs(
+  deadline: number,
+  phase: 'command' | 'preparation',
+  now = Date.now(),
+): number {
+  return Math.max(
+    1,
+    Math.min(phase === 'preparation' ? PREPARATION_COMMAND_TIMEOUT_MS : COMMAND_TIMEOUT_MS, deadline - now),
+  );
+}
 export async function runProcess(
   file: string,
   args: string[],
@@ -508,6 +528,8 @@ export async function runProcess(
     raw?: boolean; // Only for inert Git framing or private helper/Docker control JSON that is validated before use. Never print or persist raw results.
     /** `splice` replaces each located sensitive span with a marker instead of withholding both channels. */
     redaction?: 'withhold' | 'splice';
+    /** Allows PREPARATION_COMMAND_TIMEOUT_MS; only preparation dependency commands set it. */
+    preparationCommand?: true;
   },
 ): Promise<ProcessResult> {
   if (!isAbsolute(file) || !isAbsolute(opts.cwd) || !existsSync(opts.cwd))
@@ -545,7 +567,13 @@ export async function runProcess(
         timedOut = true;
         kill();
       },
-      Math.max(1, Math.min(opts.timeoutMs ?? 30_000, 300_000)),
+      Math.max(
+        1,
+        Math.min(
+          opts.timeoutMs ?? 30_000,
+          opts.preparationCommand ? PREPARATION_COMMAND_TIMEOUT_MS : COMMAND_TIMEOUT_MS,
+        ),
+      ),
     );
     const capture = (target: Buffer[]) => (chunk: Buffer) => {
       bytes += chunk.length;

lib/cso/scanner-executor.ts

@@ -378,6 +378,7 @@ function failure(plan: ScannerPlan, error: unknown, version?: string): ScannerOu
     PREREQUISITE: 'PREREQUISITE',
     INCOMPATIBLE_INPUT: 'PREREQUISITE',
     ASSERTION_FAILED: 'TOOL_FAILED',
+    PREPARED_EXPORT_REJECTED: 'TOOL_FAILED',
   };
   const code = codes[e.code];
   return {

test/cso-docker-mounts.test.ts

@@ -2,7 +2,9 @@ import { afterEach, describe, expect, spyOn, test } from 'bun:test';
 import * as fs from 'node:fs';
 import * as os from 'node:os';
 import * as path from 'node:path';
-import { CONTAINER_SHM_BYTES, DockerGroup, type ContainerSpec } from '../lib/cso/docker';
+import { CONTAINER_SHM_BYTES, DockerGroup, preparedExportRejection, type ContainerSpec } from '../lib/cso/docker';
+import * as processModule from '../lib/cso/process';
+import { COMMAND_TIMEOUT_MS, PREPARATION_COMMAND_TIMEOUT_MS, commandTimeoutMs } from '../lib/cso/process';
 
 const roots: string[] = [];
 const restores: Array<() => void> = [];
@@ -142,3 +144,59 @@ describe.skipIf(process.platform === 'win32')('CSO Docker nonrecursive bind moun
     expect(fs.existsSync(path.join(f.root, 'resources.journal'))).toBe(false);
   });
 });
+
+describe('CSO preparation command ceiling', () => {
+  test('only the preparation phase gets the longer ceiling, and neither outlives the deadline', () => {
+    const now = 1_000_000, later = now + 2 * 60 * 60_000;
+    expect(PREPARATION_COMMAND_TIMEOUT_MS).toBe(900_000);
+    expect(COMMAND_TIMEOUT_MS).toBe(300_000);
+    expect(commandTimeoutMs(later, 'command', now)).toBe(COMMAND_TIMEOUT_MS);
+    expect(commandTimeoutMs(later, 'preparation', now)).toBe(PREPARATION_COMMAND_TIMEOUT_MS);
+    expect(commandTimeoutMs(now + 60_000, 'preparation', now)).toBe(60_000);
+    expect(commandTimeoutMs(now + 60_000, 'command', now)).toBe(60_000);
+    expect(commandTimeoutMs(now - 1, 'preparation', now)).toBe(1);
+  });
+
+  test('execCapture passes the preparation ceiling to the child only when asked, bounded by the group deadline', async () => {
+    const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cso-t-'));
+    roots.push(root);
+    const seen: Array<{ timeoutMs?: number; preparationCommand?: true }> = [];
+    const run = spyOn(processModule, 'runProcess').mockImplementation(async (_file, _args, opts) => {
+      seen.push({ timeoutMs: opts.timeoutMs, preparationCommand: opts.preparationCommand });
+      return { code: 0, stdout: '', stderr: '', timedOut: false, truncated: false, capturedBytes: 0 };
+    });
+    restores.push(() => run.mockRestore());
+    const group = (deadline: number) =>
+      new (DockerGroup as any)({ executable: '/usr/bin/docker' }, 'timeout-ceiling', root, deadline, {}) as DockerGroup;
+    const far = group(Date.now() + 2 * 60 * 60_000), near = group(Date.now() + 60_000);
+    await far.execCapture('x', ['/bin/true']);
+    await far.execCapture('x', ['/bin/true'], { preparationCommand: true });
+    await near.execCapture('x', ['/bin/true'], { preparationCommand: true });
+    expect(seen[0]).toEqual({ timeoutMs: COMMAND_TIMEOUT_MS, preparationCommand: undefined });
+    expect(seen[1]).toEqual({ timeoutMs: PREPARATION_COMMAND_TIMEOUT_MS, preparationCommand: true });
+    expect(seen[2].preparationCommand).toBe(true);
+    expect(seen[2].timeoutMs!).toBeLessThanOrEqual(60_000);
+  });
+
+  test('only the acquisition and offline dependency command loops request the preparation ceiling', () => {
+    const directory = path.join(import.meta.dir, '../lib/cso');
+    const uses = fs.readdirSync(directory).filter(name => name.endsWith('.ts')).flatMap(name => {
+      const lines = fs.readFileSync(path.join(directory, name), 'utf8').split('\n');
+      return lines.flatMap((line, index) => line.includes('preparationCommand: true') ? [{ name, context: lines.slice(Math.max(0, index - 4), index).join('\n') }] : []);
+    });
+    expect(uses.map(use => use.name)).toEqual(['preparation-docker.ts', 'preparation-docker.ts']);
+    for (const use of uses) expect(use.context).toContain('[command.executable, ...command.args]');
+  });
+});
+
+describe('CSO prepared export rejection reason', () => {
+  test('names the helper reason or errno and never echoes paths', () => {
+    expect(preparedExportRejection('4 |   throw new Error("prepared tree contains a hard-linked file");\n            ^\nerror: prepared tree contains a hard-linked file\n      at f (/$bunfs/root/preparation:4:9)\n'))
+      .toBe('prepared tree contains a hard-linked file');
+    expect(preparedExportRejection("ENOENT: no such file or directory, lstat '/work/secret-name'\n    path: \"/work/secret-name\",\n syscall: \"lstat\",\n   errno: -2,\n    code: \"ENOENT\"\n"))
+      .toBe('filesystem error ENOENT during lstat');
+    expect(preparedExportRejection('error: prepared /work/secret-name\n')).toBe('unrecognized helper failure');
+    expect(preparedExportRejection('error: prepared tree contains a FIFO, socket, device, or other special object\n')).toBe('prepared tree contains a FIFO, socket, device, or other special object');
+    expect(preparedExportRejection('')).toBe('unrecognized helper failure');
+  });
+});