diff --git a/CHANGELOG.md b/CHANGELOG.md index 833fba746..9f1b7656f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,9 +2,9 @@ ## [1.91.28.0] - 2026-10-05 -**Native runtime qualification can pass for Node, Bun and Python. Rails gets past every earlier failure, but building its native gems from source still takes longer than the five-minute command limit.** +**Native runtime qualification can pass for Node, Bun, Python and Rails.** -Main run 37345275258 of `cso-runtime-images.yml` staged every image, but eight `qualify-native` rows failed. Two causes showed up in the logs, and six more were hiding behind them. All eight are fixed below. Each was reproduced on a local Docker daemon against freshly built images. +Main run 37345275258 of `cso-runtime-images.yml` staged every image, but eight `qualify-native` rows failed. Two causes showed up in the logs, and more were hiding behind them. Every cause below is fixed, and each was reproduced on a local amd64 Docker daemon against freshly built images. - **Promotion evidence is recomputed for injected runtimes.** The cold-start test put the staged runtimes into its test catalog without re-binding `promotion.evidenceDigest`, so Bun, Python and Rails failed with `RUNTIME_PROMOTION_EVIDENCE_MISMATCH`. A shared `installStagedRuntime` helper re-binds the digest, and the test now validates the catalog it builds. - **npm dependency acquisition works.** npm 11 refuses to load the same file as both its user and global config. The preparation commands passed `/opt/cso/empty-config` for both, so every Node acquisition failed. The global config now points at its own empty file, `/opt/cso/empty-globalconfig`, which the Node image creates. @@ -15,8 +15,10 @@ Main run 37345275258 of `cso-runtime-images.yml` staged every image, but eight ` - **Gem acquisition is batched.** Each `gem fetch` process spends about 11 seconds loading the RubyGems index, so acquisition now runs one `gem fetch` per platform for all exact gems. - **Rails commands wait for the source copy.** `db:prepare` ran while `run-app` was still copying the prepared tree, and failed with "Could not locate Gemfile". Rails application and test containers now start through `startHeldApplication`, which waits until the copy has finished. - **Test fixtures.** The Python cold fixture called a method that `unittest.TestCase` does not have, and the Rails cold fixture had no `bin/rails`, so `rails server` printed the `rails new` help instead of booting. +- **Dependency installs get a bounded longer ceiling.** The Rails fixture's lockfile lists only the `ruby` platform, so `bundle install --local` compiles nokogiri, sqlite3 and pg. At the app container's 0.85 CPU share that takes about 385 seconds, past the 300-second limit on one command. Dependency fetch and install commands in the preparation phases now get `PREPARATION_COMMAND_TIMEOUT_MS` (900 seconds). Every other command keeps 300 seconds, both stay inside the journey deadline, and a free test pins which calls can ask for the longer ceiling. The CPU share and `ISOLATION_POLICY_HASH` are unchanged. +- **Export refusals name their reason.** A prepared-tree export the helper refuses now fails with `PREPARED_EXPORT_REJECTED`, the helper's exit status and its fixed reason or filesystem errno. Target paths are never included, and nothing retries. One unexplained refusal was seen locally in 20 exports, so a recurrence on CI will now say why. -The images change (`/opt/cso/preparation` and the Node recipe), so qualification needs a fresh staging run on main. The Rails fixture's lockfile lists only the `ruby` platform, so `bundle install --local` compiles nokogiri, sqlite3 and pg. At the app container's 0.85 CPU share that takes about 385 seconds, which is longer than the 300-second limit on a single command. Rails stays red until that limit or the CPU share changes. +The images change (`/opt/cso/preparation` and the Node recipe), so qualification needs a fresh staging run on main. ## [1.91.27.0] - 2026-10-05 diff --git a/docs/cso-release-qualification.md b/docs/cso-release-qualification.md index 2721f31eb..5af039395 100644 --- a/docs/cso-release-qualification.md +++ b/docs/cso-release-qualification.md @@ -107,6 +107,25 @@ copies the source; `db:prepare` starts only once that copy has finished. Changes to `/opt/cso/preparation` or an image recipe need a new staging run before native qualification can pass. +Each supervised command has a 300-second ceiling (`COMMAND_TIMEOUT_MS` in +`lib/cso/process.ts`). Only the dependency fetch and install commands of the +acquisition and offline preparation phases get `PREPARATION_COMMAND_TIMEOUT_MS`, +900 seconds. A Rails lockfile that lists only the `ruby` platform compiles +nokogiri, sqlite3 and pg offline, which takes about 385 seconds at the app +role's 0.85 CPU share. Both ceilings are cut to the time left before the +journey's aggregate deadline, so the longer ceiling never extends a journey. +Helper probes, exports, application starts, `db:prepare` and project tests keep +the 300-second ceiling. A free test in `test/cso-docker-mounts.test.ts` pins +which calls may request the longer ceiling. The CPU share is unchanged, so +`ISOLATION_POLICY_HASH` and the profiles bound to it are unaffected. + +When the in-image helper refuses to export a prepared tree, preparation fails +with error code `PREPARED_EXPORT_REJECTED` instead of the generic +`TOOL_FAILED`. The message carries the helper's exit status and its fixed +reason, such as `prepared tree contains a hard-linked file`, or the filesystem +errno and syscall. It never includes target paths. Nothing retries. Treat a +recurrence as evidence for a defect, not as a flake to rerun. + For hashed Python requirements resolved by pip, the broker records the requested index URL, allowed contacted hosts, and verified wheel hash. Pip's internal `files.pythonhosted.org` response URL is opaque to that TLS boundary, so diff --git a/lib/cso/contracts.ts b/lib/cso/contracts.ts index 743537345..485b2918e 100644 --- a/lib/cso/contracts.ts +++ b/lib/cso/contracts.ts @@ -24,7 +24,8 @@ export type ErrorCode = | 'CANCELLED' | 'PREREQUISITE' | 'INCOMPATIBLE_INPUT' - | 'ASSERTION_FAILED'; + | 'ASSERTION_FAILED' + | 'PREPARED_EXPORT_REJECTED'; export class CsoError extends Error { constructor( public code: ErrorCode, diff --git a/lib/cso/docker.ts b/lib/cso/docker.ts index 0a2d2381e..51ce2566b 100644 --- a/lib/cso/docker.ts +++ b/lib/cso/docker.ts @@ -4,7 +4,7 @@ import { canonical, CsoError, MAX_OUTPUT, sha256 } from './contracts'; import { spawn } from 'node:child_process'; import { dirname } from 'node:path'; import { GROUP_LIMITS, Role, ROLE_LIMITS, Lease, admit, markSupervised, release, total } from './admission'; -import { childEnvironment, executable, runProcess } from './process'; +import { childEnvironment, commandTimeoutMs, executable, runProcess } from './process'; import { secureDirectory } from './state'; export const CONTAINER_SHM_BYTES = 8 * 1024 * 1024; export const ISOLATION_POLICY_HASH = sha256( @@ -403,6 +403,19 @@ export function validateSingleContainerProcessOutput(output: string): void { 'Offline lifecycle left a background process; prepared output was withheld', ); } +/** + * Names why the in-image export helper refused a prepared tree. Only the + * helper's fixed "prepared ..." messages and filesystem errno/syscall + * names pass through, so target-controlled paths never reach the error. + */ +export function preparedExportRejection(stderr: string): string { + const reason = stderr.match(/^error: (prepared [A-Za-z ,-]+)$/m)?.[1]; + if (reason) return reason; + const errno = stderr.match(/^\s*code: "(E[A-Z]+)",?$/m)?.[1], + syscall = stderr.match(/^\s*syscall: "([a-z]+)",?$/m)?.[1]; + if (errno) return `filesystem error ${errno}${syscall ? ` during ${syscall}` : ''}`; + return 'unrecognized helper failure'; +} export function heldApplicationReady(output: string): boolean { const lines = output .split('\n') @@ -888,7 +901,12 @@ export class DockerGroup { async execCapture( id: string, command: string[], - options: { workdir?: string; env?: Record; redaction?: 'withhold' | 'splice' } = {}, + options: { + workdir?: string; + env?: Record; + redaction?: 'withhold' | 'splice'; + preparationCommand?: true; + } = {}, ): Promise<{ code: number; stdout: string; stderr: string }> { if (!command.length || !command[0].startsWith('/')) throw new CsoError('INVALID_SCHEMA', 'Exec needs an absolute executable'); @@ -896,7 +914,7 @@ export class DockerGroup { throw new CsoError('INVALID_SCHEMA', 'Exec working directory is outside the preparation contract'); if (this.remainingOutput <= 0) throw new CsoError('REDACTION_FAILED', 'Reproduction-group output budget is exhausted'); - const remaining = Math.max(1, Math.min(300_000, this.deadline - Date.now())); + const remaining = commandTimeoutMs(this.deadline, options.preparationCommand ? 'preparation' : 'command'); const args = ['exec']; if (options.workdir) args.push('--workdir', options.workdir); for (const [key, value] of Object.entries(options.env ?? {})) { @@ -911,6 +929,7 @@ export class DockerGroup { timeoutMs: remaining, maxBytes: this.remainingOutput, redaction: options.redaction, + preparationCommand: options.preparationCommand, }); this.remainingOutput = Math.max(0, this.remainingOutput - r.capturedBytes); if (r.timedOut) throw new CsoError('DEADLINE', 'Target command exceeded the reproduction deadline'); diff --git a/lib/cso/preparation-docker.ts b/lib/cso/preparation-docker.ts index dba444343..e36e0a34d 100644 --- a/lib/cso/preparation-docker.ts +++ b/lib/cso/preparation-docker.ts @@ -7,7 +7,7 @@ import { spawn } from 'node:child_process'; import { dirname, isAbsolute, join, resolve, sep } from 'node:path'; import { atomicWriteSync } from '../fs-atomic'; import { canonical, CsoError, sha256 } from './contracts'; -import { DockerGroup, type DockerEndpoint } from './docker'; +import { DockerGroup, preparedExportRejection, type DockerEndpoint } from './docker'; import { secureDirectory } from './state'; import { admittedPreparationRuntime, @@ -1235,7 +1235,12 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner group: DockerGroup, id: string, command: string[], - options: { workdir?: string; env?: Record } = {}, + options: { + workdir?: string; + env?: Record; + preparationCommand?: true; + redaction?: 'splice'; + } = {}, ) { const forbidden = new Set([ 'BUN_OPTIONS', @@ -1262,7 +1267,11 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner .map(([key, value]) => `${key}=${value}`), ...command, ]; - return group.execCapture(id, argv, { workdir: options.workdir }); + return group.execCapture(id, argv, { + workdir: options.workdir, + preparationCommand: options.preparationCommand, + redaction: options.redaction, + }); } async acquire(request: PreparationAcquireRequest): Promise { @@ -1364,6 +1373,7 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner result = await this.execClean(group, container, [command.executable, ...command.args], { workdir: command.cwd, env: { ...command.env, ...proxy }, + preparationCommand: true, }); broker.assertClean(); commandResults.push(commandReceipt(command, index, result.code)); @@ -1580,6 +1590,7 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner result = await this.execClean(group, app, [command.executable, ...command.args], { workdir: command.cwd, env: command.env, + preparationCommand: true, }); commands.push(commandReceipt(command, index, result.code)); if (result.code !== 0) @@ -1589,15 +1600,23 @@ export class DockerPreparationSandboxRunner implements PreparationSandboxRunner if (finalized.code !== 0) fail('TOOL_FAILED', 'Offline preparation scratch cleanup failed'); await group.assertOnlyInitProcess(app); const containerExport = `/work/.gstack-cso-export-${randomBytes(12).toString('hex')}`; - const exported = await this.execClean(group, app, [ - '/opt/cso/preparation', - 'export-prepared', - '/work', - containerExport, - String(request.limits.writableBytes), - ]); + const exported = await this.execClean( + group, + app, + [ + '/opt/cso/preparation', + 'export-prepared', + '/work', + containerExport, + String(request.limits.writableBytes), + ], + { redaction: 'splice' }, + ); if (exported.code !== 0) - fail('TOOL_FAILED', 'Qualified prepared-tree export rejected offline application output'); + fail( + 'PREPARED_EXPORT_REJECTED', + `Qualified prepared-tree export rejected offline application output (exit ${exported.code}: ${preparedExportRejection(exported.stderr)})`, + ); await group.assertOnlyInitProcess(app); await group.pause(app); const inertExport = secureDirectory(join(executionCopies, 'prepared-export')), diff --git a/lib/cso/process.ts b/lib/cso/process.ts index 6f3b7a77f..15d4d40f3 100644 --- a/lib/cso/process.ts +++ b/lib/cso/process.ts @@ -496,6 +496,26 @@ function hardenGit(file: string, args: string[]): { args: string[]; configs?: Gi ], }; } +/** Ceiling for one supervised child command. */ +export const COMMAND_TIMEOUT_MS = 300_000; +/** + * Ceiling for one dependency fetch or install command inside a preparation + * container. Rails lockfiles that pin only the `ruby` platform compile native + * gems offline, which takes about 385 s at the app role's CPU share. Every + * other command keeps COMMAND_TIMEOUT_MS, and both stay inside the caller's + * aggregate deadline. + */ +export const PREPARATION_COMMAND_TIMEOUT_MS = 900_000; +export function commandTimeoutMs( + deadline: number, + phase: 'command' | 'preparation', + now = Date.now(), +): number { + return Math.max( + 1, + Math.min(phase === 'preparation' ? PREPARATION_COMMAND_TIMEOUT_MS : COMMAND_TIMEOUT_MS, deadline - now), + ); +} export async function runProcess( file: string, args: string[], @@ -508,6 +528,8 @@ export async function runProcess( raw?: boolean; // Only for inert Git framing or private helper/Docker control JSON that is validated before use. Never print or persist raw results. /** `splice` replaces each located sensitive span with a marker instead of withholding both channels. */ redaction?: 'withhold' | 'splice'; + /** Allows PREPARATION_COMMAND_TIMEOUT_MS; only preparation dependency commands set it. */ + preparationCommand?: true; }, ): Promise { if (!isAbsolute(file) || !isAbsolute(opts.cwd) || !existsSync(opts.cwd)) @@ -545,7 +567,13 @@ export async function runProcess( timedOut = true; kill(); }, - Math.max(1, Math.min(opts.timeoutMs ?? 30_000, 300_000)), + Math.max( + 1, + Math.min( + opts.timeoutMs ?? 30_000, + opts.preparationCommand ? PREPARATION_COMMAND_TIMEOUT_MS : COMMAND_TIMEOUT_MS, + ), + ), ); const capture = (target: Buffer[]) => (chunk: Buffer) => { bytes += chunk.length; diff --git a/lib/cso/scanner-executor.ts b/lib/cso/scanner-executor.ts index 0d1f24ffc..db62f05f9 100644 --- a/lib/cso/scanner-executor.ts +++ b/lib/cso/scanner-executor.ts @@ -378,6 +378,7 @@ function failure(plan: ScannerPlan, error: unknown, version?: string): ScannerOu PREREQUISITE: 'PREREQUISITE', INCOMPATIBLE_INPUT: 'PREREQUISITE', ASSERTION_FAILED: 'TOOL_FAILED', + PREPARED_EXPORT_REJECTED: 'TOOL_FAILED', }; const code = codes[e.code]; return { diff --git a/test/cso-docker-mounts.test.ts b/test/cso-docker-mounts.test.ts index 81ccaf778..fdcad2e43 100644 --- a/test/cso-docker-mounts.test.ts +++ b/test/cso-docker-mounts.test.ts @@ -2,7 +2,9 @@ import { afterEach, describe, expect, spyOn, test } from 'bun:test'; import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; -import { CONTAINER_SHM_BYTES, DockerGroup, type ContainerSpec } from '../lib/cso/docker'; +import { CONTAINER_SHM_BYTES, DockerGroup, preparedExportRejection, type ContainerSpec } from '../lib/cso/docker'; +import * as processModule from '../lib/cso/process'; +import { COMMAND_TIMEOUT_MS, PREPARATION_COMMAND_TIMEOUT_MS, commandTimeoutMs } from '../lib/cso/process'; const roots: string[] = []; const restores: Array<() => void> = []; @@ -142,3 +144,59 @@ describe.skipIf(process.platform === 'win32')('CSO Docker nonrecursive bind moun expect(fs.existsSync(path.join(f.root, 'resources.journal'))).toBe(false); }); }); + +describe('CSO preparation command ceiling', () => { + test('only the preparation phase gets the longer ceiling, and neither outlives the deadline', () => { + const now = 1_000_000, later = now + 2 * 60 * 60_000; + expect(PREPARATION_COMMAND_TIMEOUT_MS).toBe(900_000); + expect(COMMAND_TIMEOUT_MS).toBe(300_000); + expect(commandTimeoutMs(later, 'command', now)).toBe(COMMAND_TIMEOUT_MS); + expect(commandTimeoutMs(later, 'preparation', now)).toBe(PREPARATION_COMMAND_TIMEOUT_MS); + expect(commandTimeoutMs(now + 60_000, 'preparation', now)).toBe(60_000); + expect(commandTimeoutMs(now + 60_000, 'command', now)).toBe(60_000); + expect(commandTimeoutMs(now - 1, 'preparation', now)).toBe(1); + }); + + test('execCapture passes the preparation ceiling to the child only when asked, bounded by the group deadline', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cso-t-')); + roots.push(root); + const seen: Array<{ timeoutMs?: number; preparationCommand?: true }> = []; + const run = spyOn(processModule, 'runProcess').mockImplementation(async (_file, _args, opts) => { + seen.push({ timeoutMs: opts.timeoutMs, preparationCommand: opts.preparationCommand }); + return { code: 0, stdout: '', stderr: '', timedOut: false, truncated: false, capturedBytes: 0 }; + }); + restores.push(() => run.mockRestore()); + const group = (deadline: number) => + new (DockerGroup as any)({ executable: '/usr/bin/docker' }, 'timeout-ceiling', root, deadline, {}) as DockerGroup; + const far = group(Date.now() + 2 * 60 * 60_000), near = group(Date.now() + 60_000); + await far.execCapture('x', ['/bin/true']); + await far.execCapture('x', ['/bin/true'], { preparationCommand: true }); + await near.execCapture('x', ['/bin/true'], { preparationCommand: true }); + expect(seen[0]).toEqual({ timeoutMs: COMMAND_TIMEOUT_MS, preparationCommand: undefined }); + expect(seen[1]).toEqual({ timeoutMs: PREPARATION_COMMAND_TIMEOUT_MS, preparationCommand: true }); + expect(seen[2].preparationCommand).toBe(true); + expect(seen[2].timeoutMs!).toBeLessThanOrEqual(60_000); + }); + + test('only the acquisition and offline dependency command loops request the preparation ceiling', () => { + const directory = path.join(import.meta.dir, '../lib/cso'); + const uses = fs.readdirSync(directory).filter(name => name.endsWith('.ts')).flatMap(name => { + const lines = fs.readFileSync(path.join(directory, name), 'utf8').split('\n'); + return lines.flatMap((line, index) => line.includes('preparationCommand: true') ? [{ name, context: lines.slice(Math.max(0, index - 4), index).join('\n') }] : []); + }); + expect(uses.map(use => use.name)).toEqual(['preparation-docker.ts', 'preparation-docker.ts']); + for (const use of uses) expect(use.context).toContain('[command.executable, ...command.args]'); + }); +}); + +describe('CSO prepared export rejection reason', () => { + test('names the helper reason or errno and never echoes paths', () => { + expect(preparedExportRejection('4 | throw new Error("prepared tree contains a hard-linked file");\n ^\nerror: prepared tree contains a hard-linked file\n at f (/$bunfs/root/preparation:4:9)\n')) + .toBe('prepared tree contains a hard-linked file'); + expect(preparedExportRejection("ENOENT: no such file or directory, lstat '/work/secret-name'\n path: \"/work/secret-name\",\n syscall: \"lstat\",\n errno: -2,\n code: \"ENOENT\"\n")) + .toBe('filesystem error ENOENT during lstat'); + expect(preparedExportRejection('error: prepared /work/secret-name\n')).toBe('unrecognized helper failure'); + expect(preparedExportRejection('error: prepared tree contains a FIFO, socket, device, or other special object\n')).toBe('prepared tree contains a FIFO, socket, device, or other special object'); + expect(preparedExportRejection('')).toBe('unrecognized helper failure'); + }); +});