From fe8fcab8c27f42b1f82136ea070e99aa925e6988 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 28 Aug 2026 19:59:06 +0000 Subject: [PATCH] =?UTF-8?q?docs(testing):=20sandbox-doctor=20=E2=80=94=20o?= =?UTF-8?q?ne=20command=20makes=20a=20cloud=20sandbox=20run=20the=20suite?= =?UTF-8?q?=20green?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured failure taxonomy for Vercel/Conductor sandboxes (missing /dev/fd, 64M /dev/shm, seccomp-supervisor access(2) EACCES under load, uid-1000 processes with FULL capabilities defeating chmod-denial tests, no X server, no git identity, Conductor git-shim exit-code laundering) plus the idempotent script that treats all of it and seeds the run recipe. Co-Authored-By: Claude Fable 5 --- docs/TESTING_INTERNALS.md | 20 +++++++++ scripts/sandbox-doctor.sh | 93 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 113 insertions(+) create mode 100755 scripts/sandbox-doctor.sh diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index bbcdd721b..bdbe0ed33 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -41,3 +41,23 @@ E2E tests stream progress in real-time (tool-by-tool via `--output-format stream fallback `~/.gstack-dev/evals/`) with auto-comparison against the previous finalized run (in-flight `_partial` files are never used as a baseline, so a run can't compare against itself). + +## Cloud sandboxes (Vercel / Conductor cloud workspaces) + +Syscall-supervised sandboxes need environment setup before `bun run test` can +run green: run `scripts/sandbox-doctor.sh` once per boot. It documents and +treats the full failure taxonomy (missing /dev/fd, 64M /dev/shm, spurious +access(2) EACCES from the seccomp supervisor under load, full-capability +processes defeating chmod-denial tests, no X server, no git identity, and +Conductor's git-shim exit-code laundering). Then: + +```bash +setpriv --ambient-caps=-all --bounding-set=-all bun run test +``` + +Two runner knobs exist for these environments (both no-ops unless set): +`GSTACK_FREE_JOBS` caps shard concurrency (2 is the measured sweet spot — one +serial mega-shard and 6-way sharding both saturate the per-process syscall +supervisor), and `GSTACK_FREE_RETRY_FLAKY=1` re-runs attributed failures once +serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so +a broken tree can't masquerade as flaky). diff --git a/scripts/sandbox-doctor.sh b/scripts/sandbox-doctor.sh new file mode 100755 index 000000000..c96c40e08 --- /dev/null +++ b/scripts/sandbox-doctor.sh @@ -0,0 +1,93 @@ +#!/bin/sh +# sandbox-doctor — make a syscall-supervised cloud sandbox (Vercel sandbox / +# Conductor cloud workspace) able to run `bun run test` green. +# +# Root causes this script treats (all measured on a live Vercel sandbox, +# Amazon Linux 2023, PID 1 = sandbox-init with a seccomp filter): +# +# 1. /dev/fd is missing on fresh boots — every bash process substitution +# `<(...)` fails with "/dev/fd/63: No such file or directory". +# 2. /dev/shm is 64M — concurrent Chromium instances crash. +# 3. The seccomp supervisor spuriously fails access(2)-family syscalls for +# BUSY processes: `git init` dies with "Cannot access work tree: +# Permission denied", bun's existsSync returns false for files written +# microseconds earlier (statx succeeds while access fails on the same +# path). Per-process pressure matters: 1 serial mega-shard and 6-way +# sharding both fail hard; 2 shards is the sweet spot. Under blanket +# denial the whole /tmp subtree is denied while $HOME stays clean, so +# tests run with TMPDIR under HOME. +# 4. Every process runs with FULL capabilities (CapEff=1ffffffffff) despite +# uid 1000 — CAP_DAC_OVERRIDE makes chmod-denial tests unfailable. +# Tests must run under `setpriv --ambient-caps=-all --bounding-set=-all`. +# 5. No X server — headed-browser tests (browse handoff) need Xvfb. +# 6. No git identity — fixtures that rely on ambient user.name/email fail. +# 7. Conductor's /conductor/bin/git shim captures $? AFTER its `if` +# construct (POSIX resets it to 0 on a false condition with no else), so +# every push/pull/fetch/clone/ls-remote FAILURE exits 0. Tests that +# inject remote failures (pre-receive hooks) see phantom successes. +# Report upstream via Conductor Help -> Send Feedback; patched locally. +# +# Idempotent. Run once per sandbox boot (or source ~/.bashrc, which this +# script also seeds). Then: +# +# DISPLAY=:99 TMPDIR=$HOME/tmp GSTACK_FREE_JOBS=2 GSTACK_FREE_RETRY_FLAKY=1 \ +# setpriv --ambient-caps=-all --bounding-set=-all bun run test +set -eu + +say() { printf 'sandbox-doctor: %s\n' "$1"; } + +# 1. /dev/fd +if [ ! -e /dev/fd ]; then + sudo ln -sfn /proc/self/fd /dev/fd + say 'restored /dev/fd -> /proc/self/fd' +fi + +# 2. /dev/shm size +if [ "$(df -k /dev/shm 2>/dev/null | awk 'NR==2 {print $2}')" -lt 1048576 ]; then + sudo mount -o remount,size=4G /dev/shm + say 'remounted /dev/shm at 4G' +fi + +# 3. TMPDIR under HOME (persisted via bashrc below; created here) +mkdir -p "$HOME/tmp" + +# 5. Xvfb for headed-browser tests +command -v Xvfb >/dev/null 2>&1 || sudo dnf install -y xorg-x11-server-Xvfb >/dev/null +pgrep -x Xvfb >/dev/null 2>&1 || { Xvfb :99 -screen 0 1280x800x24 >/dev/null 2>&1 & say 'started Xvfb on :99'; } + +# 6. git identity (only if absent — never clobber a configured one) +git config --global user.name >/dev/null 2>&1 || { + git config --global user.name "$(whoami)" + git config --global user.email "$(whoami)@localhost" + say 'seeded global git identity' +} + +# 7. Conductor git-shim exit-code bug +if [ -f /conductor/bin/git ] && grep -q '^status=\$?' /conductor/bin/git 2>/dev/null; then + sudo python3 - <<'EOF' +src = open('/conductor/bin/git').read() +old = 'exit 0\nfi\nstatus=$?' +new = 'exit 0\nelse\n\tstatus=$?\nfi' +if old in src: + open('/conductor/bin/git', 'w').write(src.replace(old, new)) + print('sandbox-doctor: patched /conductor/bin/git exit-code laundering') +EOF +fi + +# Persist the env recipe for interactive shells. +if ! grep -q 'GSTACK sandbox test env' "$HOME/.bashrc" 2>/dev/null; then + cat >> "$HOME/.bashrc" <<'EOF' + +# GSTACK sandbox test env (written by scripts/sandbox-doctor.sh) +export TMPDIR="$HOME/tmp" +export GSTACK_FREE_JOBS=2 +export GSTACK_FREE_RETRY_FLAKY=1 +export DISPLAY=:99 +[ -e /dev/fd ] || sudo ln -sfn /proc/self/fd /dev/fd 2>/dev/null +pgrep -x Xvfb >/dev/null 2>&1 || (Xvfb :99 -screen 0 1280x800x24 >/dev/null 2>&1 &) +EOF + say 'seeded ~/.bashrc test env' +fi + +say 'done. run tests with:' +say ' setpriv --ambient-caps=-all --bounding-set=-all bun run test'