Garry TanandClaude Fable 5
fe8fcab8c2
Measured failure taxonomy for Vercel/Conductor sandboxes (missing /dev/fd, 64M /dev/shm, seccomp-supervisor access(2) EACCES under load, uid-1000 processes with FULL capabilities defeating chmod-denial tests, no X server, no git identity, Conductor git-shim exit-code laundering) plus the idempotent script that treats all of it and seeds the run recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3.6 KiB
Testing internals: env keys, hermetic E2E
Moved verbatim from CLAUDE.md (token-load reduction). Read this before
writing or debugging E2E tests, passing env: to a runner, or touching
test/helpers/hermetic-env.ts.
Env keys in Conductor workspaces. The GSTACK_* env-shim (v1.39.2.0+,
lib/conductor-env-shim.ts) promotes GSTACK_ANTHROPIC_API_KEY /
GSTACK_OPENAI_API_KEY to their canonical names inside gstack's TS binaries.
Tests run through gstack entrypoints inherit this promotion automatically.
Don't echo the key value to stdout, logs, or shell history. The historical
"never pass env: to runAgentSdkTest" rule is retired: the failure was
partial-env replacement (the SDK's Options.env REPLACES the child's entire
environment, so an object without the key broke auth). The runner now always
passes a COMPLETE hermetic env with per-test env: merged last, so per-test
overrides are safe; ambient process.env.ANTHROPIC_API_KEY mutation also
still works (the env builder reads process.env at call time).
Hermetic local E2E (default). Every E2E runner (claude -p, PTY, Agent
SDK, codex, gemini) spawns children through test/helpers/hermetic-env.ts:
allowlist-scrubbed env (operator CONDUCTOR_*, CLAUDE_*, GSTACK_*,
MCP_*, GBRAIN_*, and credentials like GH_TOKEN never reach children),
a fresh seeded CLAUDE_CONFIG_DIR (no operator ~/.claude CLAUDE.md /
MCP servers / skills), a temp GSTACK_HOME, and --strict-mcp-config.
Local eval signal matches CI. Debug against real operator state with
EVALS_HERMETIC=0 (restores the legacy env AND drops the strict-MCP flag).
Per-test env: overrides merge last, so deliberate contamination
(CONDUCTOR_WORKSPACE_PATH, per-test GSTACK_HOME) keeps working. The
hermetic config dir seeds NO skills by default; a PTY test that types a
/skill slash command must pass seedSkills: true to the PTY runner, which
points the child's CLAUDE_CONFIG_DIR at hermeticSkillsConfigDir() — a
seeded registry that symlinks the LIVE working tree's SKILL.md files (by
design: the skills ARE the subject under test; a snapshot would measure stale
copies). Wiring is pinned by test/hermetic-wiring.test.ts (static tripwire),
two gate-tier canaries in test/skill-e2e-hermetic-canary.test.ts, and the
seeding tripwires in test/hermetic-skills-seeding.test.ts /
test/pty-skill-seeding-wiring.test.ts.
E2E tests stream progress in real-time (tool-by-tool via --output-format stream-json --verbose). Results are persisted to ~/.gstack/projects/<slug>/evals/ (legacy
fallback ~/.gstack-dev/evals/) with auto-comparison
against the previous finalized run (in-flight _partial files are never used as
a baseline, so a run can't compare against itself).
Cloud sandboxes (Vercel / Conductor cloud workspaces)
Syscall-supervised sandboxes need environment setup before bun run test can
run green: run scripts/sandbox-doctor.sh once per boot. It documents and
treats the full failure taxonomy (missing /dev/fd, 64M /dev/shm, spurious
access(2) EACCES from the seccomp supervisor under load, full-capability
processes defeating chmod-denial tests, no X server, no git identity, and
Conductor's git-shim exit-code laundering). Then:
setpriv --ambient-caps=-all --bounding-set=-all bun run test
Two runner knobs exist for these environments (both no-ops unless set):
GSTACK_FREE_JOBS caps shard concurrency (2 is the measured sweet spot — one
serial mega-shard and 6-way sharding both saturate the per-process syscall
supervisor), and GSTACK_FREE_RETRY_FLAKY=1 re-runs attributed failures once
serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so
a broken tree can't masquerade as flaky).