mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-09 22:48:57 +02:00
Garry TanandClaude Fable 5
fe8fcab8c2
Measured failure taxonomy for Vercel/Conductor sandboxes (missing /dev/fd, 64M /dev/shm, seccomp-supervisor access(2) EACCES under load, uid-1000 processes with FULL capabilities defeating chmod-denial tests, no X server, no git identity, Conductor git-shim exit-code laundering) plus the idempotent script that treats all of it and seeds the run recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
64 lines
3.6 KiB
Markdown
64 lines
3.6 KiB
Markdown
# Testing internals: env keys, hermetic E2E
|
|
|
|
Moved verbatim from CLAUDE.md (token-load reduction). Read this before
|
|
writing or debugging E2E tests, passing `env:` to a runner, or touching
|
|
`test/helpers/hermetic-env.ts`.
|
|
|
|
**Env keys in Conductor workspaces.** The `GSTACK_*` env-shim (v1.39.2.0+,
|
|
`lib/conductor-env-shim.ts`) promotes `GSTACK_ANTHROPIC_API_KEY` /
|
|
`GSTACK_OPENAI_API_KEY` to their canonical names inside gstack's TS binaries.
|
|
Tests run through gstack entrypoints inherit this promotion automatically.
|
|
Don't echo the key value to stdout, logs, or shell history. The historical
|
|
"never pass `env:` to `runAgentSdkTest`" rule is retired: the failure was
|
|
partial-env replacement (the SDK's `Options.env` REPLACES the child's entire
|
|
environment, so an object without the key broke auth). The runner now always
|
|
passes a COMPLETE hermetic env with per-test `env:` merged last, so per-test
|
|
overrides are safe; ambient `process.env.ANTHROPIC_API_KEY` mutation also
|
|
still works (the env builder reads process.env at call time).
|
|
|
|
**Hermetic local E2E (default).** Every E2E runner (claude -p, PTY, Agent
|
|
SDK, codex, gemini) spawns children through `test/helpers/hermetic-env.ts`:
|
|
allowlist-scrubbed env (operator `CONDUCTOR_*`, `CLAUDE_*`, `GSTACK_*`,
|
|
`MCP_*`, `GBRAIN_*`, and credentials like `GH_TOKEN` never reach children),
|
|
a fresh seeded `CLAUDE_CONFIG_DIR` (no operator `~/.claude` CLAUDE.md /
|
|
MCP servers / skills), a temp `GSTACK_HOME`, and `--strict-mcp-config`.
|
|
Local eval signal matches CI. Debug against real operator state with
|
|
`EVALS_HERMETIC=0` (restores the legacy env AND drops the strict-MCP flag).
|
|
Per-test `env:` overrides merge last, so deliberate contamination
|
|
(`CONDUCTOR_WORKSPACE_PATH`, per-test `GSTACK_HOME`) keeps working. The
|
|
hermetic config dir seeds NO skills by default; a PTY test that types a
|
|
`/skill` slash command must pass `seedSkills: true` to the PTY runner, which
|
|
points the child's `CLAUDE_CONFIG_DIR` at `hermeticSkillsConfigDir()` — a
|
|
seeded registry that symlinks the LIVE working tree's SKILL.md files (by
|
|
design: the skills ARE the subject under test; a snapshot would measure stale
|
|
copies). Wiring is pinned by `test/hermetic-wiring.test.ts` (static tripwire),
|
|
two gate-tier canaries in `test/skill-e2e-hermetic-canary.test.ts`, and the
|
|
seeding tripwires in `test/hermetic-skills-seeding.test.ts` /
|
|
`test/pty-skill-seeding-wiring.test.ts`.
|
|
|
|
E2E tests stream progress in real-time (tool-by-tool via `--output-format stream-json
|
|
--verbose`). Results are persisted to `~/.gstack/projects/<slug>/evals/` (legacy
|
|
fallback `~/.gstack-dev/evals/`) with auto-comparison
|
|
against the previous finalized run (in-flight `_partial` files are never used as
|
|
a baseline, so a run can't compare against itself).
|
|
|
|
## Cloud sandboxes (Vercel / Conductor cloud workspaces)
|
|
|
|
Syscall-supervised sandboxes need environment setup before `bun run test` can
|
|
run green: run `scripts/sandbox-doctor.sh` once per boot. It documents and
|
|
treats the full failure taxonomy (missing /dev/fd, 64M /dev/shm, spurious
|
|
access(2) EACCES from the seccomp supervisor under load, full-capability
|
|
processes defeating chmod-denial tests, no X server, no git identity, and
|
|
Conductor's git-shim exit-code laundering). Then:
|
|
|
|
```bash
|
|
setpriv --ambient-caps=-all --bounding-set=-all bun run test
|
|
```
|
|
|
|
Two runner knobs exist for these environments (both no-ops unless set):
|
|
`GSTACK_FREE_JOBS` caps shard concurrency (2 is the measured sweet spot — one
|
|
serial mega-shard and 6-way sharding both saturate the per-process syscall
|
|
supervisor), and `GSTACK_FREE_RETRY_FLAKY=1` re-runs attributed failures once
|
|
serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so
|
|
a broken tree can't masquerade as flaky).
|