Running as root breaks Claude CLI (refuses to start). Running as runner
breaks bun (can't write to root-owned /tmp dirs from Docker build).
Fix: run as --user runner, but redirect BUN_TMPDIR and TMPDIR to
/home/runner/.cache/bun which is writable by the runner user.
GITHUB_ENV exports apply to all subsequent steps.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GH Actions overrides Dockerfile USER and HOME, creating permission
conflicts no matter what we set. Running as root (the GH default for
container jobs) gives bun full /tmp access. Claude CLI already uses
--dangerously-skip-permissions in the session runner.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The --user runner container option doesn't set up the user environment
properly — bun can't write temp files even with TMPDIR overrides.
Switch to USER runner in the Dockerfile which properly sets HOME and
creates the user context. Also pre-create ~/.bun owned by runner.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bun's tempdir AccessDenied persists because the container /tmp is
root-owned. Fix at both layers:
1. Dockerfile: chmod 1777 /tmp during build
2. Workflow: chmod + TMPDIR/BUN_TMPDIR fallback at runtime
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bun fails with "unable to write files to tempdir: AccessDenied" when
the container user doesn't own /tmp. This cascades to Playwright
(can't launch Chromium) and browse (server won't start).
Fix: create writable temp dirs at job start. If /tmp isn't writable,
fall back to $HOME/tmp via TMPDIR.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two issues preventing browse E2E from working in CI:
1. Playwright installed Chromium as root but container runs as runner —
browser binaries were inaccessible. Fix: set PLAYWRIGHT_BROWSERS_PATH
to /opt/playwright-browsers and chmod a+rX.
2. Browse binary needs ~/.gstack/ writable for server lock files.
Fix: pre-create /home/runner/.gstack/ owned by runner.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Browse E2E tests (browse basic, browse snapshot) need Playwright +
Chromium to render pages. The CI container didn't have a browser
installed, so the agent spent all turns trying to start the browse
server and failing.
Adds Playwright system deps + Chromium browser to the Docker image.
~400MB image size increase but enables full browse test coverage in CI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enable within-file E2E test concurrency for 3x faster runs
Switch all E2E tests from serial test() to testConcurrentIfSelected()
so tests within each file run in parallel. Wall clock drops from ~18min
to ~6min (limited by the longest single test, not sequential sum).
The concurrent helper was already built in e2e-helpers.ts but never
wired up. Each test runs in its own describe block with its own
beforeAll/tmpdir — no shared state conflicts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add CI eval workflow on Ubicloud runners
Single-job GitHub Actions workflow that runs E2E evals on every PR using
Ubicloud runners ($0.006/run — 10x cheaper than GitHub standard). Uses
EVALS_CONCURRENCY=40 with the new within-file concurrency for ~6min
wall clock. Downloads previous eval artifact from main for comparison,
uploads results, and posts a PR comment with pass/fail + cost.
Ubicloud setup required: connect GitHub repo via ubicloud.com dashboard,
add ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY as repo secrets.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.11.6.0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: optimize CI eval PR comment — aggregate all suites, update-not-duplicate
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: parallelize CI evals — 12 runners (1 per suite) for ~3min wall clock
Matrix strategy spins up 12 ubicloud-standard-2 runners simultaneously,
one per test file. Separate report job aggregates all artifacts into a
single PR comment. Bun dependency cache cuts install from ~30s to ~3s.
Runner cost: ~$0.048 (from $0.024) — negligible vs $3-4 API costs.
Wall clock: ~3-4min (from ~8min).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add Docker CI image with pre-baked toolchain + deps
Dockerfile.ci pre-installs bun, node, claude CLI, gh CLI, and
node_modules so eval runners skip all setup. Image rebuilds weekly
and on lockfile/Dockerfile changes via ci-image.yml.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: parallelize CI evals — 12 runners (1 per suite) for ~3min wall clock
Switch eval workflow to use Docker container image with pre-baked
toolchain. Each of 12 matrix runners pulls the image, hardlinks
cached node_modules, builds browse, and runs one test suite.
Setup drops from ~70s to ~19s per runner. Wall clock is dominated
by the slowest individual test, not sequential sum.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: self-bootstrapping CI — build Docker image inline, cache by content hash
Move Docker image build into the evals workflow as a dependency job.
Image tag is keyed on hash of Dockerfile+lockfile+package.json — only
rebuilds when those change. Eliminates chicken-and-egg problem where
the image must exist before the first PR run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: bun.lockb → bun.lock + auth before manifest check
This project uses bun.lock (text format), not bun.lockb (binary).
Also move Docker login before manifest inspect so GHCR auth works.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: bun.lock is gitignored — use package.json only for Docker cache
bun.lock is in .gitignore so it doesn't exist after checkout.
Dockerfile and workflows now use package.json only for deps caching.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: symlink node_modules instead of hardlink (cross-device)
Docker image layers and workspace are on different filesystems,
so cp -al (hardlink) fails. Use ln -s (symlink) instead — zero
copy overhead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* debug: add claude CLI smoke test step to diagnose exit_code_1
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: retrigger eval workflow
* ci: add workflow_dispatch trigger for manual runs
* debug: more verbose claude CLI diagnostics
* fix: run eval container as non-root — claude CLI rejects --dangerously-skip-permissions as root
Claude Code CLI blocks --dangerously-skip-permissions when running
as uid=0 for security. Add a 'runner' user to the Docker image and
set --user runner on the container.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: install bun to /usr/local so non-root runner user can access it
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: unset CI/GITHUB_ACTIONS env vars for eval runs
Claude CLI routing behavior changes when CI=true — it skips skill
invocation and uses Bash directly. Unsetting these markers makes
Claude behave like a local environment for consistent eval results.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* revert: remove CI env unset — didn't fix routing
Unsetting CI/GITHUB_ACTIONS didn't improve routing test results
(still 1/11 in container). The issue is model behavior in
containerized environments, not env vars. Routing tests will be
tracked as a known CI gap.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: copy CLAUDE.md into routing test tmpDirs for skill context
In containerized CI, Claude lacks the project context (CLAUDE.md)
that guides routing decisions locally. Without it, Claude answers
directly with Bash/Agent instead of invoking specific skills.
Copying CLAUDE.md gives Claude the same context it has locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: routing tests use createRoutingWorkDir with full project context
Routing tests now copy CLAUDE.md, README.md, package.json, ETHOS.md,
and all SKILL.md files into each test tmpDir. This gives Claude the
same project context it has locally, which is needed for correct
skill routing decisions in containerized CI environments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: install skills at top-level .claude/skills/ for CI discovery
Claude Code discovers project skills from .claude/skills/<name>/SKILL.md
at the top level only. Nesting under .claude/skills/gstack/<name>/ caused
Claude to see only one "gstack" skill instead of individual skills like
/ship, /qa, /review. This explains 10/11 routing failures in CI — Claude
invoked "gstack" or used Bash directly instead of routing to specific skills.
Also adds workflow_dispatch trigger and --user runner container option.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.11.10.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: CI report needs checkout + routing needs user-level skill install
Two fixes:
1. Report job: add actions/checkout so `gh pr comment` has git context.
Also add pull-requests:write permission for comment posting.
2. Routing tests: install skills to BOTH project-level (.claude/skills/)
AND user-level (~/.claude/skills/) since Claude Code discovers from
both locations. In CI containers, $HOME differs from workdir.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Garry Tan