The Claude agent inside browse E2E tests sometimes runs
`pkill -f "browse"` when the browse server doesn't respond.
This matches the bun test process name (which contains
"skill-e2e-browse" in its args), killing the entire test runner.
Rename skill-e2e-browse.test.ts → skill-e2e-bws.test.ts so
`pkill -f "browse"` no longer matches the parent process.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Browse E2E tests launch concurrent Claude sessions + Playwright + browse
server. The standard-2 (2 vCPU / 8GB) container was getting OOM-killed
~30s in. Upgrade to standard-8 (8 vCPU / 32GB) for browse tests only —
all other suites stay on standard-2.
Uses matrix.suite.runner with a default fallback so only browse tests
get the bigger runner.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
shellcheck disable directives in GitHub Actions run blocks only cover
the next command, not the entire script. Quote $COMMENT_ID and PR
number variables directly instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The SC2086 disable only covered the first command — the `for f in $RESULTS`
loop and printf-style string building triggered SC2086 and SC2059 warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move actionlint.yaml to .github/ where rhysd/actionlint Docker action finds it
- Move shellcheck disable=SC2086 to top of script block (covers both loops)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enable within-file E2E test concurrency for 3x faster runs
Switch all E2E tests from serial test() to testConcurrentIfSelected()
so tests within each file run in parallel. Wall clock drops from ~18min
to ~6min (limited by the longest single test, not sequential sum).
The concurrent helper was already built in e2e-helpers.ts but never
wired up. Each test runs in its own describe block with its own
beforeAll/tmpdir — no shared state conflicts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add CI eval workflow on Ubicloud runners
Single-job GitHub Actions workflow that runs E2E evals on every PR using
Ubicloud runners ($0.006/run — 10x cheaper than GitHub standard). Uses
EVALS_CONCURRENCY=40 with the new within-file concurrency for ~6min
wall clock. Downloads previous eval artifact from main for comparison,
uploads results, and posts a PR comment with pass/fail + cost.
Ubicloud setup required: connect GitHub repo via ubicloud.com dashboard,
add ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY as repo secrets.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.11.6.0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: optimize CI eval PR comment — aggregate all suites, update-not-duplicate
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: parallelize CI evals — 12 runners (1 per suite) for ~3min wall clock
Matrix strategy spins up 12 ubicloud-standard-2 runners simultaneously,
one per test file. Separate report job aggregates all artifacts into a
single PR comment. Bun dependency cache cuts install from ~30s to ~3s.
Runner cost: ~$0.048 (from $0.024) — negligible vs $3-4 API costs.
Wall clock: ~3-4min (from ~8min).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add Docker CI image with pre-baked toolchain + deps
Dockerfile.ci pre-installs bun, node, claude CLI, gh CLI, and
node_modules so eval runners skip all setup. Image rebuilds weekly
and on lockfile/Dockerfile changes via ci-image.yml.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: parallelize CI evals — 12 runners (1 per suite) for ~3min wall clock
Switch eval workflow to use Docker container image with pre-baked
toolchain. Each of 12 matrix runners pulls the image, hardlinks
cached node_modules, builds browse, and runs one test suite.
Setup drops from ~70s to ~19s per runner. Wall clock is dominated
by the slowest individual test, not sequential sum.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: self-bootstrapping CI — build Docker image inline, cache by content hash
Move Docker image build into the evals workflow as a dependency job.
Image tag is keyed on hash of Dockerfile+lockfile+package.json — only
rebuilds when those change. Eliminates chicken-and-egg problem where
the image must exist before the first PR run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: bun.lockb → bun.lock + auth before manifest check
This project uses bun.lock (text format), not bun.lockb (binary).
Also move Docker login before manifest inspect so GHCR auth works.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: bun.lock is gitignored — use package.json only for Docker cache
bun.lock is in .gitignore so it doesn't exist after checkout.
Dockerfile and workflows now use package.json only for deps caching.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: symlink node_modules instead of hardlink (cross-device)
Docker image layers and workspace are on different filesystems,
so cp -al (hardlink) fails. Use ln -s (symlink) instead — zero
copy overhead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* debug: add claude CLI smoke test step to diagnose exit_code_1
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: retrigger eval workflow
* ci: add workflow_dispatch trigger for manual runs
* debug: more verbose claude CLI diagnostics
* fix: run eval container as non-root — claude CLI rejects --dangerously-skip-permissions as root
Claude Code CLI blocks --dangerously-skip-permissions when running
as uid=0 for security. Add a 'runner' user to the Docker image and
set --user runner on the container.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: install bun to /usr/local so non-root runner user can access it
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: unset CI/GITHUB_ACTIONS env vars for eval runs
Claude CLI routing behavior changes when CI=true — it skips skill
invocation and uses Bash directly. Unsetting these markers makes
Claude behave like a local environment for consistent eval results.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* revert: remove CI env unset — didn't fix routing
Unsetting CI/GITHUB_ACTIONS didn't improve routing test results
(still 1/11 in container). The issue is model behavior in
containerized environments, not env vars. Routing tests will be
tracked as a known CI gap.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: copy CLAUDE.md into routing test tmpDirs for skill context
In containerized CI, Claude lacks the project context (CLAUDE.md)
that guides routing decisions locally. Without it, Claude answers
directly with Bash/Agent instead of invoking specific skills.
Copying CLAUDE.md gives Claude the same context it has locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: routing tests use createRoutingWorkDir with full project context
Routing tests now copy CLAUDE.md, README.md, package.json, ETHOS.md,
and all SKILL.md files into each test tmpDir. This gives Claude the
same project context it has locally, which is needed for correct
skill routing decisions in containerized CI environments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: install skills at top-level .claude/skills/ for CI discovery
Claude Code discovers project skills from .claude/skills/<name>/SKILL.md
at the top level only. Nesting under .claude/skills/gstack/<name>/ caused
Claude to see only one "gstack" skill instead of individual skills like
/ship, /qa, /review. This explains 10/11 routing failures in CI — Claude
invoked "gstack" or used Bash directly instead of routing to specific skills.
Also adds workflow_dispatch trigger and --user runner container option.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.11.10.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: CI report needs checkout + routing needs user-level skill install
Two fixes:
1. Report job: add actions/checkout so `gh pr comment` has git context.
Also add pull-requests:write permission for comment posting.
2. Routing tests: install skills to BOTH project-level (.claude/skills/)
AND user-level (~/.claude/skills/) since Claude Code discovers from
both locations. In CI containers, $HOME differs from workdir.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Garry Tan