From 6d7dad245b1d5382b43344a334183b775e8336af Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 15 May 2026 17:26:44 -0700 Subject: [PATCH] docs: gbrain+gstack setup, Conductor env mapping (v1.39.2.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit USING_GBRAIN_WITH_GSTACK.md: new "What you get after setup" section, Path 4 (remote MCP / split-engine), /sync-gbrain workflow stages + watermark mechanics, "Conductor + GSTACK_* env vars" section, env vars table extended, two troubleshooting entries (silent embedding failure and FILE_TOO_LARGE watermark block). CONTRIBUTING.md "Conductor workspaces": new paragraph on the GSTACK_* prefix pattern and the four entry points importing the shim. VERSION 1.39.1.0 → 1.39.2.0 and CHANGELOG entry covering the shim + docs (full release-summary format with before/after table). Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 43 +++++++++++++++++ CONTRIBUTING.md | 2 + USING_GBRAIN_WITH_GSTACK.md | 92 ++++++++++++++++++++++++++++++++++++- VERSION | 2 +- package.json | 2 +- 5 files changed, 138 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cf89b49b2..7286f523d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,48 @@ # Changelog +## [1.39.2.0] - 2026-05-15 + +## **Conductor workspaces wire `GSTACK_*` keys straight into gbrain embeddings and paid evals.** +## **No more sourcing keys from your shell before every paid run.** + +Conductor exposes API keys under a `GSTACK_` prefix so it never collides with whatever the host system has set. The trade-off is that gbrain's embedding pipeline and `@anthropic-ai/claude-agent-sdk` read the canonical `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` names. The new `lib/conductor-env-shim.ts` bridges the gap: it promotes the prefixed form to canonical when canonical is empty. Four TS entry points import the shim as a side effect (`gstack-gbrain-sync.ts`, `gstack-model-benchmark`, `preflight-agent-sdk.ts`, `e2e-helpers.ts`). `USING_GBRAIN_WITH_GSTACK.md` and `CONTRIBUTING.md` document the pattern, plus the checklist for adding the import to new entry points. + +### The numbers that matter + +Source: working-tree verification before commit. Three observable scenarios in a fresh Conductor workspace with only `GSTACK_OPENAI_API_KEY` and `GSTACK_ANTHROPIC_API_KEY` in env. + +| Surface | Before | After | +|---|---|---| +| `/sync-gbrain` embeddings | 50+ lines of `[gbrain] embedding failed for code file ...: OpenAI embedding requires OPENAI_API_KEY`; pages indexed structurally but semantic search degrades to BM25 | 3294 chunks embedded; `gbrain search "browser security canary token"` returns ranked code regions at 0.95 top score | +| `bun run test:evals` | `ANTHROPIC_API_KEY not set, judge requires Anthropic access` from `test/helpers/benchmark-judge.ts:15` before any test runs | Shim promotes at module import; paid evals proceed normally | +| Adding a new paid-API entry point | Manual env mapping every invocation, or every new entry point ships broken inside Conductor | One import line: `import "../lib/conductor-env-shim";` at the top of the file | + +### What this means for Conductor users + +If you run gstack inside Conductor, `/sync-gbrain` embeddings, paid evals, and the agent SDK just work without sourcing keys from your shell. The shim is 15 lines, side-effect-only, and the import is one line per consumer. The new "Conductor + GSTACK_* env vars" section in `USING_GBRAIN_WITH_GSTACK.md` and the updated "Conductor workspaces" block in `CONTRIBUTING.md` cover the pattern so you don't have to reverse-engineer it from a stack trace. + +### Itemized changes + +#### Added + +- `lib/conductor-env-shim.ts` (new, 15 lines) — side-effect IIFE that promotes `GSTACK_FOO_API_KEY` to `FOO_API_KEY` when the canonical name is empty. Currently covers `ANTHROPIC_API_KEY` and `OPENAI_API_KEY`. +- `USING_GBRAIN_WITH_GSTACK.md` "What you get after setup" section — semantic code search + cross-session memory framed as concrete capabilities. +- `USING_GBRAIN_WITH_GSTACK.md` Path 4 (remote gbrain MCP / split-engine) section — covers brain-via-remote-MCP + code-via-local-PGLite, the two engines being independent, when to pick this path. +- `USING_GBRAIN_WITH_GSTACK.md` `/sync-gbrain` workflow section — three stages (code, memory, brain-sync), pre-flight gating on local engine health, watermark + `--skip-failed` mechanics, capability check governing the CLAUDE.md guidance block. +- `USING_GBRAIN_WITH_GSTACK.md` "Conductor + GSTACK_* env vars" section — explains the prefix pattern, lists the four entry points that import the shim, points contributors at `CONTRIBUTING.md`. +- `USING_GBRAIN_WITH_GSTACK.md` troubleshooting entries: "`/sync-gbrain` reports OK but `gbrain search` returns nothing semantic" (embeddings failed silently) and "`gbrain sync` blocked at a commit hash, `FILE_TOO_LARGE`" (5 MB hard limit, fix via `--skip-failed`). + +#### Changed + +- `bin/gstack-gbrain-sync.ts`, `bin/gstack-model-benchmark`, `scripts/preflight-agent-sdk.ts`, `test/helpers/e2e-helpers.ts` — added `import "../lib/conductor-env-shim";` at the top of each. One line each, side-effect-only. +- `USING_GBRAIN_WITH_GSTACK.md` "three paths" → "four paths" header now that Path 4 (remote MCP) is documented as a first-class choice. +- `USING_GBRAIN_WITH_GSTACK.md` environment variables table — added rows for `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GSTACK_OPENAI_API_KEY`, `GSTACK_ANTHROPIC_API_KEY` covering what reads each one and the GSTACK_-prefix fallback. +- `CONTRIBUTING.md` "Conductor workspaces" — new paragraph documenting the `GSTACK_*` prefix injection pattern, the shim file, and the four entry points that already import it. + +#### For contributors + +- New TS entry points that hit Anthropic or OpenAI APIs (paid evals, `claude-agent-sdk`, gbrain embeddings, model benchmarks) should add `import "../lib/conductor-env-shim";` as the first import. Without it, the entry point ships broken inside Conductor even though it works in a bare shell. The contributor checklist in `CONTRIBUTING.md`'s "Conductor workspaces" block names the four entry points already wired up. + ## [1.39.1.0] - 2026-05-15 ## **Plan-mode reviews now enforce a blocking ExitPlanMode gate.** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7f40fa4d8..df635f75a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -333,6 +333,8 @@ When Conductor creates a new workspace, `bin/dev-setup` runs automatically. It d **First-time setup:** Put your `ANTHROPIC_API_KEY` in `.env` in the main repo (see `.env.example`). Every Conductor workspace inherits it automatically. +**`GSTACK_*` env prefix (Conductor-injected keys).** Recent Conductor versions inject `GSTACK_ANTHROPIC_API_KEY` and `GSTACK_OPENAI_API_KEY` into every workspace's process env directly — independent of the `.env` copy path. The prefix prevents collision with any `ANTHROPIC_API_KEY` you set elsewhere on your system. To keep paid evals, `/sync-gbrain` embeddings, and `claude-agent-sdk` calls working in a Conductor workspace, TS entry points import `lib/conductor-env-shim.ts` as a side effect, which promotes `GSTACK_FOO_API_KEY` to `FOO_API_KEY` when the canonical name is empty. If you add a new TS entry point that hits a paid API, add `import "../lib/conductor-env-shim";` to the top of the file. Today the shim is imported from `bin/gstack-gbrain-sync.ts`, `bin/gstack-model-benchmark`, `scripts/preflight-agent-sdk.ts`, and `test/helpers/e2e-helpers.ts`. + ## Things to know - **SKILL.md files are generated.** Edit the `.tmpl` template, not the `.md`. Run `bun run gen:skill-docs` to regenerate. diff --git a/USING_GBRAIN_WITH_GSTACK.md b/USING_GBRAIN_WITH_GSTACK.md index 17dea2b06..ca0f309b6 100644 --- a/USING_GBRAIN_WITH_GSTACK.md +++ b/USING_GBRAIN_WITH_GSTACK.md @@ -16,7 +16,16 @@ This is the full monty: every scenario, every flag, every helper bin, every trou That's it. The skill detects your current state, asks three questions at most, and walks you through install, init, MCP registration for Claude Code, and per-repo trust policy. On a clean Mac with nothing installed it finishes in under five minutes. On a Mac where something's already set up it takes seconds (it detects the existing state and skips done work). -## The three paths +## What you get after setup + +Once `/setup-gbrain` finishes, your coding agent has two retrieval surfaces it didn't have before: + +- **Semantic code search across this repo.** `gbrain search "browser security canary"` returns ranked file regions, not exact-match grep hits. `gbrain code-def`, `code-refs`, `code-callers`, `code-callees` walk the call graph by symbol — useful when you don't know which file holds the implementation but you know what it does. The agent prefers these over Grep when the question is semantic; CLAUDE.md gets a `## GBrain Search Guidance` block that teaches it the routing rules. +- **Cross-session memory.** Plans, retros, decisions, and learnings from past sessions live in `~/.gstack/` and (if you opted in to artifacts sync) get pushed to a private git repo that gbrain indexes. `gbrain search "what did we decide about auth?"` actually finds the prior CEO plan instead of you re-describing context every session. + +If you also enabled remote MCP (Path 4 below), brain queries route to a shared brain server that other machines can write to — your laptop, your desktop, and a teammate's machine all see the same memory. + +## The four paths You pick one when the skill asks "Where should your brain live?" @@ -52,6 +61,19 @@ Best for: try-it-first, no account, no cloud, no sharing. Or a dedicated "this M This is the best first choice if you just want to see what gbrain feels like before committing to cloud. You can always migrate later with `/setup-gbrain --switch`. +### Path 4: Remote gbrain MCP (split-engine) + +Best for: your brain runs on another machine you control (Tailscale, ngrok, internal LAN) or a teammate's server. You want the cross-machine memory benefit without standing up a local database, and you still want symbol-aware code search on this Mac. + +**What happens:** You paste an MCP URL (e.g. `https://wintermute.tail554574.ts.net:3131/mcp`) and a bearer token. The skill verifies the URL over the wire, registers gbrain as an HTTP MCP in `~/.claude.json` at user scope, and offers to also stand up a tiny local PGLite for code search (~30 seconds, ~120 MB disk). + +If you accept the local PGLite, you end up in **split-engine mode**: + +- **Brain/context queries** (`mcp__gbrain__search`, `mcp__gbrain__query`, `mcp__gbrain__get_page`) route to the remote MCP. Plans, retros, learnings, cross-machine memory — all on the shared server. +- **Code queries** (`gbrain code-def`, `code-refs`, `code-callers`, `code-callees`, `gbrain search` for code) route to the local PGLite via the `.gbrain-source` pin in each worktree. Indexed locally, fast, never leaves the machine. + +The two engines are independent. Wiping the local PGLite doesn't touch the remote brain; rotating the remote MCP bearer doesn't affect local code search. This is also the right configuration if your remote brain admin can't (or shouldn't) index every developer's checkout — local code stays local. + ## MCP registration for Claude Code By default the skill asks "Give Claude Code a typed tool surface for gbrain?" If you say yes, it runs: @@ -95,6 +117,35 @@ SSH and HTTPS remote variants collapse to the same key: `https://github.com/foo/ Storage: `~/.gstack/gbrain-repo-policy.json`, mode 0600, schema-versioned so future migrations stay deterministic. +## Keeping the brain current with `/sync-gbrain` + +`/setup-gbrain` is one-time onboarding. `/sync-gbrain` is the verb you run every time you want gbrain to see fresh changes in this repo's code. + +```bash +/sync-gbrain # incremental: mtime fast-path, ~seconds on a clean tree +/sync-gbrain --full # full reindex (~25-35 minutes on a big Mac) +/sync-gbrain --code-only # only the code stage; skip memory + brain-sync +/sync-gbrain --dry-run # preview what would sync; no writes +``` + +The skill runs three stages — code, memory, brain-sync — independently. A failure in one doesn't block the others. State persists to `~/.gstack/.gbrain-sync-state.json` so re-running picks up cleanly. + +**What it does on a fresh worktree:** + +1. **Pre-flight.** Checks `gbrain_local_status` (the local engine's health). If the engine is `broken-db` or `broken-config`, the skill STOPs with a remediation menu — it refuses to silently degrade. If the local engine is missing and you're in remote-MCP mode (Path 4), the code stage SKIPs cleanly and only brain-sync runs. +2. **Code stage.** Registers the cwd as a federated source via `gbrain sources add`, writes a `.gbrain-source` pin file in the repo root (kubectl-style context — every worktree gets its own pin, so Conductor sibling worktrees don't collide), runs `gbrain sync --strategy code`. +3. **Memory stage.** Stages your `~/.gstack/` transcripts + curated memory. In local-stdio MCP mode, ingests into the local engine. In remote-http MCP mode, persists staged markdown to `~/.gstack/transcripts/run--/` for the remote brain admin's pull pipeline. +4. **Brain-sync stage.** Pushes curated artifacts (plans, designs, retros) to your private artifacts repo if you have one configured. +5. **CLAUDE.md guidance.** Capability-checks the round-trip (write a page → search → find it). If green, writes the `## GBrain Search Guidance` block to your project's CLAUDE.md. If red, REMOVES the block — the agent should never be told to use a tool that isn't installed. + +**The watermark.** Sync state advances by commit hash. If gbrain hits a file it can't index (5 MB hard limit per file, or a file vanished mid-sync), the watermark stays put and subsequent syncs retry. To acknowledge an unfixable failure and move past it: + +```bash +gbrain sync --source --skip-failed +``` + +Re-runnable, idempotent, safe to run from multiple terminals on the same machine (locked at `~/.gstack/.sync-gbrain.lock`). + ## Switching engines later Picked PGLite and now want to join a team brain? One command: @@ -200,6 +251,25 @@ Gbrain itself ships with these that gstack wraps: | `SUPABASE_API_BASE` | `gstack-gbrain-supabase-provision` | Override the Management API host. Used by tests to point at a mock server. | | `GBRAIN_INSTALL_DIR` | `gstack-gbrain-install` | Override default install path (`~/gbrain`) | | `GSTACK_HOME` | every bin helper | Override `~/.gstack` state dir. Heavy test use. | +| `OPENAI_API_KEY` | `gbrain embed` subprocess | Required for embeddings during `gbrain sync` / `/sync-gbrain`. Without it, pages are imported structurally (symbol tables, chunks) but semantic search degrades — you'll see `[gbrain] embedding failed for code file ... OpenAI embedding requires OPENAI_API_KEY` in the sync log. | +| `ANTHROPIC_API_KEY` | `claude-agent-sdk`, paid evals | Required for `bun run test:evals` and any direct `query()` call against Claude. | +| `GSTACK_OPENAI_API_KEY` | `lib/conductor-env-shim.ts` | Conductor-injected fallback. Promoted to `OPENAI_API_KEY` when the canonical name is empty. | +| `GSTACK_ANTHROPIC_API_KEY` | `lib/conductor-env-shim.ts` | Same pattern as above for Anthropic. | + +## Conductor + GSTACK_* env vars + +If you run gstack inside a [Conductor](https://conductor.build) workspace, the process env doesn't inherit your interactive shell — `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` are typically empty even when they're set in `~/.zshrc`. Recent Conductor versions instead inject `GSTACK_ANTHROPIC_API_KEY` and `GSTACK_OPENAI_API_KEY` directly into the workspace env. + +`lib/conductor-env-shim.ts` bridges the gap: when imported as a side effect (`import "../lib/conductor-env-shim";`), it promotes `GSTACK_FOO_API_KEY` to `FOO_API_KEY` for any subprocess that doesn't see the canonical name. The shim is already wired into: + +- `bin/gstack-gbrain-sync.ts` — so `/sync-gbrain` picks up OpenAI for embeddings +- `bin/gstack-model-benchmark` — so `--judge` runs work without manual env mapping +- `scripts/preflight-agent-sdk.ts` — so paid-eval auth probes work +- `test/helpers/e2e-helpers.ts` — so `bun run test:evals` finds Anthropic + +If you add a new TS entry point that hits a paid API or needs gbrain embeddings, add the same one-line import at the top. See [CONTRIBUTING.md "Conductor workspaces"](CONTRIBUTING.md#conductor-workspaces) for the contributor checklist. + +`bin/gstack-codex-probe` is bash and doesn't read these directly — it relies on `~/.codex/` auth managed by the Codex CLI. ## Security model @@ -267,6 +337,26 @@ You edited `~/.gstack/gbrain-repo-policy.json` by hand with legacy `allow` value `/health` treats that as yellow, not red. Check `gbrain doctor --json | jq .checks` to see which sub-checks are warning. Typical causes: resolver MECE overlap (skill names clashing) or DB connection not yet configured. +### `/sync-gbrain` reports `OK` but `gbrain search` returns nothing semantic + +Embeddings probably failed during import. Symbol queries (`code-def`, `code-refs`) still work because they don't need embeddings, but `gbrain search ""` falls back to a degraded BM25 path. Look in the sync output for lines like: + +``` +[gbrain] embedding failed for code file : OpenAI embedding requires OPENAI_API_KEY +``` + +The fix is to put `OPENAI_API_KEY` in the process env before re-running. On a bare Mac shell, source it from `~/.zshrc` before calling. In Conductor, set `GSTACK_OPENAI_API_KEY` at the workspace level — `lib/conductor-env-shim.ts` promotes it to canonical automatically when imported. Re-run `/sync-gbrain --code-only` to backfill embeddings on already-imported pages. + +### `gbrain sync` blocked at a commit hash — `FILE_TOO_LARGE` + +A file in your tree exceeds gbrain's 5 MB hard limit (`MAX_FILE_SIZE` in `gbrain/src/core/import-file.ts`). Common culprits: response replay caches, captured screenshots, large JSON fixtures. Gbrain doesn't honor `.gitignore`-style exclude lists for code sync; the only knob is acknowledging the failure: + +```bash +gbrain sync --source --skip-failed +``` + +Watermark advances past the offending commit. The same file fails again if it changes; re-skip when that happens. + ### Switching PGLite → Supabase hangs Another gstack session in a sibling Conductor workspace may be holding a lock on your local PGLite file via its preamble's `gstack-brain-sync` call. Close other workspaces, re-run `/setup-gbrain --switch`. The timeout is bounded at 180s so you'll never actually wait forever. diff --git a/VERSION b/VERSION index 57fdbd724..939a56892 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.39.1.0 +1.39.2.0 diff --git a/package.json b/package.json index 601eb963c..592493d5e 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.39.1.0", + "version": "1.39.2.0", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module",