mirror of
https://github.com/garrytan/gstack.git
synced 2026-06-02 16:21:38 +02:00
3bef43bc5a
* fix(jsonl-merge): make equal-ts resolution converge across machines The JSONL append merge driver sorted timestamped entries by (0, ts) with no further tiebreaker. Equal-ts entries then fell back to stable-sort insertion order (base, ours, theirs), but git assigns the local side to "ours", so two machines resolving the same conflict emitted equal-ts lines in opposite order. The merged files diverged and never converged. gstack-telemetry-log uses second-granularity timestamps, so same-ts collisions are routine. Add the line content as the final sort tiebreaker so the order is total and side-independent. Add a regression test that runs the driver with the two sides swapped and asserts identical output. * fix(gen-skill-docs): quote frontmatter descriptions with interior colons (#1778) Generated SKILL.md frontmatter emitted the catalog-trimmed description: as a plain YAML scalar. A description with an interior ": " (e.g. "Ship workflow: detect...") parses as a nested mapping under strict YAML loaders, so Codex/OpenAI skill loading rejected those skills. applyCatalogTrim now routes the value through toYamlInlineScalar, which quotes (via JSON.stringify) only when a plain scalar would be invalid — interior ": ", inline " #", leading indicator char, or surrounding whitespace. Strings that are already valid plain scalars pass through unchanged to keep regen diffs small. The frontmatter test now parses every generated block (Claude + Codex hosts) with Bun.YAML.parse instead of string-checking that name:/description: substrings exist, so the regression can't reappear. Runs under `bun test` (already in CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(skills): regenerate SKILL.md after frontmatter quoting fix (#1778) 9 catalog-trimmed descriptions whose values contain an interior colon or inline- comment marker are now quoted. Generated output only; rerun of bun run gen:skill-docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(gbrain-sources): centralize sources-list shape handling in parseSourcesList (#1576) #1576's crash in sourceLocalPath was already fixed in v1.42.0.0 (dual-shape handling). But the readers disagreed: sourceLocalPath accepted both the wrapped {sources:[...]} object (v0.20+) and a bare array, while probeSource and sourcePageCount accepted only the wrapped shape. Extract one parseSourcesList() normalizer and route all three through it, so the shape assumption lives in a single place. This is also the base the #1734 remote_url audit builds on. parseSourcesList returns [] for null/garbage rather than throwing; callers treat 'no rows' as absent. New test/gbrain-sources-parse.test.ts pins both shapes plus the garbage paths and confirms config.remote_url survives for the audit. #1576 is closeable as already-fixed in v1.42.0.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gbrain): spawn gbrain + brain-sync through a shell on Windows (#1731) On Windows, bun/npm install gbrain as a gbrain.cmd/.ps1 shim and gstack-brain-sync is a bash shebang script. spawnSync/spawn/execFileSync resolve neither without a shell, so the child spawn failed ENOENT — on the sync orchestrator this surfaced as 'brain-sync exited undefined' (#1731). Add NEEDS_SHELL_ON_WINDOWS (process.platform === 'win32') in gbrain-exec and pass it as shell: to every gbrain/brain-sync child spawn: spawnGbrain, spawnGbrainAsync, execGbrainText (gbrain-exec), the two sources-list/remove/add spawns (gbrain-sources), the version + probe spawns (gbrain-local-status), and the two brain-sync spawns in the orchestrator. POSIX keeps the cheaper no-shell path. macOS/Linux CI can't exercise the Windows path, so test/gbrain-spawn-windows-shell.ts is a static-grep tripwire: it fails CI if a gbrain/brain-sync spawn is added without the shell flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(catalog-trim): expect YAML-quoted descriptions with interior colons (#1778) The quoting fix wraps colon-bearing catalog descriptions in double quotes; two catalog-trim assertions still pinned the old unquoted form. Tolerate the optional quotes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gbrain-sync): defensive guards against destructive gbrain ops (#1734) The orchestrator shelled out to gbrain's destructive subcommands as if they were safe. gbrain can rm-rf a user's working tree during an autopilot race (its own bug, upstream gbrain #1526); gstack now defends itself. New lib/gbrain-guards.ts gates the two destructive reach points, all checked immediately before the op: - Autopilot refuse (multi-signal, affirmative-only): refuse a destructive op when a live 'gbrain autopilot' process (primary) or a known autopilot lock file (secondary; checked under both GBRAIN_HOME and ~/.gbrain since gbrain #1226 ignores GBRAIN_HOME) is present. No signal → proceed; inability to introspect never bricks a normal sync. - sources remove: routed through safeSourcesRemove → decideSourceRemove. Fail CLOSED — refuse to remove a user-managed source (remote_url set, local_path outside gbrain's clones) when gbrain has no --keep-storage to protect the files (it doesn't in 0.41.x). Also fail closed when the source list can't be read. Path containment uses realpath so a symlink can't smuggle a delete out of clones. - sync --strategy code: decideCodeSync refuses URL-managed sources (remote_url set) unless --allow-reclone is passed, since the walk can auto-reclone (rm-rf). Capability detection memoizes per process keyed to gbrain's identity (no stale persistent cache); --keep-storage can't be probed (generic help) so it defaults unsupported → fail closed. Every guard surfaces a visible reason; autopilot/reclone refusals fail the code stage (verdict ERR) rather than silently skipping protection. test/gbrain-guards.test.ts covers all branches hermetically (injected rows + probe overrides): autopilot signals, fail-closed remove, keep-storage path, reclone gate, realpath/symlink containment. Supersedes #1736 (which guarded a nonexistent path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(sync-gbrain): warn against running during autopilot; prefer --path sources (#1734) Adds a Safety note to the /sync-gbrain guidance (template + regenerated SKILL.md + this repo's CLAUDE.md): don't run while autopilot is active, and prefer `gbrain sources add --path` over URL-managed sources, which can auto-reclone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(memory-ingest): configurable import timeout + resume-on-timeout messaging (#1611) The gbrain import (the long pole on big brains) had a hardcoded 30-min timeout, so large memory corpora got SIGTERM'd mid-import on /sync-gbrain --full. Make it configurable via GSTACK_INGEST_TIMEOUT_MS (default 30 min, validated 1min–24h). gstack can't drive gbrain's internal resume, but the existing SIGTERM forwarder already preserves gbrain's import-checkpoint.json, so the next run resumes. On a timeout we now say so explicitly ('checkpoint preserved — re-run /sync-gbrain to resume, raise GSTACK_INGEST_TIMEOUT_MS for big brains') instead of surfacing a bare 'exited null'. True gstack-driven ingest-resume is deferred to gbrain (.context/gbrain-asks.md). Also guards the module's main() behind import.meta.main so resolveImportTimeoutMs is unit-testable; the orchestrator runs it as a subprocess where main still fires. New test/memory-ingest-timeout.test.ts pins default/override/invalid resolution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(browse): stop the headed daemon crash-loop + silent headless downgrade (#1781) A headed session against a beacon-heavy page (analytics/extension load) could tip the single-threaded daemon into a self-inflicted crash-loop: a brief HTTP stall was read as a crash, the restart didn't clear the dead Chromium's SingletonLock, the relaunch failed, and the session silently came back headless. Four fixes: 1. Busy-vs-dead (sendCommand): on a connection error, if the process is alive give /health a bounded probe (3x/250ms) and just retry the command — never kill+restart a live-but-busy server. A 30s timeout now reports 'busy, not restarting' when the process is alive instead of exiting into a kill cycle. 2. Profile-lock cleanup on (re)start: startServer reaps the orphaned Chromium holding the SingletonLock and clears Singleton{Lock,Socket,Cookie} before relaunch, so the auto-restart path gets the same clean profile the manual connect preamble did. 3. Headed persistence: the restart env reapplies BROWSE_HEADED from this invocation OR the persisted server state (mode==='headed'), so a restart from a plain command never downgrades a headed window to invisible headless. Extracted to buildRestartEnv. 4. Force-clean disconnect reaps the Chromium child tree (via the SingletonLock PID) so the next connect starts clean instead of fighting an orphan. Plus macOS window surfacing: connect + focus raise 'Google Chrome for Testing' to the active Space (best-effort osascript) with a Mission Control hint — the first thing users read as 'I can't see the browser'. Shared lock helpers (chromiumProfileDir / cleanChromiumProfileLocks / killOrphanChromium) dedupe the connect, disconnect, and restart paths. browse/test/restart-env.test.ts pins the headed-persistence decision; the full crash-loop repro is an E2E (periodic). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(gbrain-install): remove the v0.18.2 pin, install latest + version floor + doctor self-test (#1744) The installer pinned gbrain at v0.18.2 while gbrain shipped v0.41.x — ~23 versions behind. Remove the hard pin: a fresh clone now stays on the latest default-branch HEAD. --pinned-commit <sha> still pins for reproducibility. Unpinning removes the version gate the pin provided, so add two install-time gates that fail closed (exit 3, matching the existing PATH-shadow/version-mismatch posture): - MIN_GBRAIN_VERSION floor (0.20.0, the sources-list/federated surface gstack needs): refuse an install below it. - gbrain doctor --fast self-test when a brain config already exists (re-install / detected clone): refuse to leave a broken gbrain in place. Pre-init installs skip it; the full /sync-gbrain --dry-run self-test runs from /setup-gbrain after init. Docs updated (USING_GBRAIN_WITH_GSTACK.md no longer says 'edit PINNED_COMMIT'). Detect-install tests bump the success-path fixtures above the floor and add a below-floor exit-3 test. The gbrain-side asks (root #1526 fix, --keep-storage, remove-lease, capability command, ingest-resume, integration CI) are written to .context/gbrain-asks.md for filing against garrytan/gbrain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(#1778): update claude-ship golden + catalog-mode assertions for quoted descriptions ship's catalog description ('Ship workflow: detect...') has an interior colon, so the #1778 fix now YAML-quotes it. Refresh the claude-ship golden baseline to the quoted output and make the catalog-mode-full trim/restore assertions quote-tolerant. codex/factory ship goldens are unaffected (they use block-scalar descriptions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gen-skill-docs): use function replacer so a $ in a description can't corrupt frontmatter (#1778) String.prototype.replace treats $&/$1/$` in the replacement as patterns. A future skill description containing $ (e.g. referencing $B/$D) would silently corrupt the generated frontmatter. Use a function replacer. Behavior-preserving for all current descriptions (regen produces no diff). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v1.55.0.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(gbrain): document configurable memory-ingest timeout for v1.55.0.0 USING_GBRAIN_WITH_GSTACK.md: note GSTACK_INGEST_TIMEOUT_MS (default 30 min, 1 min-24h range) on the /sync-gbrain memory stage, plus checkpoint-resume on timeout. Fills the reference gap left by the configurable-import-timeout fix (#1611) shipped in v1.55.0.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jayesh Betala <jayesh.betala7@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
232 lines
9.2 KiB
TypeScript
232 lines
9.2 KiB
TypeScript
/**
|
|
* Centralized gbrain CLI invocation.
|
|
*
|
|
* Every `gbrain ...` spawn from `bin/gstack-gbrain-sync.ts` and
|
|
* `bin/gstack-memory-ingest.ts` MUST go through `spawnGbrain` (or
|
|
* `execGbrainJson`), and the invariant test
|
|
* `test/gbrain-exec-invariant.test.ts` enforces this with a static-source
|
|
* grep. The helper layer guarantees three properties:
|
|
*
|
|
* 1. **DATABASE_URL is seeded from gbrain's own config**, not from the
|
|
* caller's `.env.local`. gbrain auto-loads `.env.local` via dotenv on
|
|
* startup. When `/sync-gbrain` runs inside a Next.js / Prisma / Rails
|
|
* project with its own `DATABASE_URL`, gbrain reads that one and not
|
|
* its own `${GBRAIN_HOME:-$HOME/.gbrain}/config.json`. Auth fails;
|
|
* code + memory stages crash; only brain-sync's git push survives.
|
|
*
|
|
* 2. **Bun-aware env passing.** Mutating `process.env.DATABASE_URL` does
|
|
* NOT propagate to children of `child_process.spawnSync`/`spawn` in
|
|
* Bun — the child gets the original startup env. So we cannot just
|
|
* set process.env; we must thread an explicit `env:` dict to every
|
|
* spawn. This is the central bug the helper exists to prevent
|
|
* regressing on.
|
|
*
|
|
* 3. **`GBRAIN_HOME` honored consistently.** Other gstack helpers
|
|
* (`detectEngineTier`) already honor `GBRAIN_HOME`. `buildGbrainEnv`
|
|
* reads from `${GBRAIN_HOME:-$HOME/.gbrain}/config.json` so all
|
|
* gstack-side gbrain calls agree on which config file matters.
|
|
*
|
|
* **Escape hatch:** `GSTACK_RESPECT_ENV_DATABASE_URL=1` returns the
|
|
* caller's env unchanged. Use only when the brain intentionally lives in
|
|
* the project's local DB (rare).
|
|
*/
|
|
|
|
import { existsSync, readFileSync } from "fs";
|
|
import { join } from "path";
|
|
import { homedir } from "os";
|
|
import { spawnSync, spawn, execFileSync, type SpawnSyncReturns, type ChildProcess, type SpawnOptions } from "child_process";
|
|
|
|
interface GbrainConfig {
|
|
database_url?: string;
|
|
}
|
|
|
|
export interface BuildGbrainEnvOptions {
|
|
/**
|
|
* Caller env to extend. Defaults to `process.env`. Tests inject a
|
|
* synthetic env so the helper can be exercised without polluting the
|
|
* real process env.
|
|
*/
|
|
baseEnv?: NodeJS.ProcessEnv;
|
|
/**
|
|
* When true, announce on stderr that we overrode the caller's
|
|
* DATABASE_URL. Suppressed for the `--quiet` sync flow.
|
|
*/
|
|
announce?: boolean;
|
|
}
|
|
|
|
/**
|
|
* Detect whether a DATABASE_URL targets a PgBouncer transaction-mode pooler.
|
|
*
|
|
* Supabase transaction-mode poolers conventionally run on port 6543 at
|
|
* `*.pooler.supabase.com`. When gbrain connects through one of these, it
|
|
* auto-disables prepared statements — but search requires them (#1435).
|
|
* Returns `true` when the URL looks like a transaction-mode pooler so the
|
|
* caller can set `GBRAIN_PREPARE=true` to re-enable prepared statements.
|
|
*/
|
|
export function isTransactionModePooler(url: string): boolean {
|
|
try {
|
|
// DATABASE_URLs use postgresql:// scheme which URL() doesn't natively
|
|
// parse host/port from, so swap to http:// for reliable parsing.
|
|
const parsed = new URL(url.replace(/^postgres(ql)?:\/\//, "http://"));
|
|
return parsed.port === "6543";
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Build an env dict with DATABASE_URL seeded from
|
|
* `${GBRAIN_HOME:-$HOME/.gbrain}/config.json`. Returns the base env
|
|
* unchanged when:
|
|
* - `GSTACK_RESPECT_ENV_DATABASE_URL=1` (intentional opt-out),
|
|
* - the config file is missing or unparseable,
|
|
* - the config has no `database_url`,
|
|
* - the caller already set DATABASE_URL to the same value.
|
|
*
|
|
* When the effective DATABASE_URL targets a PgBouncer transaction-mode
|
|
* pooler (port 6543), sets `GBRAIN_PREPARE=true` so gbrain re-enables
|
|
* prepared statements needed for search (#1435). Caller can override
|
|
* with `GBRAIN_PREPARE=false` in the base env.
|
|
*
|
|
* Always returns a fresh object — mutating the returned env never
|
|
* affects the caller's env. Tests assert on effective values, not
|
|
* object identity.
|
|
*/
|
|
export function buildGbrainEnv(opts: BuildGbrainEnvOptions = {}): NodeJS.ProcessEnv {
|
|
const baseEnv = opts.baseEnv || process.env;
|
|
const out: NodeJS.ProcessEnv = { ...baseEnv };
|
|
if (baseEnv.GSTACK_RESPECT_ENV_DATABASE_URL === "1") return out;
|
|
|
|
const homeBase = baseEnv.HOME || homedir();
|
|
const gbrainHome = baseEnv.GBRAIN_HOME || join(homeBase, ".gbrain");
|
|
const configPath = join(gbrainHome, "config.json");
|
|
if (!existsSync(configPath)) return out;
|
|
|
|
let cfg: GbrainConfig = {};
|
|
try {
|
|
cfg = JSON.parse(readFileSync(configPath, "utf-8")) as GbrainConfig;
|
|
} catch {
|
|
return out;
|
|
}
|
|
if (!cfg.database_url) return out;
|
|
|
|
const hadCaller = baseEnv.DATABASE_URL !== undefined;
|
|
const alreadyMatch = baseEnv.DATABASE_URL === cfg.database_url;
|
|
if (!alreadyMatch) {
|
|
out.DATABASE_URL = cfg.database_url;
|
|
if (opts.announce) {
|
|
const note = hadCaller ? " (overrode value from caller env / .env.local)" : "";
|
|
process.stderr.write(`[gbrain-exec] seeded DATABASE_URL from ${configPath}${note}\n`);
|
|
}
|
|
}
|
|
|
|
// PgBouncer transaction-mode pooler detection (#1435): when the effective
|
|
// DATABASE_URL targets port 6543 (Supabase transaction-mode convention),
|
|
// gbrain auto-disables prepared statements — but search needs them.
|
|
// Set GBRAIN_PREPARE=true unless the caller explicitly opted out.
|
|
const effectiveUrl = out.DATABASE_URL || cfg.database_url;
|
|
if (effectiveUrl && !out.GBRAIN_PREPARE && isTransactionModePooler(effectiveUrl)) {
|
|
out.GBRAIN_PREPARE = "true";
|
|
if (opts.announce) {
|
|
process.stderr.write(
|
|
`[gbrain-exec] set GBRAIN_PREPARE=true (port 6543 transaction-mode pooler detected)\n`,
|
|
);
|
|
}
|
|
}
|
|
|
|
return out;
|
|
}
|
|
|
|
/**
|
|
* Windows can't directly spawn the `gbrain` launcher (bun/npm install it as a
|
|
* `gbrain.cmd`/`.ps1` shim) or a shebang script like the bash `gstack-brain-sync`
|
|
* — `spawnSync`/`spawn` resolve those only through a shell's PATHEXT + interpreter
|
|
* lookup. Without `shell: true` the child spawn fails ENOENT, which on the sync
|
|
* orchestrator surfaced as "brain-sync exited undefined" (#1731). Gate on platform
|
|
* so POSIX keeps the cheaper no-shell path. Exported so the static-grep tripwire
|
|
* (test/gbrain-spawn-windows-shell.test.ts) can assert every gbrain/brain-sync
|
|
* spawn carries it.
|
|
*/
|
|
export const NEEDS_SHELL_ON_WINDOWS = process.platform === "win32";
|
|
|
|
export interface SpawnGbrainOptions {
|
|
/** Timeout in milliseconds. Defaults to 30s. */
|
|
timeout?: number;
|
|
/** Working directory for the child process. */
|
|
cwd?: string;
|
|
/** Stdio configuration. Defaults to capturing both stdout and stderr. */
|
|
stdio?: "inherit" | "pipe" | "ignore" | Array<"inherit" | "pipe" | "ignore">;
|
|
/**
|
|
* Base env to extend before seeding DATABASE_URL. Defaults to
|
|
* `process.env`. Tests inject a synthetic env so the spawn picks up a
|
|
* gbrain shim on PATH and a fake `~/.gbrain/config.json`.
|
|
*/
|
|
baseEnv?: NodeJS.ProcessEnv;
|
|
/** Whether to announce DATABASE_URL seeding on stderr. */
|
|
announce?: boolean;
|
|
}
|
|
|
|
/**
|
|
* Spawn `gbrain <args>` with the seeded env. Returns the raw
|
|
* `SpawnSyncReturns<string>` so callers can inspect `status`, `stdout`,
|
|
* `stderr` exactly as they would with `spawnSync` directly.
|
|
*/
|
|
export function spawnGbrain(args: string[], opts: SpawnGbrainOptions = {}): SpawnSyncReturns<string> {
|
|
return spawnSync("gbrain", args, {
|
|
encoding: "utf-8",
|
|
timeout: opts.timeout ?? 30_000,
|
|
cwd: opts.cwd,
|
|
stdio: opts.stdio || ["ignore", "pipe", "pipe"],
|
|
env: buildGbrainEnv({ baseEnv: opts.baseEnv, announce: opts.announce }),
|
|
shell: NEEDS_SHELL_ON_WINDOWS, // #1731: gbrain is a .cmd shim on Windows
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Run `gbrain <args>` and parse stdout as JSON. Returns `null` on
|
|
* non-zero exit, parse failure, or timeout. Useful for `gbrain sources
|
|
* list --json` and similar.
|
|
*/
|
|
export function execGbrainJson<T = unknown>(args: string[], opts: SpawnGbrainOptions = {}): T | null {
|
|
const r = spawnGbrain(args, opts);
|
|
if (r.status !== 0) return null;
|
|
try {
|
|
return JSON.parse(r.stdout || "null") as T;
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Async streaming variant for callers that need to attach stdout/stderr
|
|
* listeners (e.g., `gbrain import` in `gstack-memory-ingest.ts`). Always
|
|
* injects the seeded env. Returns the raw `ChildProcess` so the caller
|
|
* can wire up its own promise around exit/timeout/signal handling.
|
|
*/
|
|
export function spawnGbrainAsync(
|
|
args: string[],
|
|
opts: { stdio?: SpawnOptions["stdio"]; cwd?: string; baseEnv?: NodeJS.ProcessEnv } = {},
|
|
): ChildProcess {
|
|
return spawn("gbrain", args, {
|
|
stdio: opts.stdio || ["ignore", "pipe", "pipe"],
|
|
cwd: opts.cwd,
|
|
env: buildGbrainEnv({ baseEnv: opts.baseEnv, announce: false }),
|
|
shell: NEEDS_SHELL_ON_WINDOWS, // #1731: gbrain is a .cmd shim on Windows
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Run `gbrain <args>` via execFileSync. Throws on non-zero exit. Useful
|
|
* for callers that want to surface gbrain's stderr as the error message.
|
|
*/
|
|
export function execGbrainText(args: string[], opts: SpawnGbrainOptions = {}): string {
|
|
return execFileSync("gbrain", args, {
|
|
encoding: "utf-8",
|
|
timeout: opts.timeout ?? 30_000,
|
|
cwd: opts.cwd,
|
|
stdio: opts.stdio || ["ignore", "pipe", "pipe"],
|
|
env: buildGbrainEnv({ baseEnv: opts.baseEnv, announce: opts.announce }),
|
|
shell: NEEDS_SHELL_ON_WINDOWS, // #1731: gbrain is a .cmd shim on Windows
|
|
});
|
|
}
|