mirror of
https://github.com/garrytan/gstack.git
synced 2026-06-10 12:03:59 +02:00
v1.57.5.0 feat: cross-session decision memory + gbrain dream-stage call graph (#1910)
* feat(gbrain-sync): add cycleCompleted() cycle-state probe Reads `gbrain doctor` cycle_freshness to classify whether a source has completed a full cycle (completed/never/unknown). A fail naming this source -> never; a fail naming only other sources -> completed; an absent or unparseable check -> unknown, so an unrelated doctor failure never masks a real state. Gates the automatic call-graph build on --full. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(gbrain-sync): --dream call-graph stage with lock-free gate + honest outcome guard Adds a source-scoped `gbrain dream --source <id>` stage that builds this worktree's call graph (code-callers/code-callees). Runs lock-free after the sync lock releases so it never blocks sibling worktrees; a .dream-in-progress marker dedupes concurrent dreams. --full auto-runs it only when the cycle was never built; explicit --dream always forces; --no-dream opts out. The stage parses the cycle's own output and reports the truth, not a flat "built": a WARN when the schema pack can't extract code symbols, when the embed phase failed for a missing key, or when 0 edges resolved; OK with the resolved-edge count otherwise. gbrain exits 0 even when it skips on a held cycle lock (e.g. autopilot), so that case reports SKIP, not success. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: ignore gbrain .sources/ local staging dir gbrain writes per-source staging and capability-check artifacts under .sources/ in the repo root. It's machine-local runtime state, not source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(gbrain): honest call-graph guidance in /sync-gbrain + pin works on gbrain>=0.41.38 sync-gbrain frames the --dream offer honestly: building a call graph requires a code-aware schema pack, and the dream stage reports a WARN when it can't. The verdict's Call graph row mirrors the dream stage's real outcome instead of assuming a completed cycle means edges exist. The ## GBrain Search Guidance block written into CLAUDE.md drops the old code-callers --source caveat: gbrain >=0.41.38.0 honors the .gbrain-source pin for code-callers/code-callees. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(jsonl-store): shared audited JSONL plumbing (injection-reject + atomic append + tolerant read) Single source of truth extracted for D2A: gstack-learnings-* and the upcoming gstack-decision-* bins share one injection-pattern list, one atomic single-line appender, and one tolerant reader. No more drift between stores. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(learnings-log): use shared hasInjection from lib/jsonl-store (D2A) Replace the inline injection-pattern copy with the shared list. One audited write-path rejection across learnings + the upcoming decision store. Behavior unchanged (35/35 learnings tests green); learnings-search keeps its inline copy because a structural test pins its bash/bun shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(decision): event-sourced decision-memory model (lib/gstack-decision) decide/supersede/redact events on lib/jsonl-store; active set is computed (no mutable status), dangling refs tolerated. Free-text is injection-checked and redact-scanned on write (HIGH secret -> reject). Scope filter (repo/branch/issue) for relevant resurfacing. File-only + reliable; gbrain not required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(decision): bounded active snapshot + compaction (redact expunges, supersede archives) writeSnapshot/readSnapshot/rebuildSnapshot give an O(active) bounded read for the session-start hot path (D1A). compact() rewrites the log to active, archives superseded decisions for history, and EXPUNGES redacted ones (dropped, never archived) so an accidentally-captured secret leaves the store for good. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(decision): gstack-decision-log + gstack-decision-search bins (non-interactive) Two bins mirroring gstack-learnings-* (D3A). log writes decide/--supersede/--redact/ --compact events + refreshes the bounded snapshot + enqueues for cross-machine sync; search reads the O(active) snapshot, scope-filtered to current branch, newest-first, --all to include superseded, --json for machines. Empty store returns silently (no snapshot write on an empty read). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(memory): surface active decisions at session start + capture nudge (Context Recovery) Context Recovery now shows recent scope-relevant active decisions (bounded read of decisions.active.json via gstack-decision-search) and instructs the agent to treat them as settled calls and to log durable decisions/reversals. Closes the Phase-1 capture->curate->resurface loop, reliable + file-only. Regen across all hosts folded in (squash-with-regen); parity 10/10, freshness green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: refresh ship golden baselines for the memory-loop preamble change Context Recovery now emits the cross-session-decisions block, so ship's preamble (all hosts) changed. Golden baselines are hand-maintained copies (gen does not write them); refresh them from the fresh gen so golden-file regression passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(memory): document the cross-session decision-memory loop in CLAUDE.md Adds a '## Cross-session decision memory' section: how to resurface (gstack-decision-search) and capture (gstack-decision-log) durable decisions, the supersede/redact/compact verbs, and a crisp durable-vs-trivial definition so the store stays signal. Reliable file-only path; gbrain not required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(memory): emit durable decisions from ship/ceo/eng/spec at structured points Wires the four skills that finalize real decisions to capture them in the cross-session decision store, from their STRUCTURED outputs (never free-text scraping): - ship: the version bump (level + why) at write time - plan-ceo-review: accepted scope + verdict (branch-scoped) - plan-eng-review: the architecture verdict + key call (branch-scoped) - spec: the filed issue's core approach (issue-scoped) All emits are non-interactive, schema-correct (content in decision/rationale, source=skill, confidence 1-10), and best-effort (|| true) so a decision-log failure never blocks the workflow. Includes regen across hosts + refreshed ship golden baselines. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(memory): optional gbrain --semantic recall for decision search Adds gstack-decision-search --semantic (with --query): appends a 'Related from memory' block from gbrain semantic search, scoped to the curated-memory source. Pure enhancement, reliability-first: a new lib/gstack-decision-semantic.ts is the ONLY decision module that touches gbrain and is imported lazily only on --semantic, so the reliable file path never loads gbrain code. Every path degrades to the reliable file results when gbrain is off, unconfigured, empty, or errors (never throws, 10s timeout). Built against the verified gbrain 0.42.x surface (text output [score] slug -- snippet, NOT JSON; curated-memory source resolved by worktree path, not a gstack-brain-<user> id). Deterministic-contract tests only: parser units, degrade-to-null when gbrain absent, and a fake-gbrain shim proving scope+search end-to-end. find-contradictions deferred (no verifiable CLI surface yet + curated memory not indexed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(gbrain-sync): self-heal stale autopilot lock (dead-pid) detectAutopilot treated a lock FILE as proof of life, so a crashed gbrain daemon left a stale lock that wedged every sync forever (observed: a dead pid refused --full indefinitely). Now read the holder pid (bare or JSON body) and check liveness via signal-0: ESRCH=dead → ignore the stale signal and keep checking; EPERM=alive (other user) → active. A stale lock never masks a live autopilot process. Pure decision function — does not delete the file; the caller may clean it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(review): drop stray trailing code fence in TODOS-format Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(test): align section-loading E2E testNames with their TOUCHFILES keys Pre-existing on main (v1.56.x): the two section-loading E2E tests used human-label testNames ('/ship section-loading') that don't match their slug keys ('ship-section-loading') in E2E_TOUCHFILES/E2E_TIERS. Every other E2E test uses the slug as its testName, and the TOUCHFILES completeness gate requires testName to be a registered key — so the gate was red. Align both testNames to their slug keys (also fixes tier lookup for these two periodic tests). Verified failing on a clean origin/main checkout before the fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: pre-landing review fixes (datamark, DRY, compact, coverage) Addresses the pre-landing review findings (all INFORMATIONAL, no criticals): - security: datamark resurfaced decision text at the render boundary (lib/gstack-decision.ts datamark() — neutralizes code fences, --- banners, <|role|>/</system> markers, control chars, newlines). Applied in gstack-decision-search human output so stored text can't masquerade as instructions in Context Recovery (codex hardening #3 / AC #7). --json stays raw. - DRY: extract resolveSlug/gitBranch/flagValue to lib/bin-context.ts; both decision bins use it instead of duplicating the helpers. - compact(): batch the archive append (one write, not N) and shrink the mid-compact crash window; simplify the opaque branch/issue ternary. - coverage: learnings-log injection rejection (D2A wiring), search --recent/ --scope + NaN-safe --recent, datamark-applied, unparseable lock body, compact-empty, corrupt-snapshot degrade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): close adversarial-review findings in decision memory Adversarial review (Claude subagent) found a CRITICAL the specialist pass missed: - F1 (CRITICAL): 'Human:'/'Assistant:' turn-prefixes bypassed BOTH the write-time denylist AND datamark(), landing verbatim in agent context inside the trusted ACTIVE DECISIONS fence. Add 'human:' (+ 'disregard previous', 'from now on') to the shared denylist, and have datamark() neutralize Human:/Assistant:/System:/User: turn-prefixes (ZWSP) at the render boundary. - F2: datamark() only stripped ASCII C0; extend to Unicode line terminators (U+0085/2028/2029) and U+007F so 'strip newlines' actually holds. - F3: validateDecide blocked only HIGH secrets; MEDIUM-tier PII (e.g. SSN) persisted silently and synced cross-machine. The store is non-interactive (no confirm path), so fail closed on MEDIUM too. - F4: compact() was a lock-free read-modify-rewrite that could clobber a concurrent append (lost decision). Add an O_EXCL compact lock + a pre-rename size recheck that aborts untouched (skipped=true) if an append landed; caller re-runs. - F7: filterByScope unknown/garbage scope fell through to 'return true' (leaked into every context); fail conservative (false). F5 (pid reuse) and F6 (pgrep over-match) are intentionally left as-is: both fail SAFE (over-refuse sync); making them precise would introduce a fail-DANGEROUS path (allowing sync during a real autopilot). True disambiguation needs gbrain to stamp the lock with a start-time, which gstack doesn't own. F8 (compact moves history to archive) is by design. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): close cross-model (Codex) adversarial findings Codex adversarial review found a HIGH the Claude pass missed plus 3 mediums: - C1 (HIGH): gstack-decision-search --all returned every decide and IGNORED redact events, so a redacted secret still resurfaced via --all until compact ran. --all now excludes redacted (redact = expunge from every read path), still showing superseded history. - C-med: semantic (external gbrain) slug/snippet were printed raw — datamark them too so a gbrain hit can't spoof role markers / fences into agent context. - C4: semanticRecall fell back to an UNSCOPED gbrain search when no curated-memory source resolved, pulling code/doc corpora mislabeled as 'related decisions'. Now returns null (degrade) when there's no worktree-backed memory source. - C5: validateDecide scanned only decision/rationale/alternatives; branch and issue are stored + surfaced (raw via --json), so include them in the injection+secret scan. C2 (snapshot staleness) / C3 (compact TOCTOU residual): accepted for a single-user store — atomic appends never lose the event, rebuilds self-heal, and the compact size-recheck leaves only a sub-ms window; full append-locking would break the lock-free append design. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v1.57.5.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
+412
-24
@@ -37,7 +37,7 @@ import { createHash } from "crypto";
|
||||
|
||||
import "../lib/conductor-env-shim";
|
||||
import { detectEngineTier, withErrorContext, canonicalizeRemote } from "../lib/gstack-memory-helpers";
|
||||
import { ensureSourceRegistered, sourcePageCount, parseSourcesList } from "../lib/gbrain-sources";
|
||||
import { ensureSourceRegistered, sourcePageCount, parseSourcesList, cycleCompleted, type CycleStatus } from "../lib/gbrain-sources";
|
||||
import { detectAutopilot, decideSourceRemove, decideCodeSync } from "../lib/gbrain-guards";
|
||||
import { localEngineStatus, type LocalEngineStatus } from "../lib/gbrain-local-status";
|
||||
import { buildGbrainEnv, spawnGbrain, execGbrainJson, NEEDS_SHELL_ON_WINDOWS } from "../lib/gbrain-exec";
|
||||
@@ -47,13 +47,17 @@ import { checkOwnedStagingDir } from "../lib/staging-guard";
|
||||
|
||||
type Mode = "incremental" | "full" | "dry-run";
|
||||
|
||||
interface CliArgs {
|
||||
export interface CliArgs {
|
||||
mode: Mode;
|
||||
quiet: boolean;
|
||||
noCode: boolean;
|
||||
noMemory: boolean;
|
||||
noBrainSync: boolean;
|
||||
codeOnly: boolean;
|
||||
/** Force the source-scoped dream cycle (builds this source's call graph). Always runs. */
|
||||
dream: boolean;
|
||||
/** Opt out of the dream cycle that `--full` would otherwise auto-run. */
|
||||
noDream: boolean;
|
||||
/** #1734: opt-in to sync a URL-managed source whose code walk may auto-reclone. */
|
||||
allowReclone: boolean;
|
||||
}
|
||||
@@ -72,6 +76,13 @@ interface StageResult {
|
||||
ok: boolean;
|
||||
duration_ms: number;
|
||||
summary: string;
|
||||
/**
|
||||
* Stage ran and did not error, but the outcome is a degraded no-op the user
|
||||
* should know about (e.g. dream completed but the schema pack can't extract
|
||||
* code symbols, so the call graph stays empty). Rendered as WARN, counts as
|
||||
* ok for the exit code — it's not a failure, just not the happy path.
|
||||
*/
|
||||
warn?: boolean;
|
||||
/** Stage-specific structured detail. Code stage carries source_id + page_count. */
|
||||
detail?: CodeStageDetail;
|
||||
}
|
||||
@@ -84,6 +95,24 @@ const STATE_PATH = join(GSTACK_HOME, ".gbrain-sync-state.json");
|
||||
const LOCK_PATH = join(GSTACK_HOME, ".sync-gbrain.lock");
|
||||
const STALE_LOCK_MS = 5 * 60 * 1000;
|
||||
|
||||
// Dream (call-graph build) is brain-global and runs LOCK-FREE after the sync
|
||||
// lock releases, so it can't use the sync lock to dedupe across worktrees. A
|
||||
// dedicated short-TTL marker prevents two worktrees from launching duplicate
|
||||
// ~35-min global jobs. TTL matches the dream timeout default so a crashed run
|
||||
// can't wedge the marker longer than one cycle.
|
||||
const DEFAULT_DREAM_TIMEOUT_MS = 45 * 60 * 1000; // 45min — dream is the slow stage
|
||||
const DREAM_MARKER_STALE_MS = DEFAULT_DREAM_TIMEOUT_MS;
|
||||
|
||||
/**
|
||||
* Marker path computed fresh per call (not a module const) so tests can mutate
|
||||
* GSTACK_HOME at runtime — same pattern as cacheFilePath() in
|
||||
* lib/gbrain-local-status.ts. Avoids the ESM static-import hoist trap where a
|
||||
* module-load-time const captures the real ~/.gstack before a test can redirect.
|
||||
*/
|
||||
export function dreamMarkerPath(): string {
|
||||
return join(process.env.GSTACK_HOME || join(homedir(), ".gstack"), ".dream-in-progress");
|
||||
}
|
||||
|
||||
// Default 35-minute timeout for code-walk + memory-ingest stages. Override via
|
||||
// GSTACK_SYNC_CODE_TIMEOUT_MS / GSTACK_SYNC_MEMORY_TIMEOUT_MS. Bounds-checked
|
||||
// in resolveStageTimeoutMs below so wildly-low values don't make resume
|
||||
@@ -100,26 +129,27 @@ const MAX_STAGE_TIMEOUT_MS = 86_400_000; // 24 hour ceiling
|
||||
export function resolveStageTimeoutMs(
|
||||
envValue: string | undefined,
|
||||
envName: string,
|
||||
defaultMs: number = DEFAULT_STAGE_TIMEOUT_MS,
|
||||
): number {
|
||||
if (envValue === undefined || envValue === "") return DEFAULT_STAGE_TIMEOUT_MS;
|
||||
if (envValue === undefined || envValue === "") return defaultMs;
|
||||
const n = Number.parseInt(envValue, 10);
|
||||
if (!Number.isFinite(n) || Number.isNaN(n) || n <= 0) {
|
||||
console.warn(
|
||||
`[sync] ${envName}="${envValue}" is not a positive integer; falling back to ${DEFAULT_STAGE_TIMEOUT_MS}ms`,
|
||||
`[sync] ${envName}="${envValue}" is not a positive integer; falling back to ${defaultMs}ms`,
|
||||
);
|
||||
return DEFAULT_STAGE_TIMEOUT_MS;
|
||||
return defaultMs;
|
||||
}
|
||||
if (n < MIN_STAGE_TIMEOUT_MS) {
|
||||
console.warn(
|
||||
`[sync] ${envName}=${n} is below the ${MIN_STAGE_TIMEOUT_MS}ms (1min) floor; falling back to ${DEFAULT_STAGE_TIMEOUT_MS}ms`,
|
||||
`[sync] ${envName}=${n} is below the ${MIN_STAGE_TIMEOUT_MS}ms (1min) floor; falling back to ${defaultMs}ms`,
|
||||
);
|
||||
return DEFAULT_STAGE_TIMEOUT_MS;
|
||||
return defaultMs;
|
||||
}
|
||||
if (n > MAX_STAGE_TIMEOUT_MS) {
|
||||
console.warn(
|
||||
`[sync] ${envName}=${n} is above the ${MAX_STAGE_TIMEOUT_MS}ms (24h) ceiling; falling back to ${DEFAULT_STAGE_TIMEOUT_MS}ms`,
|
||||
`[sync] ${envName}=${n} is above the ${MAX_STAGE_TIMEOUT_MS}ms (24h) ceiling; falling back to ${defaultMs}ms`,
|
||||
);
|
||||
return DEFAULT_STAGE_TIMEOUT_MS;
|
||||
return defaultMs;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
@@ -209,12 +239,19 @@ Options:
|
||||
--no-memory Skip the gstack-memory-ingest stage (transcripts + artifacts).
|
||||
--no-brain-sync Skip the gstack-brain-sync git pipeline stage.
|
||||
--code-only Only run the code-import stage (alias for --no-memory --no-brain-sync).
|
||||
--dream Force the source-scoped dream cycle that builds this
|
||||
source's call graph (gbrain code-callers/code-callees).
|
||||
Runs lock-free AFTER the sync stages. ~minutes. Default
|
||||
timeout 45min, override GSTACK_SYNC_DREAM_TIMEOUT_MS.
|
||||
--no-dream Opt out of the dream cycle that --full would auto-run.
|
||||
--allow-reclone Permit the code walk for URL-managed sources (remote_url set)
|
||||
even though gbrain may auto-reclone the working tree (#1734).
|
||||
--help This text.
|
||||
|
||||
Stages run in order: code → memory ingest → curated git push.
|
||||
Each stage failure is non-fatal; subsequent stages still run.
|
||||
Stages run in order: code → memory ingest → curated git push, then (lock-free)
|
||||
the optional dream call-graph build. --full auto-runs dream ONLY when the call
|
||||
graph was never built; --dream always forces it. Each stage failure is
|
||||
non-fatal; subsequent stages still run.
|
||||
`);
|
||||
}
|
||||
|
||||
@@ -226,6 +263,8 @@ function parseArgs(): CliArgs {
|
||||
let noMemory = false;
|
||||
let noBrainSync = false;
|
||||
let codeOnly = false;
|
||||
let dream = false;
|
||||
let noDream = false;
|
||||
let allowReclone = false;
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
@@ -244,6 +283,10 @@ function parseArgs(): CliArgs {
|
||||
noMemory = true;
|
||||
noBrainSync = true;
|
||||
break;
|
||||
// --dream forces the cycle; --full only chains it at the call site (so
|
||||
// --no-dream can override) — do NOT set dream from --full here.
|
||||
case "--dream": dream = true; break;
|
||||
case "--no-dream": noDream = true; break;
|
||||
case "--help":
|
||||
case "-h":
|
||||
printUsage();
|
||||
@@ -255,7 +298,7 @@ function parseArgs(): CliArgs {
|
||||
}
|
||||
}
|
||||
|
||||
return { mode, quiet, noCode, noMemory, noBrainSync, codeOnly, allowReclone };
|
||||
return { mode, quiet, noCode, noMemory, noBrainSync, codeOnly, dream, noDream, allowReclone };
|
||||
}
|
||||
|
||||
// ── Helpers ────────────────────────────────────────────────────────────────
|
||||
@@ -610,6 +653,58 @@ function releaseLock(): void {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire the dream marker (`~/.gstack/.dream-in-progress`). Returns false when
|
||||
* a FRESH marker already exists (another worktree is mid-dream) — the caller
|
||||
* then SKIPs rather than launching a duplicate ~35-min global job. A stale
|
||||
* marker (older than DREAM_MARKER_STALE_MS, i.e. a crashed run) is taken over.
|
||||
* Mirrors acquireLock but with the dream TTL and its own path.
|
||||
*/
|
||||
export function acquireDreamMarker(): boolean {
|
||||
const path = dreamMarkerPath();
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
if (existsSync(path)) {
|
||||
try {
|
||||
const stat = statSync(path);
|
||||
if (Date.now() - stat.mtimeMs > DREAM_MARKER_STALE_MS) {
|
||||
unlinkSync(path);
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
const info: LockInfo = { pid: process.pid, started_at: new Date().toISOString() };
|
||||
try {
|
||||
writeFileSync(path, JSON.stringify(info), { encoding: "utf-8", flag: "wx" });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export function releaseDreamMarker(): void {
|
||||
try {
|
||||
const path = dreamMarkerPath();
|
||||
if (!existsSync(path)) return;
|
||||
const info = JSON.parse(readFileSync(path, "utf-8")) as LockInfo;
|
||||
if (info.pid === process.pid) unlinkSync(path);
|
||||
} catch {
|
||||
// Best-effort cleanup.
|
||||
}
|
||||
}
|
||||
|
||||
/** Read the pid recorded in a fresh dream marker, for the "already running" message. */
|
||||
function dreamMarkerPid(): number | null {
|
||||
try {
|
||||
const info = JSON.parse(readFileSync(dreamMarkerPath(), "utf-8")) as LockInfo;
|
||||
return typeof info.pid === "number" ? info.pid : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Stage runners ──────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
@@ -624,7 +719,7 @@ function releaseLock(): void {
|
||||
* broken-db → "config points at unreachable DB; see /setup-gbrain Step 1.5"
|
||||
*/
|
||||
function skipStageForLocalStatus(
|
||||
stage: "code" | "memory",
|
||||
stage: "code" | "memory" | "dream",
|
||||
status: LocalEngineStatus,
|
||||
t0: number,
|
||||
): StageResult {
|
||||
@@ -1047,6 +1142,240 @@ function runBrainSyncPush(args: CliArgs): StageResult {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether the dream (call-graph build) cycle should run. PURE so the
|
||||
* gate matrix is unit-testable without spawning a real ~35-min dream.
|
||||
*
|
||||
* - explicit --dream → always run (force), regardless of cycle state / --no-code.
|
||||
* - --full → run ONLY when the call graph was never built (cycle === "never"),
|
||||
* and only when not opted out via --no-dream / --no-code. "completed" skips
|
||||
* (edges already built); "unknown" skips (a flaky doctor must not trigger a
|
||||
* surprise 35-min cycle — see gbrain-doctor-overstrict).
|
||||
* - everything else → skip.
|
||||
*
|
||||
* `cycle` is only consulted on the --full auto path; pass null when forcing.
|
||||
*/
|
||||
export function shouldRunDream(args: CliArgs, cycle: CycleStatus | null): boolean {
|
||||
if (args.dream) return true;
|
||||
if (args.mode === "full" && !args.noDream && !args.noCode) {
|
||||
return cycle === "never";
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `gbrain dream` — the brain-global maintenance cycle whose
|
||||
* resolve_symbol_edges phase builds the call graph. Runs LOCK-FREE (called
|
||||
* after the sync lock releases) so it never freezes sibling worktrees; the
|
||||
* `.dream-in-progress` marker dedupes concurrent dreams instead.
|
||||
*
|
||||
* Returns a StageResult (never throws). SKIP (ran:false, ok:true) for: dry-run
|
||||
* preview, local engine not ok, or a fresh marker present. ERR (ran:true,
|
||||
* ok:false) for: non-zero/timeout exit, or a spawn-setup failure (missing
|
||||
* binary / malformed env) — a broken install must be visible, not disguised as
|
||||
* optional maintenance.
|
||||
*/
|
||||
export async function runDream(args: CliArgs): Promise<StageResult> {
|
||||
const t0 = Date.now();
|
||||
|
||||
if (args.mode === "dry-run") {
|
||||
const root = repoRoot();
|
||||
const sourceId = root ? deriveCodeSourceId(root) : null;
|
||||
return {
|
||||
name: "dream",
|
||||
ran: false,
|
||||
ok: true,
|
||||
duration_ms: 0,
|
||||
summary: sourceId
|
||||
? `would: gbrain dream --source ${sourceId} (build this source's call graph)`
|
||||
: "would: gbrain dream (call-graph build)",
|
||||
};
|
||||
}
|
||||
|
||||
const localStatus = localEngineStatus({ noCache: false });
|
||||
if (localStatus !== "ok") {
|
||||
return skipStageForLocalStatus("dream", localStatus, t0);
|
||||
}
|
||||
|
||||
// Dedupe concurrent dreams across worktrees (lock-free path).
|
||||
if (!acquireDreamMarker()) {
|
||||
const pid = dreamMarkerPid();
|
||||
return {
|
||||
name: "dream",
|
||||
ran: false,
|
||||
ok: true,
|
||||
duration_ms: Date.now() - t0,
|
||||
summary: `dream already running${pid !== null ? ` (pid ${pid})` : ""} — skipped`,
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
const dreamTimeoutMs = resolveStageTimeoutMs(
|
||||
process.env.GSTACK_SYNC_DREAM_TIMEOUT_MS,
|
||||
"GSTACK_SYNC_DREAM_TIMEOUT_MS",
|
||||
DEFAULT_DREAM_TIMEOUT_MS,
|
||||
);
|
||||
|
||||
// Scope the cycle to THIS worktree's code source: `gbrain dream --source <id>`.
|
||||
// Verified empirically (not just from `gbrain --help`): plain `gbrain dream`
|
||||
// cycles the brain's default source and never runs the source-scoped `extract`
|
||||
// phase for our code source, so the call graph for the pinned source stays
|
||||
// empty. `gbrain dream --source <id>` runs the per-source cycle (the form
|
||||
// `gbrain doctor` recommends for stale sources) and is what actually populates
|
||||
// code-callers/code-callees for this worktree. Falls back to plain `dream`
|
||||
// only when we can't derive the source id (not in a git repo).
|
||||
const root = repoRoot();
|
||||
const sourceId = root ? deriveCodeSourceId(root) : null;
|
||||
const dreamArgs = sourceId ? ["dream", "--source", sourceId] : ["dream"];
|
||||
|
||||
// spawnGbrain seeds DATABASE_URL from gbrain's config via buildGbrainEnv.
|
||||
//
|
||||
// We CAPTURE output (pipe) rather than inherit because `gbrain dream` exits 0
|
||||
// even when it SKIPS the cycle — when another cycle already holds gbrain's own
|
||||
// DB lock (e.g. a running `gbrain autopilot`), it prints "Skipped: another
|
||||
// cycle is already running. (locked)" and exits 0. Trusting the exit code
|
||||
// alone would falsely report "call graph built". Trade-off: no live streaming
|
||||
// for a long cycle; we echo the captured output afterward instead.
|
||||
if (!args.quiet) {
|
||||
process.stderr.write("[dream] running gbrain cycle (call-graph build; this can take a few minutes)...\n");
|
||||
}
|
||||
let result: ReturnType<typeof spawnGbrain>;
|
||||
try {
|
||||
result = spawnGbrain(dreamArgs, {
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
timeout: dreamTimeoutMs,
|
||||
baseEnv: process.env,
|
||||
announce: !args.quiet,
|
||||
});
|
||||
} catch (err) {
|
||||
// Spawn-setup failure (missing binary, bad env): ERR, not a benign skip.
|
||||
return {
|
||||
name: "dream",
|
||||
ran: true,
|
||||
ok: false,
|
||||
duration_ms: Date.now() - t0,
|
||||
summary: `gbrain dream failed to start: ${(err as Error).message}`,
|
||||
};
|
||||
}
|
||||
|
||||
if (result.error) {
|
||||
const e = result.error as NodeJS.ErrnoException;
|
||||
const why = e.code === "ENOENT" ? "gbrain not on PATH" : e.message;
|
||||
return {
|
||||
name: "dream",
|
||||
ran: true,
|
||||
ok: false,
|
||||
duration_ms: Date.now() - t0,
|
||||
summary: `gbrain dream failed to start: ${why}`,
|
||||
};
|
||||
}
|
||||
|
||||
const out = `${result.stdout || ""}${result.stderr || ""}`;
|
||||
if (!args.quiet && out.trim()) {
|
||||
process.stderr.write(out.endsWith("\n") ? out : `${out}\n`);
|
||||
}
|
||||
|
||||
if (result.status !== 0) {
|
||||
return {
|
||||
name: "dream",
|
||||
ran: true,
|
||||
ok: false,
|
||||
duration_ms: Date.now() - t0,
|
||||
summary: `gbrain dream exited ${result.status === null ? "null (killed by signal / timeout)" : result.status}`,
|
||||
};
|
||||
}
|
||||
|
||||
// Exit 0 but the cycle was SKIPPED because gbrain's own lock is held by
|
||||
// another cycle (typically `gbrain autopilot`). Report SKIP, not "built" —
|
||||
// the graph builds on that other cycle, not this invocation.
|
||||
if (/already running|\block(?:ed)?\b|Skipped:/i.test(out)) {
|
||||
return {
|
||||
name: "dream",
|
||||
ran: false,
|
||||
ok: true,
|
||||
duration_ms: Date.now() - t0,
|
||||
summary: "skipped — a gbrain cycle is already running (e.g. autopilot); the call graph builds on that cycle",
|
||||
};
|
||||
}
|
||||
|
||||
// Exit 0 and the cycle actually ran. Parse the cycle's OWN output to report
|
||||
// the truth, not a flat "built": `gbrain dream` exits 0 even when the call
|
||||
// graph could not be built, and a misleading "built" turns a multi-minute
|
||||
// no-op into a silent dead end. gbrain only surfaces these conditions in the
|
||||
// cycle log (there is no pre-flight pack-capability query as of 0.41.x), so
|
||||
// string-matching the log is the available signal; an unrecognized log
|
||||
// degrades to the generic success summary below.
|
||||
const dreamWarn = classifyDreamOutcome(out);
|
||||
if (dreamWarn) {
|
||||
return {
|
||||
name: "dream",
|
||||
ran: true,
|
||||
ok: true,
|
||||
warn: true,
|
||||
duration_ms: Date.now() - t0,
|
||||
summary: dreamWarn,
|
||||
};
|
||||
}
|
||||
|
||||
const edges = parseResolvedEdges(out);
|
||||
return {
|
||||
name: "dream",
|
||||
ran: true,
|
||||
ok: true,
|
||||
duration_ms: Date.now() - t0,
|
||||
summary:
|
||||
edges !== null
|
||||
? `call graph built (${edges} edge${edges === 1 ? "" : "s"} resolved)`
|
||||
: "call graph built (resolve_symbol_edges complete)",
|
||||
};
|
||||
} finally {
|
||||
releaseDreamMarker();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse `<n>` from a `resolve_symbol_edges ... resolved <n>` cycle-log line.
|
||||
* Returns null when the line is absent (older gbrain / different pack). The
|
||||
* `[^\n]*?` is newline-bounded so it matches the `✓ resolve_symbol_edges ...`
|
||||
* summary line, not the bracketed `[cycle.resolve_symbol_edges] start` markers.
|
||||
*/
|
||||
export function parseResolvedEdges(out: string): number | null {
|
||||
const m = out.match(/resolve_symbol_edges\b[^\n]*?\bresolved\s+(\d+)/i);
|
||||
return m ? parseInt(m[1], 10) : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Inspect a completed (exit-0) `gbrain dream` log and return a WARN summary when
|
||||
* the cycle ran but could not actually build the call graph. Returns null on the
|
||||
* happy path (caller emits the normal "call graph built" summary). Order matters:
|
||||
* the pack-capability gap is the most actionable, so it wins over a 0-edge count
|
||||
* (both appear together when the pack lacks the code-symbol phase).
|
||||
*/
|
||||
export function classifyDreamOutcome(out: string): string | null {
|
||||
// The active schema pack doesn't declare the code-symbol extraction phase, so
|
||||
// no symbols are extracted and resolve_symbol_edges has nothing to match.
|
||||
if (/does not declare this phase/i.test(out)) {
|
||||
return (
|
||||
"dream ran, but this source's schema pack does not extract code symbols, " +
|
||||
"so the call graph stays empty. Switch this source to a code-aware schema " +
|
||||
"pack (`gbrain schema use <pack>`) to enable code-callers/code-callees."
|
||||
);
|
||||
}
|
||||
// The embed phase failed for a missing key; symbols can't index without it.
|
||||
if (/embed phase failed/i.test(out) || /requires\s+\S*_API_KEY/i.test(out)) {
|
||||
return (
|
||||
"dream ran, but the embed phase failed (missing embedding API key), so " +
|
||||
"symbols won't index. Ensure the embedding provider's key is set for the " +
|
||||
"gbrain process, then re-run /sync-gbrain --dream."
|
||||
);
|
||||
}
|
||||
// Cycle ran and embedded fine, but matched zero call-graph edges.
|
||||
if (parseResolvedEdges(out) === 0) {
|
||||
return "dream ran but resolved 0 call-graph edges (no code symbols matched for this source yet).";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// ── State file ─────────────────────────────────────────────────────────────
|
||||
|
||||
interface SyncState {
|
||||
@@ -1085,10 +1414,28 @@ function saveSyncState(state: SyncState): void {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist the dream stage result with read-modify-write semantics.
|
||||
*
|
||||
* Dream runs AFTER the sync lock releases, so a sibling worktree may have
|
||||
* written newer state in the meantime. Overwriting the whole file with our
|
||||
* pre-dream snapshot + dream result would clobber that sibling's sync. Instead
|
||||
* re-read the CURRENT state, replace only the `dream` entry in last_stages, and
|
||||
* atomic-rename. (Atomic rename alone isn't race-safe; the re-read + targeted
|
||||
* merge is what prevents the clobber.)
|
||||
*/
|
||||
function mergeDreamIntoState(dream: StageResult): void {
|
||||
const fresh = loadSyncState();
|
||||
const others = (fresh.last_stages || []).filter((s) => s.name !== "dream");
|
||||
fresh.last_stages = [...others, dream];
|
||||
fresh.last_sync = new Date().toISOString();
|
||||
saveSyncState(fresh);
|
||||
}
|
||||
|
||||
// ── Output ─────────────────────────────────────────────────────────────────
|
||||
|
||||
function formatStage(s: StageResult): string {
|
||||
const status = !s.ran ? "SKIP" : s.ok ? "OK" : "ERR";
|
||||
export function formatStage(s: StageResult): string {
|
||||
const status = !s.ran ? "SKIP" : !s.ok ? "ERR" : s.warn ? "WARN" : "OK";
|
||||
const dur = s.duration_ms > 0 ? ` (${(s.duration_ms / 1000).toFixed(1)}s)` : "";
|
||||
return ` ${status.padEnd(5)} ${s.name.padEnd(12)} ${s.summary}${dur}`;
|
||||
}
|
||||
@@ -1124,9 +1471,9 @@ async function main(): Promise<void> {
|
||||
process.on("SIGTERM", () => { cleanup(); process.exit(143); });
|
||||
|
||||
let exitCode = 0;
|
||||
const stages: StageResult[] = [];
|
||||
try {
|
||||
const state = loadSyncState();
|
||||
const stages: StageResult[] = [];
|
||||
|
||||
if (!args.noCode) {
|
||||
stages.push(await withErrorContext("sync:code", () => runCodeImport(args), "gstack-gbrain-sync"));
|
||||
@@ -1145,20 +1492,61 @@ async function main(): Promise<void> {
|
||||
saveSyncState(state);
|
||||
}
|
||||
|
||||
if (!args.quiet || args.mode === "dry-run") {
|
||||
console.log(`\ngstack-gbrain-sync (${args.mode}):`);
|
||||
for (const s of stages) console.log(formatStage(s));
|
||||
const okCount = stages.filter((s) => s.ok).length;
|
||||
const errCount = stages.filter((s) => !s.ok && s.ran).length;
|
||||
console.log(`\n ${okCount} ok, ${errCount} error, ${stages.length - okCount - errCount} skipped`);
|
||||
}
|
||||
|
||||
const anyError = stages.some((s) => s.ran && !s.ok);
|
||||
exitCode = anyError ? 1 : 0;
|
||||
} finally {
|
||||
// Release the sync lock BEFORE the dream cycle. Dream is a source-scoped
|
||||
// cycle that can run several minutes; holding the machine-wide lock that
|
||||
// long would freeze every other worktree's /sync-gbrain. Dream is guarded
|
||||
// by its own marker.
|
||||
cleanup();
|
||||
}
|
||||
|
||||
// ── Dream (call-graph build) — LOCK-FREE, after the sync lock releases ─────
|
||||
let dreamStage: StageResult | null = null;
|
||||
if (args.mode === "dry-run") {
|
||||
// Preview only; never probes doctor or spawns. `--dry-run` and `--full` are
|
||||
// mutually exclusive modes (last one wins in parseArgs), so the only dream
|
||||
// preview that applies to a dry-run is the explicit --dream force.
|
||||
if (args.dream) {
|
||||
dreamStage = await runDream(args);
|
||||
}
|
||||
} else {
|
||||
// Resolve cycle state only on the --full auto path (perf: the steady-state
|
||||
// incremental sync never pays a doctor subprocess). Explicit --dream forces.
|
||||
let cycle: CycleStatus | null = null;
|
||||
if (!args.dream && args.mode === "full" && !args.noDream && !args.noCode) {
|
||||
const root = repoRoot();
|
||||
cycle = root ? cycleCompleted(deriveCodeSourceId(root), process.env) : "unknown";
|
||||
}
|
||||
if (shouldRunDream(args, cycle)) {
|
||||
dreamStage = await runDream(args);
|
||||
mergeDreamIntoState(dreamStage);
|
||||
if (dreamStage.ran && !dreamStage.ok) exitCode = 1;
|
||||
} else if (cycle === "unknown") {
|
||||
// --full wanted to auto-build but doctor couldn't confirm the graph state.
|
||||
// Surface a WARN-style SKIP so the user knows to run --dream if needed,
|
||||
// rather than silently doing nothing (a flaky doctor must not trigger a
|
||||
// surprise 35-min run — gbrain-doctor-overstrict).
|
||||
dreamStage = {
|
||||
name: "dream",
|
||||
ran: false,
|
||||
ok: true,
|
||||
duration_ms: 0,
|
||||
summary: "call-graph state unknown (doctor unavailable) — run /sync-gbrain --dream if code-callers returns 0",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if (!args.quiet || args.mode === "dry-run") {
|
||||
const allStages = dreamStage ? [...stages, dreamStage] : stages;
|
||||
console.log(`\ngstack-gbrain-sync (${args.mode}):`);
|
||||
for (const s of allStages) console.log(formatStage(s));
|
||||
const okCount = allStages.filter((s) => s.ok).length;
|
||||
const errCount = allStages.filter((s) => !s.ok && s.ran).length;
|
||||
console.log(`\n ${okCount} ok, ${errCount} error, ${allStages.length - okCount - errCount} skipped`);
|
||||
}
|
||||
|
||||
process.exit(exitCode);
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user