mirror of
https://github.com/garrytan/gstack.git
synced 2026-06-23 02:00:00 +02:00
v1.58.4.0 fix: high-priority community bug wave + PTY plan-mode smoke gate (#2077)
* fix(gbrain): stop forcing GBRAIN_PREPARE on transaction-mode poolers (#1965) buildGbrainEnv auto-set GBRAIN_PREPARE=true whenever DATABASE_URL targeted port 6543, and the /sync-gbrain capability check exported it for the rest of the skill run. Both had the semantics inverted: gbrain auto-disables prepared statements on transaction-mode poolers because they break every write there ("prepared statement does not exist"); GBRAIN_PREPARE=true is gbrain's documented override for SESSION-mode poolers on 6543, not a requirement for transaction mode. The #1435 search symptom the auto-set worked around was fixed gbrain-side. Remove both force-sets. A caller-set GBRAIN_PREPARE (either value) still passes through untouched, preserving the session-mode-on-6543 escape hatch. isTransactionModePooler stays exported. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(gbrain): classify probe timeout as its own status; sync proceeds instead of skipping (#1964) The 5s engine probe misclassified healthy-but-slow engines (cold Supabase pooler connections measured at 6.9-10.7s) as broken-config, so /sync-gbrain silently skipped code+memory and told the user their config was malformed. - New "timeout" status: probe killed at the deadline with no recognized stderr pattern. Default deadline is now 15s, overridable via GSTACK_GBRAIN_PROBE_TIMEOUT_MS (tests set 300ms against a fake that sleeps 2s). - Sync stages PROCEED on timeout with a stderr warning naming the env knob; a genuinely-dead engine surfaces its real error at the first operation instead of a false config diagnosis. - Consistency everywhere "ok" gated behavior: gstack-gbrain-detect --is-ok exits 0 on timeout, and gen-skill-docs' detection gate accepts it, so a slow engine no longer silently suppresses brain-aware features. - Status cache: key now includes the effective probe timeout (raising it invalidates a cached timeout) and GBRAIN_HOME; config detection honors GBRAIN_HOME so relocated-home users stop being misclassified as missing-config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bins): cygpath-normalize SCRIPT_DIR for bun imports; surface learnings-log errors (#1950) Under Windows git-bash, pwd yields a POSIX path (/c/Users/...) that Bun on Windows cannot resolve as an ES module specifier. gstack-learnings-log interpolates SCRIPT_DIR into a bun -e import, so every invocation died with "Cannot find module" — and 2>/dev/null swallowed the error, silently dropping every AI-logged learning for Windows users. - 3-line cygpath -m guard in gstack-learnings-log and gstack-question-log (which gains the same import shape in the next commit). Matches the duplicated IS_WINDOWS convention in setup; no shared shell lib exists. - learnings-log adopts question-log's set +e / TMPERR capture pattern wholesale: validation errors now print to stderr. The old `if [ $? -ne 0 ]` check was dead code under set -euo pipefail — the script exited at the failing assignment before reaching it. - New test/bin-windows-bun-import-paths.test.ts: static invariant (any bash bin interpolating $SCRIPT_DIR into a bun -e import must carry the guard) + behavioral end-to-end run invoked via `bash <bin>` — added to the windows-free-tests workflow list so the conversion is proven on the only platform where the bug exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(question-log): dedupe INJECTION_PATTERNS via lib/jsonl-store (#1934) bin/gstack-question-log carried a local copy of the injection-pattern list, so pattern fixes to lib/jsonl-store.ts never propagated — including the /override[:\s]/i false-positive fix arriving via community PR #1940. Import the shared hasInjection instead (enabled by the previous commit's cygpath guard). question-log also gets the lib's stricter superset (human:, disregard, from-now-on, approve-all patterns). Tests pin the contract in a #1940-order-independent way: an "Override: ignore all previous instructions" header is rejected, "prose overrides the deterministic table" is accepted, and a static invariant keeps local INJECTION_PATTERNS duplicates out of the bin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(security): community-pulse + both dashboards never report fake zeros (#1947) The security-signaling surface failed open at three layers — every failure mode read as a reassuring "0 attacks" / "0 installs": - community-pulse edge function: supabase-js returns {data,error} without throwing, and all five queries discarded `error` — a DB outage produced real-looking zeros via the SUCCESS path, and the catch (also returning zeros with HTTP 200) was unreachable for query failures. Every query now destructures and throws; the catch serves the stale cache (marked "stale": true) when one exists, else 503 {"error":"pulse_unavailable"}. Success responses carry "status":"ok" so clients can distinguish authoritative data from legacy backends. NOTE: the edge function deploys out-of-band (supabase functions deploy community-pulse). - gstack-security-dashboard: captures the HTTP status; non-200 / network failure / error body / missing section → "unknown — backend error"; jq missing → "unknown — install jq" (the lossy grep fallback broke on nested arrays and under-reported attacks as zero — removed); a 200 without the new marker shows figures with an "unverified (legacy backend)" note. Also fixes a latent display bug: the TOTAL grep matched the digit 7 inside "attacks_last_7_days" and misreported every count. - gstack-community-dashboard: same class — curl || echo "{}" plus grep || echo "0" printed "Weekly active installs: 0" on any failure. Now "unknown — backend error (HTTP N)". test/security-dashboard-fallback.test.ts pins the matrix (200+marker, 200-legacy, 503, network failure) x (jq present, jq absent) for both bins: "unknown" states never render as 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): redact error_message spans before they leave the machine (#1947) error_message was uploaded with only quote/newline escaping — stack traces and failed-API errors can embed credentials, private paths, and hostnames, and the sync path strips only _repo_slug/_branch. New lib/redact-engine.ts export redactFindingSpans(): replaces EVERY finding's span with <REDACTED-{id}> regardless of tier (applyRedactions is the interactive PII-only path and exits nonzero on credential findings, so it can't serve machine egress). Returns null when a span can't be located — callers drop the whole payload rather than risk a leak. gstack-telemetry-log pipes error_message through it at LOG time, so the local JSONL at rest is clean too; surrounding text survives for crash triage. FAIL CLOSED: bun missing, engine error, or non-JSON-string output all null the field. Tests pin: embedded ghp_ token → <REDACTED-github.pat> with context intact; redactor unavailable → null; raw bytes on disk never contain the token. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(redact): prepush guard fails closed on git failure; /ship owns hook install (#1946) Two gaps closed: 1. Fail closed. The git() helper returned "" on ANY non-zero exit or maxBuffer overflow (status null), addedLinesFor produced an empty string, and the push sailed through unscanned — fail-open on exactly the oversized-diff case where a large secret-bearing blob is most likely. The diff call now uses a strict variant that throws; main blocks with a clear message naming the GSTACK_REDACT_PREPUSH=skip escape valve. Probe calls (symbolic-ref, rev-parse, merge-base) keep the permissive helper — their failures are normal control flow. 2. Install path. The hook was installed by nothing ("opt-in, installed by nothing" was the issue's words). ./setup runs in the gstack checkout — the wrong repo for a per-project hook — so it gets a one-line hint only. /ship owns per-repo install: config redact_prepush_hook=true + hook missing → silent install (consent already given); config unset + no ~/.gstack/.redact-prepush-prompted marker → one-time machine-wide AskUserQuestion offer, answer persisted. ship/SKILL.md regenerated in this same commit (check-freshness bisect discipline). Tests: unscannable diff (bogus SHAs) → exit 1 + valve named; empty-but- successful diff → exit 0; static asserts pin setup as hint-only and the ship template as the installer surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(redact): six new credential patterns — GitLab, HuggingFace, npm, DigitalOcean, Bearer, GCP SA (#1946) Coverage gaps from the #1946 security review, including token types for tooling gstack itself drives (glab): HIGH (block): gitlab.token (glpat-/glptt-/gldt-), huggingface.token (hf_), npm.token (npm_), digitalocean.token (dop_v1_), gcp.service_account (the JSON-escaped "private_key" form that dodges pem.private_key's literal-block match when minified, confirmed by "private_key_id" proximity). MEDIUM (warn): auth.bearer — the most FP-prone shape in the set (docs are full of "Authorization: Bearer <token>"), so it requires header-context proximity and the same entropy>=3.0 + placeholder validator recipe as env.kv. "Bearer YOUR_TOKEN_HERE" never fires; calibration over coverage, per the cries-wolf principle. All shapes are linear-time; test/redact-pattern-lint.test.ts covers them automatically. Engine tests add positive + placeholder-negative cases per pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: coverage-audit additions for the fix wave Ship Step 7 gap-fill (all passing, 248 tests across the touched suites): memory + dream stage probe-timeout proceeds, gbrain-detect override paths, stale-flag passthrough, 200-body-missing-.security fail-closed case, telemetry redaction edges, and credential-pattern edge cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pre-landing review fixes Review army findings (1 critical, auto-fixed with regression tests): - CRITICAL (security specialist, verified live): redactFindingSpans spliced only the regex capture span, and pem.private_key / gcp.service_account capture just the BEGIN-header — the key body survived "redaction" and shipped via telemetry. Marker-only patterns now drop the whole payload (null, fail closed). Overlapping spans (Bearer+JWT on the same bytes) are coalesced before splicing so stale offsets can't leave partial secret bytes behind. - gitStrict: drop the dead `|| r.status === null` disjunct (null !== 0 already covers it); add the signal-kill/null-status regression test the docstring promised. - security-dashboard human mode flags stale snapshots ("figures may be out of date") instead of presenting frozen counts as current. - community-dashboard marker check uses jq when available — the grep-only variant misclassified whitespaced/reserialized bodies as legacy. - telemetry fail-closed test now shadows bun with a failing stub (deterministic on any host layout); stale "five status cases" describe title renamed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial review fixes (Claude + Codex cross-model passes) Both adversarial passes ran against the wave; every FIXABLE finding landed with a regression test: - probeTimeoutMs clamps to >=1ms: a fractional override floored to 0, and execFileSync treats timeout:0 as NO timeout — the probe that exists to bound hangs could hang forever (found by both models independently). - /ship silent hook install now requires the hooks dir to live inside .git: with core.hooksPath (husky's COMMITTED .husky/), the chaining installer would have renamed the team's committed pre-push and written a machine-local wrapper into the working tree (found by both models). - gstack-config gbrain-refresh accepts the "timeout" status — the last consumer still gating on literal "ok" (Codex); gstack-gbrain-detect's config-derived fields honor GBRAIN_HOME so the detection JSON can't report status ok alongside config_exists false (Codex). - prepush: a remote sha absent locally (shallow clone / stale fetch) falls back to the merge-base/empty-tree range — scans MORE, never blocks a legitimate push into training users toward --no-verify. - dashboards: curl's own 000 no longer doubles to "HTTP 000000"; the community dashboard flags stale snapshots like the security one; array sections parse via jq (the sed/grep loops truncated at the first ']'); the no-jq marker grep tolerates whitespace. - telemetry: multi-line redactor output nulls the field instead of corrupting the JSONL record; setup's hint fires only when the config key is genuinely unset (an explicit false is a recorded decline); the /ship prompt marker honors GSTACK_HOME. Kept as designed (cross-model tension noted): Bearer stays MEDIUM in the prepush gate — a HIGH Bearer would block every docs example; the entropy validator can't eliminate that FP class, and MEDIUM warns visibly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v1.57.11.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: P1 TODO — eval harness live progress + incremental persistence Root-caused during this ship: a killed eval run was indistinguishable from a healthy one for hours (per-file output buffering across mega test files, no incremental eval-store writes, no honest liveness signal). Full context and starting points in the entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: fix operational-learning E2E fixture — copy lib/jsonl-store.ts Pre-existing breakage, proven on main: gstack-learnings-log has imported lib/jsonl-store.ts (shared injection patterns) since v1.57.5.0 / #1910, but the fixture copies only the bin scripts — the bin exits 1 before writing anything, on main silently (stderr swallowed) and on this branch loudly (the #1950 error-surfacing made the four-day-old failure visible). A real install always ships bin/ and lib/ together; the fixture now does too. Verified: the fixture-shaped invocation writes the learning (exit 0) with lib present, exits 1 on both main and this branch without it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ios-qa): isolate E2E tests under --concurrent (3 real races) The ios-qa E2E file failed intermittently under `bun test --concurrent` (the eval harness default). Three distinct shared-state races, all fixed: 1. Shared pidfile: a module-level `workDir` reassigned in beforeEach was clobbered by parallel tests, so concurrent daemons collided on the same pidfile and the loser returned `already_running`. Each test now gets its own dir via makeWorkDir(). 2. process.env path globals: tests set GSTACK_IOS_AUDIT_PATH / _ATTEMPTS_PATH / _ALLOWLIST_PATH on the shared process env; concurrent tests stomped each other's audit/attempts destinations. Threaded auditPath/attemptsPath/allowlistPath through DaemonOptions (and mintForCaller) as explicit args — env is no longer load-bearing. 3. afterEach cleanup race: the per-test cleanup drained a shared dir array, so the first test to finish deleted still-running tests' workDirs mid-assertion. Moved to afterAll (cleans once, after all settle). Verified: 5/5 clean full-suite runs at --max-concurrency 15 (was intermittent); daemon unit suite 91/91; daemon source compiles. The paths default to the env-derived locations when options are omitted, so the production CLI path is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(pty): pin spawned claude to EVALS model chain (default claude-sonnet-4-6) launchClaudePty spawned the interactive `claude` TUI with no --model flag, so the child inherited the operator's ~/.claude/settings.json model. On a slow-thinking model that meant 5+ min of extended thinking on empty plan-mode context, timing out the plan-mode smoke tests regardless of contention. Pin the model via opts.model ?? EVALS_MODEL ?? 'claude-sonnet-4-6' — byte-identical to session-runner.ts:144, so PTY and `claude -p` evals always agree. Pushed before extraArgs (last flag wins, so a per-test --model still overrides). Placement leaves the spawn region byte-stable for a clean merge with the in-flight hermetic-env branch. Plumbed model through the three plan-skill wrappers. Static-grep tripwires guard the pin, its fallback chain, the before-extraArgs ordering, and all three wrapper forwards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(pty): detect markdown bold-bullet prose AUQs (fixes office-hours smoke) office-hours auto-mode renders its mode question as `- **Building a startup**` markdown bullets (office-hours/SKILL.md.tmpl:102) with no letter/number marker. isProseAUQVisible only matched `A)`-style lettered or `1.`-style numbered options, so the question went undetected: the model surfaced it at ~2m19s (well under the 300s budget) but the harness kept scoring the run "working" off the spinner glyphs and timed out — a false timeout on a question that was already on screen. Add Pattern 3: when an interrogative line ('?') is present AND 3+ bold-bullet markers (`- **`) appear in the 4KB tail, classify as a prose AUQ. Bold is the discriminator vs incidental prose bullets; the line anchor is dropped (stripAnsi can collapse option lines) and the existing `❯ 1.` cursor gate still defers to a live native list. Wires through the existing classifyVisible 'asked' path and the timeout high-water-mark, so office-hours now classifies 'asked' instead of 'timeout'. Five unit cases: the office-hours render passes; no-'?', <3-bullet, plain-bullet, and native-cursor cases stay false. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(pty): detect stripAnsi-collapsed prose AUQs + judge spinner-precedence The plan-eng/plan-design plan-mode + finding-floor smokes timed out even when the skill HAD rendered a complete prose AskUserQuestion and was waiting: the PTY strips cursor-positioning escapes, collapsing the option newlines/spaces so "A) ..." arrives as "A(recommended)" / "-B:" and "Reply with A, B, or C" as "ReplywithA,B,orC". Every line-anchored detector (Patterns 1-3) returns false on those bytes, so proseAUQEverObserved never latched and the run timed out on a question that was already on screen. Add Pattern 4/5: a two-signal collapsed-form detector — a reply/recommendation marker (space-insensitive "reply with [A-D]", "Recommendation:", or "(recommended)") AND 2+ distinct A-D letters each punctuated by ) : or (. The conjunction is what separates a real AUQ from incidental report prose; verified true on the verbatim failing-run buffers where Patterns 1-3 return false. Also fix the Haiku judge spinner bias: of 614 verdicts, 569 were 'working' and 95 of those noted a question was visible — Claude Code keeps the spinner animating at an idle prose decision, so the judge coin-flipped. Add a precedence override: when an option list AND a Recommendation/Reply instruction are both visible, classify WAITING even with spinner glyphs. Kept the strict dual-signal gate (never option-list-alone) so auto-decide-preserved doesn't flip. 5 unit tests pin the two-signal contract (2 true on real collapsed bytes, 3 false guards). 90 -> 95 pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(plan-review): ask-first scope gate for plan-eng + plan-design review On an empty/cold invocation, plan-eng-review and plan-design-review would dive straight into repo exploration (plan-eng) or a 7-pass mockup+audit (plan-design) and only ask the user much later, if at all. plan-ceo-review already asks first via an unconditional Step-0 gate and behaves well; these two did not. Add a hard-STOP scope gate as the FIRST operational instruction in each skill (above the design-doc check / pre-review audit / mockup defaults it explicitly overrides): the first tool call must be AskUserQuestion confirming the review target, before any git/Read/Grep/Glob/Bash or mockup generation. Under --disallowedTools the options render as plain column-0 lettered prose with a Recommendation + "Reply with A, B, or C" line so the answer is detectable. This is correct cold-start UX (confirm what to review before grinding a full review on nothing) and it is the product half of the plan-mode smoke fix; the harness collapsed-form detector is the deterministic half that catches the ask however it renders. Templates + regenerated SKILL.md (default variant). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(tiers): reclassify stochastic plan-eng/plan-design ask-first smokes as periodic plan-eng-review and plan-design-review run a long explore/audit before their first AskUserQuestion, so whether the plan-mode + finding-floor smokes reach a terminal outcome within the 300s/600s budget depends on stochastic ask-first compliance (measured ~50-67%/run even with the hardened gate). Per the "non-deterministic -> periodic" tiering rule, move the four affected smokes (plan-eng/plan-design review-plan-mode + finding-floor) to periodic. The deterministic harness fix (collapsed-form detector + judge precedence) and the ask-first gate lift these from always-failing to mostly-passing and are the real product+harness improvements; periodic monitoring tracks the rate weekly without blocking PRs on an LLM coin-flip. plan-ceo/plan-devex ask-first reliably and stay gate-tier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(evals): gate the deterministic PTY plan-mode smokes in CI The real-PTY plan-mode smokes never ran in CI — the gate was local-only. Add an e2e-pty-plan-smoke matrix suite running the two deterministically-reliable ones (office-hours-auto-mode, plan-mode-no-op) so a regression there blocks PRs. The stochastic plan-eng/plan-design ask-first smokes stay periodic (touchfiles E2E_TIERS) and are not CI-gated. A fresh CI container has no ~/.claude.json, so the spawned interactive `claude` would wedge on the onboarding + API-key-approval dialog. Add a scoped seed step (hasCompletedOnboarding + key approval, its own ANTHROPIC_API_KEY env) before the run — mirrors what the hermetic E2E child env seeds. Per-suite timeout override (35 min) via matrix.suite.timeout so the PTY suite has headroom for --retry 2 without bumping the other 12 suites. Report runner count 12 -> 13. Validate via workflow_dispatch before relying on the gate (PTY-in-CI is new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(evals): install gstack skill registry for the PTY smoke suite The first dry-run of e2e-pty-plan-smoke failed: the spawned interactive `claude` printed "Unknown command: /plan-ceo-review". .claude/skills is gitignored, so a fresh CI checkout has no gstack skill registry and the TUI can't resolve /office-hours or /plan-ceo-review. Add a Register step (scoped to the suite, after Seed, before Run) that mirrors setup's --no-prefix user-scoped registry minimally: $HOME/.claude/skills/gstack -> repo (resolves the preambles' absolute ~/.claude/skills/gstack/bin/* and <skill>/sections/* paths) + per-skill SKILL.md/sections symlinks for the two skills these tests invoke. HOME is /github/home in this container and the runner adds no HOME/CLAUDE_CONFIG_DIR override (no hermetic mode), so $HOME is the right anchor — the Seed step already proved claude reads it. No ./setup (binary build + Chromium + fonts + /dev/tty prompt); SKILL.md + bin/ + sections/ are committed. Self-validating: fails the step loudly on a dangling symlink or missing `name:` frontmatter, so a moved target surfaces here instead of as a silent 35-min "Unknown command" timeout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v1.58.4.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+9
-22
@@ -58,10 +58,10 @@ export interface BuildGbrainEnvOptions {
|
||||
* Detect whether a DATABASE_URL targets a PgBouncer transaction-mode pooler.
|
||||
*
|
||||
* Supabase transaction-mode poolers conventionally run on port 6543 at
|
||||
* `*.pooler.supabase.com`. When gbrain connects through one of these, it
|
||||
* auto-disables prepared statements — but search requires them (#1435).
|
||||
* Returns `true` when the URL looks like a transaction-mode pooler so the
|
||||
* caller can set `GBRAIN_PREPARE=true` to re-enable prepared statements.
|
||||
* `*.pooler.supabase.com`. gbrain auto-disables prepared statements on these
|
||||
* (prepared statements break under transaction pooling — #1965); its banner
|
||||
* documents `GBRAIN_PREPARE=true` as the override for poolers that actually
|
||||
* run in session mode on 6543.
|
||||
*/
|
||||
export function isTransactionModePooler(url: string): boolean {
|
||||
try {
|
||||
@@ -83,10 +83,11 @@ export function isTransactionModePooler(url: string): boolean {
|
||||
* - the config has no `database_url`,
|
||||
* - the caller already set DATABASE_URL to the same value.
|
||||
*
|
||||
* When the effective DATABASE_URL targets a PgBouncer transaction-mode
|
||||
* pooler (port 6543), sets `GBRAIN_PREPARE=true` so gbrain re-enables
|
||||
* prepared statements needed for search (#1435). Caller can override
|
||||
* with `GBRAIN_PREPARE=false` in the base env.
|
||||
* GBRAIN_PREPARE is never set here (#1965): gbrain auto-disables prepared
|
||||
* statements on transaction-mode poolers itself, and forcing them on breaks
|
||||
* every write with "prepared statement does not exist". A caller-set
|
||||
* GBRAIN_PREPARE (either value) passes through untouched — that remains the
|
||||
* documented override for session-mode poolers on port 6543.
|
||||
*
|
||||
* Always returns a fresh object — mutating the returned env never
|
||||
* affects the caller's env. Tests assert on effective values, not
|
||||
@@ -120,20 +121,6 @@ export function buildGbrainEnv(opts: BuildGbrainEnvOptions = {}): NodeJS.Process
|
||||
}
|
||||
}
|
||||
|
||||
// PgBouncer transaction-mode pooler detection (#1435): when the effective
|
||||
// DATABASE_URL targets port 6543 (Supabase transaction-mode convention),
|
||||
// gbrain auto-disables prepared statements — but search needs them.
|
||||
// Set GBRAIN_PREPARE=true unless the caller explicitly opted out.
|
||||
const effectiveUrl = out.DATABASE_URL || cfg.database_url;
|
||||
if (effectiveUrl && !out.GBRAIN_PREPARE && isTransactionModePooler(effectiveUrl)) {
|
||||
out.GBRAIN_PREPARE = "true";
|
||||
if (opts.announce) {
|
||||
process.stderr.write(
|
||||
`[gbrain-exec] set GBRAIN_PREPARE=true (port 6543 transaction-mode pooler detected)\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
|
||||
+57
-16
@@ -1,5 +1,5 @@
|
||||
/**
|
||||
* gbrain-local-status — classify the local gbrain engine into 5 states.
|
||||
* gbrain-local-status — classify the local gbrain engine into 6 states.
|
||||
*
|
||||
* Shared between bin/gstack-gbrain-detect (preamble probe on every skill start)
|
||||
* and bin/gstack-gbrain-sync.ts (orchestrator SKIP-when-not-ok semantics).
|
||||
@@ -9,15 +9,19 @@
|
||||
* - Probe: `gbrain sources list --json`. Cheap (~80ms), actually hits the DB.
|
||||
* Uses the same stderr patterns as lib/gbrain-sources.ts:66-67.
|
||||
* - Cache: 60s TTL at ~/.gstack/.gbrain-local-status-cache.json, keyed on
|
||||
* {home, path_hash, gbrain_bin_path, gbrain_version, config_mtime}.
|
||||
* {home, gbrain_home, path_hash, gbrain_bin_path, gbrain_version,
|
||||
* config_mtime, probe_timeout_ms}.
|
||||
* - --no-cache bypass: /setup-gbrain and /sync-gbrain pass it after any
|
||||
* state-mutating operation so the next read sees fresh status.
|
||||
*
|
||||
* No-cli → gbrain not on PATH.
|
||||
* Missing → CLI present, ~/.gbrain/config.json absent.
|
||||
* Missing → CLI present, config.json absent (honors GBRAIN_HOME).
|
||||
* Broken-config → config exists but `gbrain sources list` fails with config parse error
|
||||
* (or any non-recognized error — defensive default per codex #8).
|
||||
* Broken-db → config exists, DB unreachable per stderr classification.
|
||||
* Timeout → probe exceeded GSTACK_GBRAIN_PROBE_TIMEOUT_MS (default 15s) with no
|
||||
* recognized error — engine is likely healthy but slow (e.g. a cold
|
||||
* pooler connection, #1964). Consumers treat this as usable.
|
||||
* Ok → DB reachable, sources list returned valid JSON.
|
||||
*/
|
||||
|
||||
@@ -42,7 +46,8 @@ export type LocalEngineStatus =
|
||||
| "no-cli"
|
||||
| "missing-config"
|
||||
| "broken-config"
|
||||
| "broken-db";
|
||||
| "broken-db"
|
||||
| "timeout";
|
||||
|
||||
export interface ClassifyOptions {
|
||||
/** Bypass the 60s cache. Used after any state-mutating operation. */
|
||||
@@ -64,20 +69,38 @@ interface CacheEntry {
|
||||
/** Cache invariants — entry is invalidated if any of these change between writes. */
|
||||
key: {
|
||||
home: string;
|
||||
gbrain_home: string; // honors GBRAIN_HOME (#1964 / codex D11)
|
||||
path_hash: string;
|
||||
gbrain_bin_path: string;
|
||||
gbrain_version: string;
|
||||
config_mtime: number; // 0 when config absent
|
||||
config_size: number; // 0 when config absent
|
||||
probe_timeout_ms: number; // raising the timeout invalidates a cached "timeout"
|
||||
};
|
||||
}
|
||||
|
||||
export const CACHE_TTL_MS = 60_000;
|
||||
export const PROBE_TIMEOUT_MS = 5_000;
|
||||
export const DEFAULT_PROBE_TIMEOUT_MS = 15_000;
|
||||
|
||||
/**
|
||||
* Effective probe timeout. `GSTACK_GBRAIN_PROBE_TIMEOUT_MS` overrides the
|
||||
* 15s default (tests set it low; users with slow poolers raise it).
|
||||
* Non-numeric or non-positive values fall back to the default.
|
||||
*/
|
||||
export function probeTimeoutMs(env?: NodeJS.ProcessEnv): number {
|
||||
const raw = (env ?? process.env).GSTACK_GBRAIN_PROBE_TIMEOUT_MS;
|
||||
if (!raw) return DEFAULT_PROBE_TIMEOUT_MS;
|
||||
const parsed = Number(raw);
|
||||
if (!Number.isFinite(parsed) || parsed <= 0) return DEFAULT_PROBE_TIMEOUT_MS;
|
||||
// Floor of 1ms: Math.floor(0.5) would yield 0, and execFileSync treats
|
||||
// timeout: 0 as NO timeout — the probe that exists to bound hangs would
|
||||
// itself hang forever (adversarial review finding 2).
|
||||
return Math.max(1, Math.floor(parsed));
|
||||
}
|
||||
|
||||
/** Effective user home — respects HOME env override (used by tests). */
|
||||
function userHome(): string {
|
||||
return process.env.HOME || homedir();
|
||||
function userHome(env?: NodeJS.ProcessEnv): string {
|
||||
return (env ?? process.env).HOME || homedir();
|
||||
}
|
||||
|
||||
/** Cache path computed fresh on each call so tests can mutate GSTACK_HOME per case. */
|
||||
@@ -88,8 +111,11 @@ export function cacheFilePath(): string {
|
||||
);
|
||||
}
|
||||
|
||||
function gbrainConfigPath(): string {
|
||||
return join(userHome(), ".gbrain", "config.json");
|
||||
/** Honors GBRAIN_HOME (codex D11) — same resolution as buildGbrainEnv. */
|
||||
function gbrainConfigPath(env?: NodeJS.ProcessEnv): string {
|
||||
const e = env ?? process.env;
|
||||
const gbrainHome = e.GBRAIN_HOME || join(userHome(e), ".gbrain");
|
||||
return join(gbrainHome, "config.json");
|
||||
}
|
||||
|
||||
function hashPath(p: string): string {
|
||||
@@ -146,9 +172,9 @@ export function readGbrainVersion(env?: NodeJS.ProcessEnv): string {
|
||||
return result;
|
||||
}
|
||||
|
||||
function configFingerprint(): { mtime: number; size: number } {
|
||||
function configFingerprint(env?: NodeJS.ProcessEnv): { mtime: number; size: number } {
|
||||
try {
|
||||
const st = statSync(gbrainConfigPath());
|
||||
const st = statSync(gbrainConfigPath(env));
|
||||
return { mtime: Math.floor(st.mtimeMs), size: st.size };
|
||||
} catch {
|
||||
return { mtime: 0, size: 0 };
|
||||
@@ -161,25 +187,29 @@ function buildCacheKey(
|
||||
env?: NodeJS.ProcessEnv,
|
||||
): CacheEntry["key"] {
|
||||
const e = env ?? process.env;
|
||||
const config = configFingerprint();
|
||||
const config = configFingerprint(e);
|
||||
return {
|
||||
home: e.HOME || "",
|
||||
gbrain_home: e.GBRAIN_HOME || "",
|
||||
path_hash: hashPath(e.PATH || ""),
|
||||
gbrain_bin_path: gbrainBin || "",
|
||||
gbrain_version: gbrainVersion,
|
||||
config_mtime: config.mtime,
|
||||
config_size: config.size,
|
||||
probe_timeout_ms: probeTimeoutMs(e),
|
||||
};
|
||||
}
|
||||
|
||||
function keysEqual(a: CacheEntry["key"], b: CacheEntry["key"]): boolean {
|
||||
return (
|
||||
a.home === b.home &&
|
||||
a.gbrain_home === b.gbrain_home &&
|
||||
a.path_hash === b.path_hash &&
|
||||
a.gbrain_bin_path === b.gbrain_bin_path &&
|
||||
a.gbrain_version === b.gbrain_version &&
|
||||
a.config_mtime === b.config_mtime &&
|
||||
a.config_size === b.config_size
|
||||
a.config_size === b.config_size &&
|
||||
a.probe_timeout_ms === b.probe_timeout_ms
|
||||
);
|
||||
}
|
||||
|
||||
@@ -226,7 +256,7 @@ function freshClassify(env?: NodeJS.ProcessEnv): LocalEngineStatus {
|
||||
if (!gbrainBin) return "no-cli";
|
||||
|
||||
// 2. Config file present?
|
||||
if (!existsSync(gbrainConfigPath())) return "missing-config";
|
||||
if (!existsSync(gbrainConfigPath(env))) return "missing-config";
|
||||
|
||||
// 3. Probe gbrain sources list.
|
||||
//
|
||||
@@ -240,14 +270,18 @@ function freshClassify(env?: NodeJS.ProcessEnv): LocalEngineStatus {
|
||||
try {
|
||||
execFileSync("gbrain", ["sources", "list", "--json"], {
|
||||
encoding: "utf-8",
|
||||
timeout: PROBE_TIMEOUT_MS,
|
||||
timeout: probeTimeoutMs(env),
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
env: buildGbrainEnv({ baseEnv: env ?? process.env }),
|
||||
shell: NEEDS_SHELL_ON_WINDOWS, // #1731: gbrain is a .cmd shim on Windows
|
||||
});
|
||||
return "ok";
|
||||
} catch (err) {
|
||||
const e = err as NodeJS.ErrnoException & { stderr?: Buffer | string };
|
||||
const e = err as NodeJS.ErrnoException & {
|
||||
stderr?: Buffer | string;
|
||||
killed?: boolean;
|
||||
signal?: NodeJS.Signals | null;
|
||||
};
|
||||
const stderr = (e.stderr ? e.stderr.toString() : "") || "";
|
||||
|
||||
// ENOENT can happen if gbrain disappeared between resolveGbrainBin and now.
|
||||
@@ -258,6 +292,13 @@ function freshClassify(env?: NodeJS.ProcessEnv): LocalEngineStatus {
|
||||
if (stderr.includes("Cannot connect to database")) return "broken-db";
|
||||
if (stderr.includes("config.json")) return "broken-config";
|
||||
|
||||
// Probe killed by the timeout with no recognized error: the engine is
|
||||
// most likely healthy but slow (cold pooler connections measured at
|
||||
// 6.9-10.7s in #1964). Don't tell the user their config is malformed.
|
||||
if (e.killed === true || e.signal === "SIGTERM" || e.code === "ETIMEDOUT") {
|
||||
return "timeout";
|
||||
}
|
||||
|
||||
// Defensive default per codex #8: unrecognized failures classify as
|
||||
// broken-config so the user sees the raw stderr surfaced upstream.
|
||||
return "broken-config";
|
||||
|
||||
@@ -427,6 +427,58 @@ export function applyRedactions(
|
||||
return { body, diff: diffLines.reverse().join("\n"), skipped };
|
||||
}
|
||||
|
||||
/**
|
||||
* Patterns whose regex captures only a MARKER, not the secret payload itself
|
||||
* (the PEM header line; the GCP JSON key prefix). Span replacement on these
|
||||
* would redact the header and forward the key body — so redactFindingSpans
|
||||
* drops the whole payload instead.
|
||||
*/
|
||||
const MARKER_ONLY_PATTERN_IDS = new Set(["pem.private_key", "gcp.service_account"]);
|
||||
|
||||
/**
|
||||
* Replace EVERY finding's span with `<REDACTED-{id}>`, regardless of tier or
|
||||
* autoRedactable. For machine egress surfaces (telemetry error_message,
|
||||
* #1947) where structure preservation doesn't matter and fail-closed beats
|
||||
* fidelity. Returns null — caller must drop the whole payload — when:
|
||||
* - any finding's span cannot be located, or
|
||||
* - any finding matched a marker-only pattern (PEM / GCP service-account
|
||||
* JSON): their regexes capture the header, not the key material, so a
|
||||
* span splice would leak the body that follows the marker.
|
||||
* Overlapping spans (e.g. a Bearer token that is also a JWT) are coalesced
|
||||
* before splicing so stale offsets never leave partial secret bytes behind.
|
||||
* (Contrast applyRedactions, which is the interactive, autoRedactable-only,
|
||||
* structure-preserving path.)
|
||||
*/
|
||||
export function redactFindingSpans(input: string, opts: ScanOptions = {}): string | null {
|
||||
const { findings } = scan(input, opts);
|
||||
if (findings.some((f) => MARKER_ONLY_PATTERN_IDS.has(f.id))) return null;
|
||||
const targets = findings.map((f) => ({ f, ...locateSpan(input, f) }));
|
||||
if (targets.some((t) => t.start < 0)) return null;
|
||||
|
||||
// Coalesce overlapping/touching ranges — splicing two intersecting spans
|
||||
// independently applies a stale end offset to already-modified text and
|
||||
// can leave trailing secret bytes in place.
|
||||
targets.sort((a, b) => a.start - b.start);
|
||||
const merged: Array<{ start: number; end: number; ids: string[] }> = [];
|
||||
for (const t of targets) {
|
||||
const last = merged[merged.length - 1];
|
||||
if (last && t.start <= last.end) {
|
||||
last.end = Math.max(last.end, t.end);
|
||||
if (!last.ids.includes(t.f.id)) last.ids.push(t.f.id);
|
||||
} else {
|
||||
merged.push({ start: t.start, end: t.end, ids: [t.f.id] });
|
||||
}
|
||||
}
|
||||
|
||||
// Right-to-left so earlier offsets remain valid after splicing.
|
||||
let body = input;
|
||||
for (let i = merged.length - 1; i >= 0; i--) {
|
||||
const m = merged[i];
|
||||
body = body.slice(0, m.start) + `<REDACTED-${m.ids.join("+")}>` + body.slice(m.end);
|
||||
}
|
||||
return body;
|
||||
}
|
||||
|
||||
function locateSpan(input: string, f: Finding): { start: number; end: number } {
|
||||
// Re-derive the offset from line/col on the original text.
|
||||
let offset = 0;
|
||||
|
||||
@@ -222,6 +222,48 @@ export const PATTERNS: RedactPattern[] = [
|
||||
description: "GitHub fine-grained PAT",
|
||||
regex: /\b(github_pat_[A-Za-z0-9_]{82})\b/,
|
||||
},
|
||||
{
|
||||
id: "gitlab.token",
|
||||
tier: "HIGH",
|
||||
category: "secret",
|
||||
description: "GitLab token (personal/pipeline-trigger/deploy)",
|
||||
// glpat- personal access, glptt- pipeline trigger, gldt- deploy token.
|
||||
// gstack drives glab first-class — these were a coverage gap (#1946).
|
||||
regex: /\b(gl(?:pat|ptt|dt)-[A-Za-z0-9_-]{20,})\b/,
|
||||
},
|
||||
{
|
||||
id: "huggingface.token",
|
||||
tier: "HIGH",
|
||||
category: "secret",
|
||||
description: "HuggingFace access token",
|
||||
regex: /\b(hf_[A-Za-z0-9]{30,})\b/,
|
||||
},
|
||||
{
|
||||
id: "npm.token",
|
||||
tier: "HIGH",
|
||||
category: "secret",
|
||||
description: "npm granular access token",
|
||||
regex: /\b(npm_[A-Za-z0-9]{36})\b/,
|
||||
},
|
||||
{
|
||||
id: "digitalocean.token",
|
||||
tier: "HIGH",
|
||||
category: "secret",
|
||||
description: "DigitalOcean personal access token",
|
||||
regex: /\b(dop_v1_[a-f0-9]{64})\b/,
|
||||
},
|
||||
{
|
||||
id: "gcp.service_account",
|
||||
tier: "HIGH",
|
||||
category: "secret",
|
||||
description: "GCP service-account JSON private key",
|
||||
// The JSON-escaped form ("private_key": "-----BEGIN PRIVATE KEY-----\n...)
|
||||
// dodges pem.private_key's literal-block match when minified to one line.
|
||||
// Proximity to "private_key_id" confirms the GCP service-account shape.
|
||||
regex: /("private_key"\s*:\s*"-----BEGIN (?:RSA |EC )?PRIVATE KEY-----)/,
|
||||
nearRegex: /"private_key_id"/,
|
||||
nearWindow: 300,
|
||||
},
|
||||
{
|
||||
id: "anthropic.key",
|
||||
tier: "HIGH",
|
||||
@@ -352,6 +394,22 @@ export const PATTERNS: RedactPattern[] = [
|
||||
!/^\$\{?[A-Za-z_]/.test(span) &&
|
||||
shannonEntropy(span) >= 3.0,
|
||||
},
|
||||
{
|
||||
id: "auth.bearer",
|
||||
tier: "MEDIUM",
|
||||
category: "secret",
|
||||
description: "Authorization Bearer token (high-entropy, header context)",
|
||||
// FP-prone shape (docs and examples are full of "Bearer <token>"), so:
|
||||
// MEDIUM tier, requires "authorization" nearby, and the same entropy
|
||||
// recipe as env.kv to kill Bearer YOUR_TOKEN_HERE placeholders.
|
||||
regex: /\bBearer[ \t]+([A-Za-z0-9._~+/=-]{20,})\b/,
|
||||
nearRegex: /authorization/i,
|
||||
nearWindow: 80,
|
||||
validate: (span) =>
|
||||
!isPlaceholderSpan(span) &&
|
||||
!/^\$\{?[A-Za-z_]/.test(span) &&
|
||||
shannonEntropy(span) >= 3.0,
|
||||
},
|
||||
|
||||
// ===== MEDIUM — PII (auto-redactable subset) =====
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user