mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-28 07:32:14 +02:00
v1.91.2.0 fix: consolidate gstack reliability wave (#2959)
* fix(memory-ingest): --scan-secrets scans the rendered page and fails closed --scan-secrets ran gitleaks on the raw transcript .jsonl, then imported a page rendered from it. gitleaks' assignment rules don't match across a JSON-escaped quote (KEY=\"v\" on disk), so a secret the rendered page shows as KEY="v" was imported unflagged. And the gate skipped a file only on scanner "gitleaks" with findings, so a scan that errored (non-zero exit, 16MB maxBuffer overflow on a file with many findings, unparseable report) or could not run (gitleaks missing, slow-probe cooldown) imported the file unscanned. Scan the rendered page body, the exact bytes writeStaged() writes, via a new secretScanText() helper, and skip the file whenever the scan did not complete. Skipped files stay out of the state file, so the next run retries them. Reword the helper warnings and setup-gbrain/memory.md, which described the fail-open as intended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(test): reconcile Bun failure markers and footer counts * fix(sync-gbrain): verify source-scoped reads without mutation * fix(test): recognize grounded TTHW target choices structurally * fix(aside): make the readiness probe work under zsh and report why it failed The probe built its deadline into `_T` and expanded it unquoted, so `$_T aside repl …` only worked in a shell that word-splits. zsh does not: it looked for a command literally named "gtimeout 30", the probe answered ASIDE_NOT_RUNNING with Aside installed and ready, and every browsing skill fell back to the bundled Chromium in silence. zsh is the macOS default and Aside is macOS-only, so on a stock Mac the probe could never report READY. The deadline becomes a function, `_gs_d`. It receives the command as "$@", already split, so sh, bash and zsh all behave the same, and the gtimeout → timeout → perl alarm chain is unchanged. A 4th arm runs the call unbounded when none of the three is present, which is what the empty `_T` did before. Not `eval`: it re-parses the string, so the parens and `;` of the perl arm become syntax and that arm dies in bash *and* zsh — on a stock Mac, the arm that actually runs. On failure the probe now prints the CLI's reason after ASIDE_NOT_RUNNING:, the shape gstack-render already uses: the first line that starts with a capital letter, i.e. the CLI's own sentence or Node's `Error:` line below its loader frame. "Not running" covers states with different fixes — no window open for the profile, a NODE_OPTIONS preload that kills the CLI — and a bare verdict sent all of them to "open the Aside app". The BROWSER SETUP prose quotes that reason before asking the user to open the app. The text pin asserted the broken invocation verbatim, so it now pins the function and asserts neither `$_T aside repl` nor an eval form comes back. A second test executes the rendered probe in sh, bash and zsh on each of the four deadline arms with stubbed binaries on a narrowed PATH, plus two failing CLIs: one that prints its own sentence, one that crashes like Node with the useful line below the frame. The deadline function costs zero bytes against the lines it replaces; the reason costs 53 per copy of the probe (44 where the reworded BROWSER SETUP line gives 9 back). That moves four guards by the measured amount: plan-devex-review's skeleton cap to 68,550 (measured 68,544), plan-ceo-review's skeleton cap to 80,150 (measured 80,111) and union ratio to 1.081 (measured 1.0803), and plan-eng-review's union ratio to 1.151 (measured 1.1504). Fixes #2842, #2941. * Clarify engineering review startup and decision flow * Fix Windows readiness fixture PATH and command shim * fix(test): recognize grounded TTHW target choices structurally * Clarify engineering review startup and decision flow * fix(test): restrict QA-only fixture tools to its no-Edit contract * v1.90.0.0 fix(sync-gbrain): guard readiness verdicts and refresh metadata * fix(browse): validate canonical upload targets * fix(gbrain): classify structured PGLite busy response * fix(browse): preserve native extension runtime APIs * Fix displayless browser handoff ownership * Accept unique installed autoplan methodology aliases * fix(skills): preserve positional literals during installation * fix(browse): checksum installer contents through stdin * fix(test): normalize Windows checksum fixture paths * test: emulate unavailable shasum in Windows checksum fixture * fix(investigate): preserve owned freeze lifecycle * fix(review): preserve N+1 retry and Red Team completion * fix: bound Aside readiness and preserve safe fallback * test: exercise setup and Chromium on native ARM * fix: preserve install ownership and ARM browser selection * Fix gbrain ingest scan boundaries and seed observation * Refresh managed ship hooks and supervise expanded paid census * Reject resumed gbrain pages excluded by current policy * Recover zombie agent locks safely and enable CI Python venv * Repair paid actor declarations and Aside pitch assertions * Bump consolidated wave to next free minor release * Clarify CEO review admin choices and option tradeoffs * Preserve CEO mode handoff anchors in clarified workflow * Make Windows portability fixtures use shell-native paths * Restore ARM Bun alias and clarify ship review gates * Refresh ship workflow golden snapshots * Fix Windows DX documentation controls without piped stdin * Decode Codex child pipes without Bun's encoded-stream stall * Bound DX pre-review audit before product questions * Clarify trusted review-start read in paid revalidation * Bump consolidated wave to next free minor release * Clarify CEO review admin choices and option tradeoffs * Preserve CEO mode handoff anchors in clarified workflow * Make Windows portability fixtures use shell-native paths * Restore ARM Bun alias and clarify ship review gates * Refresh ship workflow golden snapshots * Fix Windows DX documentation controls without piped stdin * Decode Codex child pipes without Bun's encoded-stream stall * Bound DX pre-review audit before product questions * Clarify trusted review-start read in paid revalidation * Reconcile new main planning flow and paid judge census * fix: reconcile rebased planning and source-bound validation * test: pin cookie workflow judge to scored Sonnet model * fix: keep terminal agent boot out of module imports * fix: preserve pending-question uncertainty in engineering review * fix: stabilize Windows reliability-wave fixtures * fix: clarify design consultation research workflow * fix: preserve independent design consultation inputs * fix: resolve design taste scope and browser research guidance * fix: make consultation opt-in preflight unambiguous * test: await native Edge owner readiness or terminal result --------- Co-authored-by: Bruce Krysiak <brucek@alum.mit.edu> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Antonio Vitalic <antoninte99@gmail.com>
This commit is contained in:
Garry Tan
co-authored by
Bruce Krysiak
Claude Opus 5.5
Antonio Vitalic
Bruce Krysiak
Claude Opus 5.5
Antonio Vitalic
parent
2a113ae7e6
commit
01593aa67c
@@ -15,13 +15,695 @@
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from "bun:test";
|
||||
import { mkdtempSync, writeFileSync, readFileSync, existsSync, rmSync, mkdirSync, statSync, chmodSync } from "fs";
|
||||
import { mkdtempSync, writeFileSync, readFileSync, existsSync, rmSync, mkdirSync, statSync, chmodSync, readdirSync, symlinkSync, utimesSync, copyFileSync } from "fs";
|
||||
import { tmpdir } from "os";
|
||||
import { join } from "path";
|
||||
import { basename, dirname, join } from "path";
|
||||
import { spawnSync } from "child_process";
|
||||
import { createHash, randomBytes } from "crypto";
|
||||
|
||||
const SCRIPT = join(import.meta.dir, "..", "bin", "gstack-memory-ingest.ts");
|
||||
|
||||
describe("requested secret scanning at the import boundary", () => {
|
||||
let home: string;
|
||||
let bin: string;
|
||||
let env: Record<string, string>;
|
||||
const realScanner = process.env.GSTACK_TEST_GITLEAKS;
|
||||
|
||||
beforeEach(() => {
|
||||
home = mkdtempSync(join(tmpdir(), "gstack-scan-"));
|
||||
bin = join(home, "bin");
|
||||
mkdirSync(bin);
|
||||
mkdirSync(join(home, "tmp"));
|
||||
env = {
|
||||
HOME: home, GSTACK_HOME: join(home, ".gstack"),
|
||||
PATH: `${bin}:/usr/bin:/bin`, TMPDIR: join(home, "tmp"),
|
||||
GIT_CONFIG_NOSYSTEM: "1", GIT_CONFIG_GLOBAL: join(home, ".gitconfig"),
|
||||
};
|
||||
writeFileSync(join(bin, "gbrain"), `#!${process.execPath}
|
||||
import { appendFileSync, cpSync, mkdirSync, readdirSync, readFileSync, writeFileSync, realpathSync, statSync, utimesSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
import { spawnSync } from 'child_process';
|
||||
const args = process.argv.slice(2);
|
||||
if (process.env.LIMIT_STAGE_WRITES === '1') {
|
||||
const reset = spawnSync('/usr/bin/prlimit', ['--pid', String(process.pid), '--fsize=unlimited:unlimited'], { timeout: 10000 });
|
||||
if (reset.status !== 0) process.exit(2);
|
||||
}
|
||||
if (args[0] === '--help') console.log(' import <dir>');
|
||||
else if (args[0] === 'doctor') console.log(JSON.stringify({ engine: 'pglite' }));
|
||||
else if (args[0] === 'import') {
|
||||
appendFileSync(join(process.env.HOME, 'imports'), 'import\\n');
|
||||
const files = [];
|
||||
function walk(dir) {
|
||||
for (const entry of readdirSync(dir, { withFileTypes: true })) {
|
||||
const path = join(dir, entry.name);
|
||||
if (entry.isDirectory()) walk(path);
|
||||
else if (entry.name.endsWith('.md')) files.push({ path: relative(args[1], path), body: readFileSync(path, 'utf8') });
|
||||
}
|
||||
}
|
||||
walk(args[1]);
|
||||
writeFileSync(join(process.env.HOME, 'imported.json'), JSON.stringify(files));
|
||||
if (process.env.SNAPSHOT_STAGE) {
|
||||
if (!process.env.SNAPSHOT_STAGE.startsWith(process.env.GSTACK_HOME + '/.staging-ingest-')) process.exit(2);
|
||||
cpSync(args[1], process.env.SNAPSHOT_STAGE, { recursive: true });
|
||||
}
|
||||
if (process.env.APPEND_DURING_IMPORT) {
|
||||
const path = realpathSync(process.env.APPEND_DURING_IMPORT);
|
||||
if (!path.startsWith(process.env.HOME + '/')) process.exit(2);
|
||||
const before = statSync(path);
|
||||
appendFileSync(path, process.env.APPEND_RECORD);
|
||||
if (process.env.RESTORE_MTIME === '1') utimesSync(path, before.atime, before.mtime);
|
||||
}
|
||||
if (process.env.REJECT_IMPORT === '1') process.exit(1);
|
||||
console.log(JSON.stringify({ status: 'ok', imported: process.env.UNDERCOUNT_IMPORT === '1' ? 0 : files.length, skipped: 0, errors: 0, total_files: files.length }));
|
||||
}
|
||||
`, { mode: 0o700 });
|
||||
});
|
||||
|
||||
afterEach(() => rmSync(home, { recursive: true, force: true }));
|
||||
|
||||
function source(text = "ordinary conversation"): string {
|
||||
const stamp = new Date().toISOString();
|
||||
const dir = join(home, ".codex", "sessions", ...stamp.slice(0, 10).split("-"));
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const path = join(dir, `rollout-${randomBytes(4).toString("hex")}.jsonl`);
|
||||
writeFileSync(path, [
|
||||
{ type: "session_meta", timestamp: stamp, payload: { id: basename(path), cwd: home } },
|
||||
{ type: "response_item", timestamp: stamp, payload: { type: "message", role: "user", content: [{ type: "input_text", text }] } },
|
||||
].map((entry) => JSON.stringify(entry)).join("\n") + "\n");
|
||||
return path;
|
||||
}
|
||||
|
||||
function scanner(mode: string): void {
|
||||
if (mode === "real") {
|
||||
copyFileSync(realScanner!, join(bin, "gitleaks"));
|
||||
chmodSync(join(bin, "gitleaks"), 0o700);
|
||||
return;
|
||||
}
|
||||
writeFileSync(join(bin, "gitleaks"), `#!${process.execPath}
|
||||
import { appendFileSync, readFileSync, writeFileSync, statSync, rmSync, realpathSync, utimesSync } from 'fs';
|
||||
import { dirname, join, relative, isAbsolute } from 'path';
|
||||
import { spawnSync } from 'child_process';
|
||||
const args = process.argv.slice(2);
|
||||
if (args[0] === 'version') { console.log('8.30.1'); process.exit(0); }
|
||||
const input = args[args.indexOf('--source') + 1];
|
||||
const report = args[args.indexOf('--report-path') + 1];
|
||||
const mode = ${JSON.stringify(mode)};
|
||||
const rel = relative(process.env.HOME, report);
|
||||
if (report !== '/dev/stdout' && (isAbsolute(rel) || rel.startsWith('..'))) process.exit(2);
|
||||
appendFileSync(join(process.env.HOME, 'scans'), JSON.stringify({ input, report, body: readFileSync(input, 'utf8'), inputMode: statSync(input).mode & 511, dirMode: statSync(dirname(report)).mode & 511, reportMode: statSync(report).mode & 511 }) + '\\n');
|
||||
if (mode === 'error') process.exit(2);
|
||||
if (mode === 'timeout') Bun.sleepSync(63000);
|
||||
if (process.env.APPEND_DURING_SCAN) {
|
||||
const path = realpathSync(process.env.APPEND_DURING_SCAN);
|
||||
if (!path.startsWith(process.env.HOME + '/')) process.exit(2);
|
||||
const before = statSync(path);
|
||||
appendFileSync(path, process.env.APPEND_RECORD);
|
||||
if (process.env.RESTORE_MTIME === '1') utimesSync(path, before.atime, before.mtime);
|
||||
}
|
||||
function emit(text) { if (report === '/dev/stdout') process.stdout.write(text); else writeFileSync(report, text); }
|
||||
if (mode === 'malformed') emit('{');
|
||||
else if (mode === 'empty') emit('');
|
||||
else if (mode === 'null') emit('null');
|
||||
else if (mode === 'invalid-finding') emit('[{}]');
|
||||
else if (mode === 'overflow') emit('[]' + ' '.repeat(16 * 1024 * 1024 - 1));
|
||||
else if (mode === 'ceiling') emit('[]' + ' '.repeat(16 * 1024 * 1024 - 2));
|
||||
else if (mode === 'missing-report') { if (report === '/dev/stdout') process.exit(2); rmSync(report); }
|
||||
else {
|
||||
const dirty = readFileSync(input, 'utf8').includes('UNSAFE="synthetic"');
|
||||
emit(dirty ? JSON.stringify([{ RuleID: 'fixture', Description: 'synthetic marker', StartLine: 1, Secret: 'synthetic' }]) : '[]');
|
||||
}
|
||||
if (process.env.LIMIT_STAGE_WRITES === '1') {
|
||||
const limit = spawnSync('/usr/bin/prlimit', ['--pid', String(process.ppid), '--fsize=2048:unlimited'], { timeout: 10000 });
|
||||
if (limit.status !== 0) process.exit(2);
|
||||
}
|
||||
`, { mode: 0o700 });
|
||||
}
|
||||
|
||||
function run(args: string[] = [], timeout = 30000) {
|
||||
const argv = [SCRIPT, "--include-unattributed", "--sources", "transcript", ...args];
|
||||
const limited = env.LIMIT_STAGE_WRITES === "1";
|
||||
const r = spawnSync(limited ? "/bin/bash" : process.execPath,
|
||||
limited ? ["-c", 'trap "" XFSZ; exec "$@"', "f3-limit", process.execPath, ...argv] : argv, {
|
||||
env, cwd: home, encoding: "utf8", timeout,
|
||||
});
|
||||
expect(r.error).toBeUndefined();
|
||||
return { stdout: r.stdout || "", stderr: r.stderr || "", status: r.status };
|
||||
}
|
||||
|
||||
function sessions(): Record<string, unknown> {
|
||||
const state = join(env.GSTACK_HOME, ".transcript-ingest-state.json");
|
||||
return existsSync(state) ? JSON.parse(readFileSync(state, "utf8")).sessions : {};
|
||||
}
|
||||
|
||||
function imported(): Array<{ path: string; body: string }> {
|
||||
const path = join(home, "imported.json");
|
||||
return existsSync(path) ? JSON.parse(readFileSync(path, "utf8")) : [];
|
||||
}
|
||||
|
||||
function resume(body: string, sourcePath?: string): string {
|
||||
const dir = join(env.GSTACK_HOME, ".staging-ingest-fixture");
|
||||
mkdirSync(join(dir, "nested"), { recursive: true });
|
||||
writeFileSync(join(dir, ".gstack-staging"), "fixture");
|
||||
let path = join(dir, "nested", "extra.md");
|
||||
if (sourcePath) {
|
||||
const meta = JSON.parse(readFileSync(sourcePath, "utf8").split("\n")[0]);
|
||||
path = join(dir, "transcripts", "codex", "_unattributed", `${meta.timestamp.slice(0, 10)}-${meta.payload.id.slice(0, 12)}.md`);
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
}
|
||||
writeFileSync(path, body);
|
||||
env.GSTACK_INGEST_RESUME_DIR = dir;
|
||||
return dir;
|
||||
}
|
||||
|
||||
function interruptedStage(): string {
|
||||
const dir = join(env.GSTACK_HOME, ".staging-ingest-interrupted");
|
||||
env.SNAPSHOT_STAGE = dir;
|
||||
env.REJECT_IMPORT = "1";
|
||||
expect(run(["--scan-secrets"]).status).toBe(1);
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
expect(sessions()).toEqual({});
|
||||
delete env.SNAPSHOT_STAGE;
|
||||
delete env.REJECT_IMPORT;
|
||||
env.GSTACK_INGEST_RESUME_DIR = dir;
|
||||
return dir;
|
||||
}
|
||||
|
||||
function appendRecord(): string {
|
||||
return JSON.stringify({ type: "response_item", timestamp: new Date().toISOString(), payload: {
|
||||
type: "message", role: "user", content: [{ type: "input_text", text: "late ordinary update" }],
|
||||
} }) + "\n";
|
||||
}
|
||||
|
||||
describe("snapshot-bound requested scans", () => {
|
||||
for (const rejected of [false, true]) {
|
||||
it(`accounts only saved pages on interrupted resume (rejected source: ${rejected})`, () => {
|
||||
scanner("clean");
|
||||
const bad = rejected ? source('UNSAFE="synthetic"') : undefined;
|
||||
const clean = source();
|
||||
const dir = interruptedStage();
|
||||
expect(imported()).toHaveLength(1);
|
||||
const result = run(["--scan-secrets"]);
|
||||
expect(result.status).toBe(0);
|
||||
expect(imported()).toHaveLength(1);
|
||||
expect(sessions()[clean]).toBeDefined();
|
||||
if (bad) expect(sessions()[bad]).toBeUndefined();
|
||||
expect(existsSync(dir)).toBe(false);
|
||||
delete env.GSTACK_INGEST_RESUME_DIR;
|
||||
rmSync(join(home, "imported.json"));
|
||||
const retry = run(["--scan-secrets"]);
|
||||
expect(retry.status).toBe(0);
|
||||
expect(imported()).toEqual([]);
|
||||
if (bad) expect(retry.stdout).toMatch(/skipped \(secret-scan\):\s+1/);
|
||||
});
|
||||
}
|
||||
|
||||
it("refuses genuine resumed under-accounting without discarding the saved stage", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const dir = interruptedStage();
|
||||
env.UNDERCOUNT_IMPORT = "1";
|
||||
const result = run(["--scan-secrets"]);
|
||||
expect(result.status).toBe(1);
|
||||
expect(result.stderr).toContain("accounted for 0 of 1 staged page(s)");
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
delete env.UNDERCOUNT_IMPORT;
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
|
||||
it("keeps an empty owned stage recoverable and never stamps an unstaged source", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const dir = resume("placeholder", path);
|
||||
for (const file of readdirSync(dir, { recursive: true })) {
|
||||
if (String(file).endsWith(".md")) rmSync(join(dir, String(file)));
|
||||
}
|
||||
expect(run(["--scan-secrets"]).status).toBe(1);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(imported()).toEqual([]);
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
});
|
||||
|
||||
it("retains a saved page without importing or stamping a removed source", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const dir = interruptedStage();
|
||||
rmSync(path);
|
||||
rmSync(join(home, "imported.json"));
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()).toEqual({});
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
});
|
||||
|
||||
it("retains the saved stage when current policy filters every source", () => {
|
||||
scanner("clean");
|
||||
source();
|
||||
const dir = interruptedStage();
|
||||
const policy = spawnSync(join(import.meta.dir, "..", "bin", "gstack-gbrain-repo-policy"), ["set", "_unattributed", "deny"], {
|
||||
env, cwd: home, encoding: "utf8", timeout: 10000,
|
||||
});
|
||||
expect(policy.status).toBe(0);
|
||||
rmSync(join(home, "imported.json"));
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()).toEqual({});
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
});
|
||||
|
||||
for (const tier of ["deny", "read-only"]) {
|
||||
for (const scanned of [false, true]) {
|
||||
it(`refuses a saved ${tier} page in a mixed-policy ${scanned ? "scanned" : "default"} resume`, () => {
|
||||
scanner("clean");
|
||||
const denied = source("denied source ordinary text");
|
||||
const allowed = source("allowed source ordinary text");
|
||||
const repo = join(home, "allowed-repo");
|
||||
mkdirSync(repo);
|
||||
expect(spawnSync("git", ["init", "-q", repo], { env, cwd: home, timeout: 10000 }).status).toBe(0);
|
||||
expect(spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://example.com/allowed.git"], { env, cwd: home, timeout: 10000 }).status).toBe(0);
|
||||
const records = readFileSync(allowed, "utf8").trim().split("\n").map((line) => JSON.parse(line));
|
||||
records[0].payload.cwd = repo;
|
||||
writeFileSync(allowed, records.map((entry) => JSON.stringify(entry)).join("\n") + "\n");
|
||||
const dir = interruptedStage();
|
||||
expect(imported()).toHaveLength(2);
|
||||
const policy = spawnSync(join(import.meta.dir, "..", "bin", "gstack-gbrain-repo-policy"), ["set", "_unattributed", tier], {
|
||||
env, cwd: home, encoding: "utf8", timeout: 10000,
|
||||
});
|
||||
expect(policy.status).toBe(0);
|
||||
rmSync(join(home, "imported.json"));
|
||||
const result = run(scanned ? ["--scan-secrets"] : []);
|
||||
expect(result.status).toBe(1);
|
||||
expect(result.stderr).toContain("[repo policy] staged page is not a current permitted source");
|
||||
expect(result.stderr).toContain("resumed import refused");
|
||||
expect(imported()).toEqual([]);
|
||||
expect(readFileSync(join(home, "imports"), "utf8")).toBe("import\n");
|
||||
expect(sessions()).toEqual({});
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
delete env.GSTACK_INGEST_RESUME_DIR;
|
||||
expect(run(scanned ? ["--scan-secrets"] : []).status).toBe(0);
|
||||
expect(imported().map((p) => p.body).join("\n")).toContain("allowed source ordinary text");
|
||||
expect(imported().map((p) => p.body).join("\n")).not.toContain("denied source ordinary text");
|
||||
expect(sessions()[allowed]).toBeDefined();
|
||||
expect(sessions()[denied]).toBeUndefined();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const scanned of [false, true]) {
|
||||
it(`rejects extra staged pages with a policy store in ${scanned ? "scanned" : "default"} resume`, () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const dir = interruptedStage();
|
||||
const policy = spawnSync(join(import.meta.dir, "..", "bin", "gstack-gbrain-repo-policy"), ["set", "_unattributed", "read-write"], {
|
||||
env, cwd: home, encoding: "utf8", timeout: 10000,
|
||||
});
|
||||
expect(policy.status).toBe(0);
|
||||
mkdirSync(join(dir, "nested"));
|
||||
writeFileSync(join(dir, "nested", "extra.md"), "unexpected ordinary content");
|
||||
rmSync(join(home, "imported.json"));
|
||||
const result = run(scanned ? ["--scan-secrets"] : []);
|
||||
expect(result.status).toBe(1);
|
||||
expect(result.stderr).toContain("[repo policy] staged page is not a current permitted source");
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
rmSync(join(dir, "nested", "extra.md"));
|
||||
expect(run(scanned ? ["--scan-secrets"] : []).status).toBe(0);
|
||||
expect(imported()).toHaveLength(1);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
}
|
||||
|
||||
for (const scanned of [false, true]) {
|
||||
it(`rejects a changed source with a policy store in ${scanned ? "scanned" : "default"} resume`, () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const dir = interruptedStage();
|
||||
const policy = spawnSync(join(import.meta.dir, "..", "bin", "gstack-gbrain-repo-policy"), ["set", "_unattributed", "read-write"], {
|
||||
env, cwd: home, encoding: "utf8", timeout: 10000,
|
||||
});
|
||||
expect(policy.status).toBe(0);
|
||||
writeFileSync(path, readFileSync(path, "utf8") + appendRecord());
|
||||
rmSync(join(home, "imported.json"));
|
||||
const result = run(scanned ? ["--scan-secrets"] : []);
|
||||
expect(result.status).toBe(1);
|
||||
expect(result.stderr).toContain("[repo policy] staged page is not a current permitted source");
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
});
|
||||
}
|
||||
|
||||
it("refuses an unreadable policy store before importing a saved stage", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const dir = interruptedStage();
|
||||
writeFileSync(join(env.GSTACK_HOME, "gbrain-repo-policy.json"), "not valid JSON");
|
||||
rmSync(join(home, "imported.json"));
|
||||
const result = run();
|
||||
expect(result.status).toBe(1);
|
||||
expect(result.stderr).toContain("repo policy store exists but");
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
});
|
||||
|
||||
for (const resumed of [false, true]) {
|
||||
it(`does not stamp an append during ${resumed ? "resumed" : "fresh"} import`, () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const time = new Date(Math.floor(Date.now() / 1000) * 1000);
|
||||
utimesSync(path, time, time);
|
||||
const dir = resumed ? interruptedStage() : undefined;
|
||||
env.APPEND_DURING_IMPORT = path;
|
||||
env.APPEND_RECORD = appendRecord();
|
||||
env.RESTORE_MTIME = "1";
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(statSync(path).mtimeMs).toBe(time.getTime());
|
||||
expect(imported().every((page) => !page.body.includes("late ordinary update"))).toBe(true);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
if (dir) expect(existsSync(dir)).toBe(true);
|
||||
delete env.APPEND_DURING_IMPORT;
|
||||
delete env.GSTACK_INGEST_RESUME_DIR;
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()[0].body).toContain("late ordinary update");
|
||||
expect(sessions()[path]).toMatchObject({
|
||||
mtime_ns: time.getTime() * 1e6,
|
||||
sha256: createHash("sha256").update(readFileSync(path)).digest("hex"),
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
for (const mode of ["no-write", "remote-http"]) {
|
||||
it(`does not stamp an append during the ${mode} scan`, () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
if (mode === "remote-http") writeFileSync(join(home, ".claude.json"), JSON.stringify({ mcpServers: { gbrain: { type: "http", url: "http://fixture.invalid/mcp" } } }));
|
||||
env.APPEND_DURING_SCAN = path;
|
||||
env.APPEND_RECORD = appendRecord();
|
||||
const args = mode === "no-write" ? ["--scan-secrets", "--no-write"] : ["--scan-secrets"];
|
||||
expect(run(args).status).toBe(0);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(imported()).toEqual([]);
|
||||
delete env.APPEND_DURING_SCAN;
|
||||
expect(run(args).status).toBe(0);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
}
|
||||
|
||||
for (const remote of [false, true]) {
|
||||
it(`never stamps a page that failed to stage (remote-http: ${remote})`, () => {
|
||||
scanner("clean");
|
||||
if (remote) writeFileSync(join(home, ".claude.json"), JSON.stringify({ mcpServers: { gbrain: { type: "http", url: "http://fixture.invalid/mcp" } } }));
|
||||
const dir = join(env.GSTACK_HOME, "projects", "demo", "ceo-plans");
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const path = join(dir, `${"p".repeat(245)}.md`);
|
||||
writeFileSync(path, "ordinary artifact content");
|
||||
const result = run(["--scan-secrets", "--sources", "ceo-plan"]);
|
||||
expect(result.stderr).toContain("[stage-error]");
|
||||
expect(result.stdout).toMatch(/failed:\s+1/);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(imported()).toEqual([]);
|
||||
});
|
||||
|
||||
(process.platform === "linux" ? it : it.skip)(`keeps OS-limited partial writes out of outgoing pages (remote-http: ${remote})`, () => {
|
||||
scanner("clean");
|
||||
if (remote) writeFileSync(join(home, ".claude.json"), JSON.stringify({ mcpServers: { gbrain: { type: "http", url: "http://fixture.invalid/mcp" } } }));
|
||||
const path = source("ordinary conversation ".repeat(300));
|
||||
env.LIMIT_STAGE_WRITES = "1";
|
||||
const result = run(["--scan-secrets"]);
|
||||
expect(result.stderr).toContain("EFBIG");
|
||||
expect(result.stdout).toMatch(/failed:\s+1/);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(readdirSync(join(home, "tmp"))).toEqual([]);
|
||||
expect(readdirSync(env.GSTACK_HOME).filter((p) => p.startsWith(".brain-ingest-write-"))).toEqual([]);
|
||||
const outgoing = join(env.GSTACK_HOME, "transcripts");
|
||||
if (existsSync(outgoing)) expect(readdirSync(outgoing, { recursive: true }).filter((p) => String(p).endsWith(".md"))).toEqual([]);
|
||||
delete env.LIMIT_STAGE_WRITES;
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
}
|
||||
|
||||
it("checks the hash on requested incremental scans even when mtime is unchanged", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const time = new Date(Math.floor(Date.now() / 1000) * 1000);
|
||||
utimesSync(path, time, time);
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
writeFileSync(path, readFileSync(path, "utf8") + appendRecord());
|
||||
utimesSync(path, time, time);
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()[0].body).toContain("late ordinary update");
|
||||
expect(readFileSync(join(home, "imports"), "utf8").trim().split("\n")).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("keeps the no-scan import stamping contract unchanged", () => {
|
||||
const path = source();
|
||||
env.APPEND_DURING_IMPORT = path;
|
||||
env.APPEND_RECORD = appendRecord();
|
||||
expect(run().status).toBe(0);
|
||||
expect(sessions()[path]).toMatchObject({ sha256: createHash("sha256").update(readFileSync(path)).digest("hex") });
|
||||
});
|
||||
});
|
||||
|
||||
it("imports clean pages once in one batch and deduplicates the next run", () => {
|
||||
scanner("clean");
|
||||
const paths = [source(), source("another ordinary conversation")];
|
||||
const r = run(["--scan-secrets"]);
|
||||
expect(r.status).toBe(0);
|
||||
expect(imported()).toHaveLength(2);
|
||||
expect(Object.keys(sessions()).sort()).toEqual(paths.sort());
|
||||
expect(run(["--scan-secrets"]).stdout).toMatch(/skipped \(dedup\):\s+2/);
|
||||
expect(readFileSync(join(home, "imports"), "utf8").trim().split("\n")).toHaveLength(1);
|
||||
const scans = readFileSync(join(home, "scans"), "utf8").trim().split("\n").map((line) => JSON.parse(line));
|
||||
expect(scans).toHaveLength(2);
|
||||
expect(scans.map((s) => s.body).sort()).toEqual(imported().map((p) => p.body).sort());
|
||||
for (const scan of scans) {
|
||||
expect(scan.dirMode).toBe(0o700);
|
||||
expect(scan.reportMode).toBe(0o600);
|
||||
expect(scan.inputMode).toBe(0o600);
|
||||
expect(existsSync(scan.report)).toBe(false);
|
||||
expect(existsSync(scan.input)).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("blocks a secret visible only after JSON decoding without recording it", () => {
|
||||
scanner("clean");
|
||||
const path = source('UNSAFE="synthetic"');
|
||||
expect(readFileSync(path, "utf8")).not.toContain('UNSAFE="synthetic"');
|
||||
const r = run(["--scan-secrets"]);
|
||||
expect(r.stdout).toMatch(/skipped \(secret-scan\):\s+1/);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
});
|
||||
|
||||
for (const mode of ["missing", "error", "malformed", "empty", "null", "invalid-finding", "overflow", "missing-report"]) {
|
||||
it(`refuses ${mode} scans and retries after repair`, () => {
|
||||
if (mode !== "missing") scanner(mode);
|
||||
const path = source();
|
||||
const r = run(["--scan-secrets"]);
|
||||
expect(r.stderr).toMatch(/secret-scan (missing|error)/);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(readdirSync(join(home, "tmp"))).toEqual([]);
|
||||
scanner("clean");
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()).toHaveLength(1);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
}
|
||||
|
||||
it("ends a detect invocation at its 60-second deadline and retries after repair", () => {
|
||||
scanner("timeout");
|
||||
const path = source();
|
||||
const r = run(["--scan-secrets"], 75000);
|
||||
expect(r.stderr).toContain("secret-scan error");
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(readdirSync(join(home, "tmp"))).toEqual([]);
|
||||
scanner("clean");
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
}, 80000);
|
||||
|
||||
it("does not stamp --no-write pages that could not pass the requested scan", () => {
|
||||
scanner("error");
|
||||
const path = source();
|
||||
run(["--scan-secrets", "--no-write"]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
scanner("clean");
|
||||
expect(run(["--scan-secrets", "--no-write"]).status).toBe(0);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
expect(imported()).toEqual([]);
|
||||
});
|
||||
|
||||
it("accepts a complete clean report exactly at the 16 MiB ceiling", () => {
|
||||
scanner("ceiling");
|
||||
const path = source();
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()).toHaveLength(1);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
|
||||
it("scans remote-http pages before persistent staging", () => {
|
||||
scanner("clean");
|
||||
writeFileSync(join(home, ".claude.json"), JSON.stringify({ mcpServers: { gbrain: { type: "http", url: "http://fixture.invalid/mcp" } } }));
|
||||
const bad = source('UNSAFE="synthetic"');
|
||||
const clean = source();
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[bad]).toBeUndefined();
|
||||
expect(sessions()[clean]).toBeDefined();
|
||||
const root = join(env.GSTACK_HOME, "transcripts");
|
||||
const pages = readdirSync(root, { recursive: true }).filter((path) => String(path).endsWith(".md"));
|
||||
expect(pages).toHaveLength(1);
|
||||
expect(readFileSync(join(root, String(pages[0])), "utf8")).toContain("ordinary conversation");
|
||||
});
|
||||
|
||||
it("leaves scanning opt-in", () => {
|
||||
const path = source('UNSAFE="synthetic"');
|
||||
expect(run().status).toBe(0);
|
||||
expect(imported()).toHaveLength(1);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
expect(existsSync(join(home, "scans"))).toBe(false);
|
||||
});
|
||||
|
||||
it("refuses unsafe extra resumed bytes and preserves the stage without success state", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const dir = resume('UNSAFE="synthetic"');
|
||||
expect(run(["--scan-secrets"]).status).toBe(1);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
});
|
||||
|
||||
it("rescans clean resumed bytes and retries a failed scanner without restaging", () => {
|
||||
const path = source();
|
||||
env.REJECT_IMPORT = "1";
|
||||
expect(run().status).toBe(1);
|
||||
const stagedBody = imported()[0].body;
|
||||
rmSync(join(home, "imported.json"));
|
||||
delete env.REJECT_IMPORT;
|
||||
const dir = resume(stagedBody, path);
|
||||
scanner("error");
|
||||
expect(run(["--scan-secrets"]).status).toBe(1);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
scanner("clean");
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported().map((p) => p.body)).toEqual([stagedBody]);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
|
||||
it("does not stamp changed source bytes that were not scanned or imported on resume", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
env.REJECT_IMPORT = "1";
|
||||
expect(run().status).toBe(1);
|
||||
const stagedBody = imported()[0].body;
|
||||
delete env.REJECT_IMPORT;
|
||||
resume(stagedBody, path);
|
||||
const records = readFileSync(path, "utf8").trim().split("\n").map((line) => JSON.parse(line));
|
||||
records[1].payload.content[0].text = 'UNSAFE="synthetic"';
|
||||
writeFileSync(path, records.map((entry) => JSON.stringify(entry)).join("\n") + "\n");
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported().map((p) => p.body)).toEqual([stagedBody]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(existsSync(env.GSTACK_INGEST_RESUME_DIR)).toBe(true);
|
||||
delete env.GSTACK_INGEST_RESUME_DIR;
|
||||
rmSync(join(home, "imported.json"));
|
||||
expect(run(["--scan-secrets"]).stdout).toMatch(/skipped \(secret-scan\):\s+1/);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
});
|
||||
|
||||
it("refuses resumed symlinks rather than scanning outside owned staging", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
const dir = resume("safe page", path);
|
||||
const target = join(home, "outside.md");
|
||||
writeFileSync(target, "outside content");
|
||||
symlinkSync(target, join(dir, "linked.md"));
|
||||
expect(run(["--scan-secrets"]).status).toBe(1);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(readFileSync(target, "utf8")).toBe("outside content");
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
});
|
||||
|
||||
it("retains the already-correct import exit-1 rejection and retry", () => {
|
||||
scanner("clean");
|
||||
const path = source();
|
||||
env.REJECT_IMPORT = "1";
|
||||
expect(run(["--scan-secrets"]).status).toBe(1);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
delete env.REJECT_IMPORT;
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
|
||||
(realScanner ? it : it.skip)("real gitleaks 8.30.1 imports clean pages and blocks recognized escaped secrets", () => {
|
||||
scanner("real");
|
||||
const version = spawnSync(realScanner!, ["version"], { env, timeout: 10000, encoding: "utf8" });
|
||||
expect(version.stdout.trim()).toBe("8.30.1");
|
||||
const clean = source();
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()).toHaveLength(1);
|
||||
expect(sessions()[clean]).toBeDefined();
|
||||
rmSync(join(home, "imported.json"));
|
||||
const secret = `LINKEDIN_CLIENT_SECRET="${randomBytes(8).toString("hex")}"`;
|
||||
const path = source(secret);
|
||||
const report = join(home, "raw-report.json");
|
||||
const raw = spawnSync(realScanner!, ["detect", "--no-git", "--source", path, "--report-format", "json", "--report-path", report, "--exit-code", "0"], { env, cwd: home, timeout: 10000, encoding: "utf8" });
|
||||
expect(raw.status).toBe(0);
|
||||
expect(JSON.parse(readFileSync(report, "utf8"))).toEqual([]);
|
||||
const rendered = join(home, "rendered.md");
|
||||
writeFileSync(rendered, secret);
|
||||
const positive = spawnSync(realScanner!, ["detect", "--no-git", "--source", rendered, "--report-format", "json", "--report-path", report, "--exit-code", "0"], { env, cwd: home, timeout: 10000, encoding: "utf8" });
|
||||
expect(positive.status).toBe(0);
|
||||
expect(JSON.parse(readFileSync(report, "utf8")).some((finding: { RuleID: string }) => finding.RuleID === "linkedin-client-secret")).toBe(true);
|
||||
const r = run(["--scan-secrets"]);
|
||||
expect(r.stdout).toMatch(/skipped \(secret-scan\):\s+1/);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(r.stderr).not.toContain(secret);
|
||||
});
|
||||
|
||||
(realScanner ? it : it.skip)("real scanner config errors remain retryable", () => {
|
||||
scanner("real");
|
||||
const path = source();
|
||||
const config = join(home, "broken.toml");
|
||||
writeFileSync(config, "[not-valid");
|
||||
env.GITLEAKS_CONFIG = config;
|
||||
expect(run(["--scan-secrets"]).stderr).toContain("secret-scan error");
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
delete env.GITLEAKS_CONFIG;
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported()).toHaveLength(1);
|
||||
expect(sessions()[path]).toBeDefined();
|
||||
});
|
||||
|
||||
(realScanner ? it : it.skip)("real scanner checks unsafe resumed pages, including extra files", () => {
|
||||
scanner("real");
|
||||
const path = source();
|
||||
const secret = `LINKEDIN_CLIENT_SECRET="${randomBytes(8).toString("hex")}"`;
|
||||
const dir = resume(secret);
|
||||
expect(run(["--scan-secrets"]).status).toBe(1);
|
||||
expect(imported()).toEqual([]);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
writeFileSync(join(dir, "nested", "extra.md"), "now clean");
|
||||
expect(run(["--scan-secrets"]).status).toBe(0);
|
||||
expect(imported().map((p) => p.body)).toEqual(["now clean"]);
|
||||
});
|
||||
});
|
||||
|
||||
// ── Helpers ────────────────────────────────────────────────────────────────
|
||||
|
||||
function makeTestHome(): string {
|
||||
@@ -445,6 +1127,44 @@ esac
|
||||
return { binDir, logFile, argsFile, stagingListFile };
|
||||
}
|
||||
|
||||
/**
|
||||
* Fake gitleaks for the --scan-secrets tests; returns its bin dir. `detect`
|
||||
* reports one finding when the CONTENT of the scanned file contains `marker`
|
||||
* (fixed-string), `[]` otherwise — so a test controls which bytes count as a
|
||||
* secret. `failDetect` exits non-zero like a crashed or misconfigured
|
||||
* gitleaks (scanner "error"); `failProbe` fails `gitleaks version`, which the
|
||||
* probe treats as absent (scanner "missing").
|
||||
*/
|
||||
function installFakeGitleaks(
|
||||
home: string,
|
||||
opts: { marker?: string; failDetect?: boolean; failProbe?: boolean },
|
||||
): string {
|
||||
const binDir = join(home, "fake-gitleaks-bin");
|
||||
mkdirSync(binDir, { recursive: true });
|
||||
const script = `#!/usr/bin/env bash
|
||||
if [ "\${1:-}" = "version" ]; then exit ${opts.failProbe ? 1 : 0}; fi
|
||||
${opts.failDetect ? 'echo "fake gitleaks: scan failed" >&2; exit 2' : ""}
|
||||
SRC=""
|
||||
REPORT=""
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--source) SRC="$2"; shift 2 ;;
|
||||
--report-path) REPORT="$2"; shift 2 ;;
|
||||
*) shift ;;
|
||||
esac
|
||||
done
|
||||
if grep -qF -- '${opts.marker ?? "no-marker-configured"}' "$SRC"; then
|
||||
echo '[{"RuleID":"fake-rule","Description":"fake finding","StartLine":1,"Match":"REDACTED","Secret":"AKIAFAKEFAKEFAKE12345"}]' > "$REPORT"
|
||||
else
|
||||
echo '[]' > "$REPORT"
|
||||
fi
|
||||
exit 0
|
||||
`;
|
||||
writeFileSync(join(binDir, "gitleaks"), script, "utf-8");
|
||||
chmodSync(join(binDir, "gitleaks"), 0o755);
|
||||
return binDir;
|
||||
}
|
||||
|
||||
describe("gstack-memory-ingest writer (gbrain v0.20+ batch `import` interface)", () => {
|
||||
it("probes the gbrain executable directly instead of shelling through command -v", () => {
|
||||
const source = readFileSync(SCRIPT, "utf-8");
|
||||
@@ -823,34 +1543,13 @@ esac
|
||||
mkdirSync(gstackHome, { recursive: true });
|
||||
const { binDir } = installFakeGbrain(home);
|
||||
|
||||
// Fake gitleaks: prints a "finding" for any file whose path contains
|
||||
// Fake gitleaks: reports a finding for any scanned page containing
|
||||
// "dirty", clean for everything else. The fake-gbrain shim doesn't
|
||||
// interfere — gitleaks is invoked from preparePages before staging.
|
||||
const fakeGitleaksDir = join(home, "fake-gitleaks-bin");
|
||||
mkdirSync(fakeGitleaksDir, { recursive: true });
|
||||
const fakeGitleaks = `#!/usr/bin/env bash
|
||||
# gitleaks detect --no-git --source <path> --report-format json --report-path /dev/stdout --exit-code 0
|
||||
# We just need to emit a JSON findings array on stdout. Find the --source arg.
|
||||
SRC=""
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--source) SRC="$2"; shift 2 ;;
|
||||
*) shift ;;
|
||||
esac
|
||||
done
|
||||
if echo "$SRC" | grep -q dirty; then
|
||||
echo '[{"RuleID":"fake-rule","Description":"fake finding","StartLine":1,"Match":"REDACTED","Secret":"AKIAFAKEFAKEFAKE12345"}]'
|
||||
else
|
||||
echo '[]'
|
||||
fi
|
||||
exit 0
|
||||
`;
|
||||
const gitleaksBin = join(fakeGitleaksDir, "gitleaks");
|
||||
writeFileSync(gitleaksBin, fakeGitleaks, "utf-8");
|
||||
chmodSync(gitleaksBin, 0o755);
|
||||
const fakeGitleaksDir = installFakeGitleaks(home, { marker: "dirty" });
|
||||
|
||||
// Two sessions: one "clean" (filename has no "dirty"), one "dirty"
|
||||
// (filename contains "dirty" so the fake gitleaks reports a finding).
|
||||
// Two sessions: one "clean", one "dirty" (its message text is "dirty",
|
||||
// so its rendered page draws a finding from the fake gitleaks).
|
||||
const sessionA =
|
||||
`{"type":"user","message":{"role":"user","content":"clean"},"timestamp":"2026-05-01T00:00:00Z","cwd":"/tmp/foo"}\n`;
|
||||
const sessionB =
|
||||
@@ -876,6 +1575,102 @@ exit 0
|
||||
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// The scan used to run on the raw .jsonl. gitleaks' assignment rules don't
|
||||
// match across a JSON-escaped quote, so a quoted secret in a transcript
|
||||
// (`KEY=\"v\"` on disk) scanned clean, then was imported as `KEY="v"`, the
|
||||
// form the rules do match (seen on real Codex sessions: pages flagged
|
||||
// linkedin-client-secret / generic-api-key whose .jsonl had scanned clean).
|
||||
// The fake flags only the unescaped form, as real gitleaks does.
|
||||
it("--scan-secrets scans the rendered page, not the raw .jsonl", () => {
|
||||
const home = makeTestHome();
|
||||
const gstackHome = join(home, ".gstack");
|
||||
mkdirSync(gstackHome, { recursive: true });
|
||||
const { binDir, stagingListFile } = installFakeGbrain(home);
|
||||
const marker = 'SYNTHETIC_TOKEN="';
|
||||
const fakeGitleaksDir = installFakeGitleaks(home, { marker });
|
||||
|
||||
const ts = "2026-05-03T00:00:00Z";
|
||||
const codexFile = writeCodexSession(
|
||||
home,
|
||||
"2026-05-03",
|
||||
[
|
||||
{ timestamp: ts, type: "session_meta", payload: { id: "sess-escaped", cwd: "/tmp/codex-app" } },
|
||||
{
|
||||
timestamp: ts,
|
||||
type: "response_item",
|
||||
payload: {
|
||||
type: "message",
|
||||
role: "user",
|
||||
content: [{ type: "input_text", text: `wire up auth:\n${marker}not-a-real-value"` }],
|
||||
},
|
||||
},
|
||||
].map((rec) => JSON.stringify(rec)).join("\n") + "\n",
|
||||
);
|
||||
// Premise: on disk the quote is escaped, so the raw file has no marker.
|
||||
expect(readFileSync(codexFile, "utf-8")).not.toContain(marker);
|
||||
writeClaudeCodeSession(
|
||||
home,
|
||||
"tmp-foo",
|
||||
"cleansess123",
|
||||
`{"type":"user","message":{"role":"user","content":"clean"},"timestamp":"2026-05-01T00:00:00Z","cwd":"/tmp/foo"}\n`,
|
||||
);
|
||||
|
||||
const r = runScript(["--bulk", "--include-unattributed", "--scan-secrets"], {
|
||||
HOME: home,
|
||||
GSTACK_HOME: gstackHome,
|
||||
PATH: `${fakeGitleaksDir}:${binDir}:${process.env.PATH || ""}`,
|
||||
});
|
||||
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(r.stdout).toMatch(/skipped \(secret-scan\):\s+1/);
|
||||
expect(r.stderr).toMatch(/\[secret-scan match\] .*\/\.codex\/sessions\/.+\.jsonl/);
|
||||
// Only the clean session reached gbrain import.
|
||||
const staged = readFileSync(stagingListFile, "utf-8");
|
||||
expect(staged).toMatch(/^\.\/transcripts\/claude-code\/.+\.md$/m);
|
||||
expect(staged).not.toContain("transcripts/codex/");
|
||||
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// "Could not scan" comes back as an empty findings list with scanner
|
||||
// "error" (gitleaks exited non-zero, overflowed the 16MB maxBuffer on a
|
||||
// file with many findings, or printed an unparseable report) or "missing"
|
||||
// (absent, unusable, or too slow to answer). The gate used to skip only on
|
||||
// scanner "gitleaks" with findings, so both let files in unscanned.
|
||||
for (const [label, opts, scanner] of [
|
||||
["gitleaks fails mid-scan", { failDetect: true }, "error"],
|
||||
["gitleaks is unusable", { failProbe: true }, "missing"],
|
||||
] as const) {
|
||||
it(`--scan-secrets fails closed when ${label} (scanner ${scanner})`, () => {
|
||||
const home = makeTestHome();
|
||||
const gstackHome = join(home, ".gstack");
|
||||
mkdirSync(gstackHome, { recursive: true });
|
||||
const { binDir, logFile } = installFakeGbrain(home);
|
||||
const fakeGitleaksDir = installFakeGitleaks(home, opts);
|
||||
writeClaudeCodeSession(
|
||||
home,
|
||||
"tmp-foo",
|
||||
"cleansess123",
|
||||
`{"type":"user","message":{"role":"user","content":"clean"},"timestamp":"2026-05-01T00:00:00Z","cwd":"/tmp/foo"}\n`,
|
||||
);
|
||||
|
||||
const r = runScript(["--bulk", "--include-unattributed", "--scan-secrets"], {
|
||||
HOME: home,
|
||||
GSTACK_HOME: gstackHome,
|
||||
PATH: `${fakeGitleaksDir}:${binDir}:${process.env.PATH || ""}`,
|
||||
});
|
||||
|
||||
expect(r.exitCode).toBe(0);
|
||||
expect(r.stdout).toMatch(/written:\s+0/);
|
||||
expect(r.stdout).toMatch(/skipped \(secret-scan\):\s+1/);
|
||||
expect(r.stderr).toContain(`[secret-scan ${scanner}]`);
|
||||
// Nothing was prepared, so gbrain import never ran.
|
||||
expect(existsSync(logFile)).toBe(false);
|
||||
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// #2105: current Codex rollout records are
|
||||
|
||||
Reference in New Issue
Block a user