mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-03 18:06:54 +02:00
v1.91.8.0 test: smaller suite, derived paid selection, retired never-green evals (#2994)
* test: delete test-infrastructure dead code (G) - exit-propagation drives the runner's real strict verdict (BunTestOutputClassifier + strictTestExitCode); delete the unused shardRunLooksTruncated predicate. - delete skill-coverage-matrix registry + its gate (nothing reads it; the floor already iterates skillCensus()). - delete touchfiles-facade export-parity tests (Bun fails missing imports at link time) and the duplicated E2E_TIERS tier-value test. - delete brain-cache-spec TRANSPORT_DEFAULT_POLICY, SKILL_RUN_RETENTION_DAYS and the now-unused BrainTrustPolicy type with their literal tests. AUTOPLAN_PREFLIGHT_BUDGET_BYTES stays: skill-preflight-budget enforces it against real resolver output. - delete audit-compliance's JSDoc-comment grep. * test: replace product tests that fake the product with real-boundary tests (F) - design: serve.test.ts drove an inline mirror server; now two tests run the real serve() on an ephemeral port (reload confinement, submit exit 0). - setup-gbrain: rollback + voyage tests execute the template-extracted init blocks (3 sites) instead of drifted local bash copies. - terminal-agent: internalHandler source greps replaced by a behavioral /internal/grant + /internal/revoke auth matrix (no/wrong/valid token). - /health: server-security-surface and the server-auth / security-audit-r2 / sidebar-tabs source greps fold into one liveness-only check on the real body; the L4 sidecar wiring gets a behavioral /pty-inject-scan test. - delete tautologies (browser-manager onDisconnect, memory-command #12), ios swiftui tap fixture self-check, memory-ingest put_page grep, detach source greps, sidebar-agent absence pins, dead-CSS pins + the dead CSS, security-audit-r2 Task 1 + the test-only meta-commands re-export, duplicate generated-SKILL.md checks. - make-pdf coverage-gaps cases move into their owner test files. * test: delete tests of dead eval code (A) - A1: the retired Eng lexical oracle (evaluateEngSeedCoverage, isEngSeedDecisionAUQ), the completion-handoff detector and the retained corpus had no paid caller since v1.87.6; delete their 26 replay files, ~2.6k helper LOC and fixtures, and the dead blocks in 8 mixed files (live hasNativePlanTerminal / batching assertions stay). - A2: dead viewport approvers in autoplan-artifact-permission and their 11 replay files + fixtures; recorder/launcher cases stay. - A3: never-wired oracles and seeders (autoplan-phase-order, eng-finding-fixture, ceo-paired-fixture, design-ui-scope, plan-skill-completion, pty-current-screen, required-reads, transcript-section-logger); plan-seed-submission now decodes through the production createPtyScreen; section manifests name their actual guard. - A4: zero-reference helper exports, plus execGit and invokeAndObserve found by the reachability pass. - 52 fixtures orphaned by the deletions; touchfile and selection-table entries for every deleted path. * test: clean up the paid eval lane (B1-B4, B6, B7) - B1: delete paid files that assert nothing or cannot pass meaningfully: skill-llm-eval-spec and skill-e2e-spec-execute (test.todo), gemini-e2e (+ gemini-session-runner; no gemini CLI in CI), ship-idempotency (red since v1.63), the two opus-4-7 *-sonnet overlay wrappers, conductor-prose (+ its source-evaluation replay), codex-e2e-plan-format; drop their keys, scripts and census rows. - B2: skill-llm-eval grades browse/sections/command-list.md with one union judge that also carries the baseline score pin; regression-vs-baseline deleted (paid run: pass, c4/c4/a4). - B3: memory-pipeline, ios-qa, ios-qa-swift-build and plan-tune-cathedral make no model calls; renamed out of the paid glob so they run on every PR. Swift builds need GSTACK_TEST_SWIFT=1; device stub deleted. - B4: codex-e2e*, outside-voice, aside and ios-device cannot run in the CI image; excluded from the weekly lane with a tracked re-entry condition. - B6: fold opus-47's negative routing controls into skill-routing-e2e journey-negatives (paid run: 3/3 unrouted) and delete the file. - B7: delete the never-green brain-privacy-gate eval; a free gstack-skill-start test now proves consent precedes artifacts egress. * test: retire the finding-count cluster and trim its helpers (C) - C0/C1: the five never-green evals (skill-e2e-autoplan-chain and skill-e2e-plan-{ceo,eng,design,devex}-finding-count) failed on harness and budget, never on skill behavior; delete them, their touchfile/tier ids, AUTOPLAN_CHAIN_BUDGET and the dedicated eighth periodic slice (--slices 7). - C2: delete the helper groups whose only paid consumers were those files (11 modules), trim claude-pty-runner and eng-seeded-coverage to the paid closure, and delete the free replay tests whose assertions exercised only that dead code (89 files, 135 orphaned fixtures). Blocks that used dead code only as input for a live subject keep their assertions: the multiSelect default moved to plan-review-decisions, runner PTY tests use inline caller policies, and the timer-safe budget checks moved to eng-finding-retry-budget. - The eight production-touching files stay except ceo-current-decision-record (its template read only feeds the retired counter). - CARVE_GUARDS.autoplan is behavioral 'none'; TODOS records the lost chain and per-finding cadence coverage with their re-entry tests. * test: fold per-incident replay series into their detector owners (D) Twelve detector families move into one owner test each: 73 incident files become describe blocks in ceo-section-loading-fixture (stale-fill race), model-overlays, coverage-audit-evidence, autoplan-phase-observer, native-auto-decide, outside-voice-evidence, eng-first-review, plan-count-completion, plan-count-file-permission, ceo-mode-option, plan-scope-selection and plan-count-prerequisite. Each block keeps its original code and fixture, so every case still runs; only tests asserting the incident file's own touchfile registration are dropped (41). Touchfile lists that named an incident now name its owner. * test: start the plan-count history PTY on its readiness marker (H) The fake CLI prints a startup marker and the runner waits for it instead of the fixed 8 s startup sleep (8.6 s -> 0.9 s locally). eng-semantic-terminal's sleeping registration cases went with C; plan-count-timeout keeps the fixed wait because it asserts deadline behavior. * test: derive paid touchfiles from each eval's static closure (E) touchfiles.test.ts now checks, per key, that the paid file's static test/helpers and test/fixtures closure (plus fixture paths it names in string literals) is covered, and names the file, path, chain and key to fix when it is not. Free *.test.ts files are no longer touchfiles, so editing a free replay test stops selecting paid evals: 950 entries removed, 653 real closure paths added. The hand-copied inventories go: periodic-fixture-selection, fake-impeccable-touchfiles and 45 per-file selection examples. Selection for the sample edits (plan-eng-review template, claude-pty-runner, plan-count-fixture, gstack-config) loses no case under either profile. CONTRIBUTING documents the rule and its lower bound. * test: skip hollow tier shards and census judges in the paid planner (B5) A paid file is now skipped for a tier lane only when every E2E id it registers is known statically and none has that tier; ids come from the touchfile registrations and literal testName/*IfSelected arguments, so a comment or skill path that quotes another id cannot unschedule it, and computed names keep today's scheduling. --list and the manifest show each skip as "skipped: no E2E_TIERS id has tier <tier>". The weekly gate census drops the LLM judges (--skip-judges); they still run in the periodic census and PR gate lanes. Gate lane 52 -> 42 files, census 41; periodic 77 -> 69. * test: run seven paid evals on the current default capture model (B8) skill-e2e-{auq-matrix,plan-format,qa-bugs,retro,workflow} pinned claude-opus-4-7 and skill-e2e-office-hours plus -brain-writeback pinned claude-sonnet-4-6; none tests a historical model, so they now capture with resolveEvalModel('capture'), and the free harness tests that execute these registrations receive the same resolver. The paid re-pin run passed all of them. skill-e2e-{design,office-hours-phase4,plan-prosons,plan} keep claude-opus-4-7: six of their cases failed on the default model (three timeouts, a missing report file, a format miss and a posture score of 3), so per the plan's fallback they keep their pins with a TODOS entry. The pre-spend estimate and drop threshold are in docs/test-audit-2026-09.md. * test: guard the reduced suite against new test-of-test files - test/test-of-test-ratchet.test.ts records the 228 free tests that import only test/ code and fails on a new one, naming the owner test to extend instead; a stale baseline entry fails with the remove instruction. - test/helpers/resolve-repo-path.ts is the one specifier/literal resolver for the ratchet and the touchfile closure invariant, with its own unit tests. - CONTRIBUTING "Test tiers" describes the paid-failure workflow (fix, then one row in the detector's owner test) and the ratchet; TEST_PORTFOLIO gains the detector -> owner-test table and no longer claims an Autoplan chain eval. - TODOS: automatic exclusion policy for chronically red periodic files (P3), the deferred native-completion table collapse, the unused CEO payment seeder; the PTY readiness item is narrowed to the paid runner. - docs/test-audit-2026-09.md collects the triage, security mapping, inventories, selection proof, behavior-commit decisions and retained false positives. * v1.91.8.0 test: smaller suite, derived paid selection, retired never-green evals Release metadata for the test-reduction branch: VERSION 1.91.8.0 (1.91.7.0 is claimed by #2983), CHANGELOG with the measured before/after table and a contributor section, durations re-recorded on Ubicloud standard-16 (857 files, 0 failures), the agents digest, CONTRIBUTING's after-measurement row, the B8 fallback TODOS entry, and the after metrics, kept-vs-plan notes, B8 run and census estimate in docs/test-audit-2026-09.md. * fix(ubicloud): skip retrieval globs that match nothing instead of reporting a failed pull * test: count issue-numbered Design findings in the UI-scope gate eval
This commit is contained in:
Garry Tan authored and GitHub committed
1 parent
dcaea52800
commit
943105f109
668 files changed
+11998
-102381
No files matched your search
@@ -1,21 +1,20 @@
|
||||
/**
|
||||
* Tests the voyage-code-3 default contract in setup-gbrain's PGLite init
|
||||
* sequences. The contract lives in the skill TEMPLATE (.tmpl), not in a TS
|
||||
* helper — the skill follows AI-readable instructions.
|
||||
* setup-gbrain's local PGLite init sequences, executed from the TEMPLATE.
|
||||
*
|
||||
* Contract (asserted here):
|
||||
* 1. When VOYAGE_API_KEY is set, gstack's PGLite init passes
|
||||
* --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024
|
||||
* 2. When VOYAGE_API_KEY is unset, those flags are omitted (gbrain's
|
||||
* auto-selected provider chain takes over)
|
||||
* The contract lives in skill template prose the model executes, not in a TS
|
||||
* helper, so this file extracts each fenced bash block that runs
|
||||
* `gbrain init --pglite --json "$@"` from the .tmpl files and runs it against
|
||||
* a fake `gbrain` in a sandboxed HOME. A template edit changes what runs here;
|
||||
* there is no hand-copied shell to drift.
|
||||
*
|
||||
* Why a separate file from gbrain-init-rollback.test.ts: that file owns the
|
||||
* .bak-rollback contract (Step 1.5 / 4.5 plan D7). This file owns the
|
||||
* embedding-model selection contract. Both extract bash from the skill
|
||||
* template and execute it against a fake gbrain.
|
||||
*
|
||||
* The fake gbrain records argv to a sentinel file so the test can assert
|
||||
* exact flags. No Voyage API calls are made.
|
||||
* Contracts:
|
||||
* 1. voyage-code-3 default: with VOYAGE_API_KEY set, every init site passes
|
||||
* --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024 as
|
||||
* separate argv words (also under zsh, #1798); unset or empty omits them.
|
||||
* 2. .bak rollback (plan D7): the two rollback-wrapped sites move an existing
|
||||
* ~/.gbrain/config.json aside, restore it byte-for-byte when init fails
|
||||
* (leaving a partial PGLite dir alone), and keep the backup for audit when
|
||||
* init succeeds.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from "bun:test";
|
||||
@@ -24,6 +23,7 @@ import {
|
||||
mkdirSync,
|
||||
writeFileSync,
|
||||
readFileSync,
|
||||
readdirSync,
|
||||
existsSync,
|
||||
rmSync,
|
||||
chmodSync,
|
||||
@@ -32,230 +32,188 @@ import { tmpdir } from "os";
|
||||
import { join } from "path";
|
||||
import { spawnSync } from "child_process";
|
||||
|
||||
interface FakeEnv {
|
||||
tmp: string;
|
||||
const SETUP_GBRAIN = join(import.meta.dir, "..", "setup-gbrain");
|
||||
const TEMPLATES = {
|
||||
skeleton: join(SETUP_GBRAIN, "SKILL.md.tmpl"),
|
||||
brainInit: join(SETUP_GBRAIN, "sections", "brain-init.md.tmpl"),
|
||||
remediation: join(SETUP_GBRAIN, "sections", "engine-remediation.md.tmpl"),
|
||||
};
|
||||
const INIT_CALL = 'gbrain init --pglite --json "$@"';
|
||||
|
||||
function initBlocks(tmplPath: string): string[] {
|
||||
const src = readFileSync(tmplPath, "utf-8");
|
||||
return [...src.matchAll(/```bash\n([\s\S]*?)```/g)]
|
||||
.map((m) => m[1])
|
||||
.filter((block) => block.includes(INIT_CALL));
|
||||
}
|
||||
|
||||
const PATH3_BLOCK = initBlocks(TEMPLATES.brainInit).find((b) => !b.includes("gstack-bak"));
|
||||
const PATH4_BLOCK = initBlocks(TEMPLATES.brainInit).find((b) => b.includes("gstack-bak"));
|
||||
const REMEDIATION_BLOCK = initBlocks(TEMPLATES.remediation).find((b) => b.includes("gstack-bak"));
|
||||
const ROLLBACK_SITES = { "Path 4 local code search": PATH4_BLOCK, "engine remediation": REMEDIATION_BLOCK };
|
||||
const ALL_SITES = { "Path 3 PGLite": PATH3_BLOCK, ...ROLLBACK_SITES };
|
||||
|
||||
interface Sandbox {
|
||||
home: string;
|
||||
bindir: string;
|
||||
configPath: string;
|
||||
argvLog: string;
|
||||
cleanup: () => void;
|
||||
}
|
||||
|
||||
function makeFakeEnv(): FakeEnv {
|
||||
const tmp = mkdtempSync(join(tmpdir(), "gbrain-voyage-init-"));
|
||||
function makeSandbox(opts: { initFails?: boolean; seedConfig?: boolean } = {}): Sandbox {
|
||||
const tmp = mkdtempSync(join(tmpdir(), "gbrain-pglite-init-"));
|
||||
const home = join(tmp, "home");
|
||||
const gbrainDir = join(home, ".gbrain");
|
||||
const bindir = join(tmp, "bin");
|
||||
const configPath = join(gbrainDir, "config.json");
|
||||
const argvLog = join(tmp, "gbrain-argv.log");
|
||||
mkdirSync(join(home, ".gbrain"), { recursive: true });
|
||||
mkdirSync(gbrainDir, { recursive: true });
|
||||
mkdirSync(bindir, { recursive: true });
|
||||
|
||||
// Fake gbrain logs every argv invocation to argvLog (one line per call),
|
||||
// succeeds on init (writes a sentinel pglite config), and returns canned
|
||||
// output for --version. Nothing else is needed for the shape test.
|
||||
const fake = `#!/bin/sh
|
||||
echo "$@" >> "${argvLog}"
|
||||
echo "$#" >> "${argvLog}.argc"
|
||||
case "$1" in
|
||||
--version)
|
||||
echo "gbrain 0.37.1.0"
|
||||
exit 0
|
||||
;;
|
||||
init)
|
||||
cat > "${home}/.gbrain/config.json" <<JSON
|
||||
{"engine":"pglite","database_path":"${home}/.gbrain/brain.pglite"}
|
||||
JSON
|
||||
echo '{"status":"success","engine":"pglite","pages":0}'
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
exit 0
|
||||
`;
|
||||
writeFileSync(join(bindir, "gbrain"), fake);
|
||||
const installer = join(home, ".claude", "skills", "gstack", "bin", "gstack-gbrain-install");
|
||||
mkdirSync(join(installer, ".."), { recursive: true });
|
||||
writeFileSync(installer, "#!/bin/sh\nexit 0\n");
|
||||
chmodSync(installer, 0o755);
|
||||
if (opts.seedConfig) {
|
||||
writeFileSync(configPath, JSON.stringify({ engine: "postgres", database_url: "postgresql://stale@localhost/gbrain" }));
|
||||
}
|
||||
const onInit = opts.initFails
|
||||
? `mkdir -p "${gbrainDir}/pglite" && : > "${gbrainDir}/pglite/partial-write.tmp"; echo "Error: disk full" >&2; exit 1`
|
||||
: `printf '{"engine":"pglite"}' > "${configPath}"; echo '{"status":"success"}'; exit 0`;
|
||||
writeFileSync(
|
||||
join(bindir, "gbrain"),
|
||||
`#!/bin/sh\necho "$@" >> "${argvLog}"\necho "$#" >> "${argvLog}.argc"\nif [ "$1" = "init" ]; then ${onInit}; fi\nexit 0\n`,
|
||||
);
|
||||
chmodSync(join(bindir, "gbrain"), 0o755);
|
||||
|
||||
return {
|
||||
tmp,
|
||||
home,
|
||||
bindir,
|
||||
argvLog,
|
||||
cleanup: () => rmSync(tmp, { recursive: true, force: true }),
|
||||
};
|
||||
return { home, bindir, configPath, argvLog, cleanup: () => rmSync(tmp, { recursive: true, force: true }) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Verbatim reimplementation of the skill template's voyage-code-3
|
||||
* conditional. The template (setup-gbrain/sections/brain-init.md.tmpl Path 3, Step 1.5
|
||||
* inside the rollback wrapper, Step 4.5 Path 4 Yes branch) instructs the
|
||||
* model to execute this bash; we execute the same bash here and assert the
|
||||
* argv passed to gbrain matches the contract.
|
||||
*
|
||||
* If the template changes the flag set or the env-var name, this test
|
||||
* should fail until the shell here is updated too — by design.
|
||||
*/
|
||||
function runInitWithVoyageGate(
|
||||
env: FakeEnv,
|
||||
voyageKey: string | undefined,
|
||||
shell: "bash" | "zsh" = "bash",
|
||||
): string[] {
|
||||
// The template's #1798 shape: flags ride the positional params, because an
|
||||
// unquoted $VAR does NOT word-split under zsh — the whole flag string
|
||||
// arrived as ONE argv word and gbrain silently fell back to its default
|
||||
// embedding model.
|
||||
const script = `
|
||||
set -u
|
||||
set --
|
||||
if [ -n "\${VOYAGE_API_KEY:-}" ]; then
|
||||
set -- --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024
|
||||
fi
|
||||
gbrain init --pglite --json "$@"
|
||||
`;
|
||||
const baseEnv: Record<string, string> = {
|
||||
...process.env,
|
||||
HOME: env.home,
|
||||
PATH: `${env.bindir}:/usr/bin:/bin`,
|
||||
};
|
||||
if (voyageKey === undefined) {
|
||||
delete baseEnv.VOYAGE_API_KEY;
|
||||
} else {
|
||||
baseEnv.VOYAGE_API_KEY = voyageKey;
|
||||
}
|
||||
const result = spawnSync(shell, ["-c", script], {
|
||||
encoding: "utf-8",
|
||||
env: baseEnv,
|
||||
timeout: 30_000,
|
||||
});
|
||||
if (result.status !== 0) {
|
||||
throw new Error(`init script exited ${result.status}: ${result.stderr}`);
|
||||
}
|
||||
return readFileSync(env.argvLog, "utf-8").trim().split("\n");
|
||||
function runBlock(sb: Sandbox, block: string, opts: { voyageKey?: string; shell?: "bash" | "zsh" } = {}) {
|
||||
const env: Record<string, string> = { ...process.env, HOME: sb.home, PATH: `${sb.bindir}:/usr/bin:/bin` };
|
||||
delete env.VOYAGE_API_KEY;
|
||||
if (opts.voyageKey !== undefined) env.VOYAGE_API_KEY = opts.voyageKey;
|
||||
const r = spawnSync(opts.shell ?? "bash", ["-c", block], { encoding: "utf-8", env, timeout: 30_000 });
|
||||
const argv = existsSync(sb.argvLog) ? readFileSync(sb.argvLog, "utf-8").trim().split("\n") : [];
|
||||
const argc = existsSync(`${sb.argvLog}.argc`)
|
||||
? readFileSync(`${sb.argvLog}.argc`, "utf-8").trim().split("\n").map(Number)
|
||||
: [];
|
||||
return { status: r.status, stderr: r.stderr ?? "", argv, argc };
|
||||
}
|
||||
|
||||
function lastArgc(env: FakeEnv): number {
|
||||
const lines = readFileSync(`${env.argvLog}.argc`, "utf-8").trim().split("\n");
|
||||
return parseInt(lines[lines.length - 1], 10);
|
||||
function backups(sb: Sandbox): string[] {
|
||||
return readdirSync(join(sb.home, ".gbrain")).filter((f) => f.includes(".gstack-bak-"));
|
||||
}
|
||||
|
||||
const HAVE_ZSH = spawnSync("zsh", ["-c", "true"], { timeout: 30_000 }).status === 0;
|
||||
|
||||
describe("voyage-code-3 default for gstack-driven PGLite init", () => {
|
||||
it("passes voyage-code-3 flags when VOYAGE_API_KEY is set", () => {
|
||||
const env = makeFakeEnv();
|
||||
describe("template extraction", () => {
|
||||
it("finds all three PGLite init blocks (a template restructure must update this file)", () => {
|
||||
expect(PATH3_BLOCK).toBeDefined();
|
||||
expect(PATH4_BLOCK).toBeDefined();
|
||||
expect(REMEDIATION_BLOCK).toBeDefined();
|
||||
expect(initBlocks(TEMPLATES.skeleton)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("voyage-code-3 default at every PGLite init site", () => {
|
||||
for (const [site, block] of Object.entries(ALL_SITES)) {
|
||||
it(`${site}: passes voyage-code-3 flags when VOYAGE_API_KEY is set`, () => {
|
||||
const sb = makeSandbox();
|
||||
try {
|
||||
const r = runBlock(sb, block!, { voyageKey: "vk_test_set" });
|
||||
expect(r.argv).toEqual(["init --pglite --json --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024"]);
|
||||
expect(r.argc).toEqual([7]);
|
||||
} finally {
|
||||
sb.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it(`${site}: omits voyage flags when VOYAGE_API_KEY is unset or empty`, () => {
|
||||
for (const voyageKey of [undefined, ""]) {
|
||||
const sb = makeSandbox();
|
||||
try {
|
||||
const r = runBlock(sb, block!, { voyageKey });
|
||||
expect(r.argv).toEqual(["init --pglite --json"]);
|
||||
} finally {
|
||||
sb.cleanup();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it(`${site}: zsh passes the flags as SEPARATE argv words (#1798)`, () => {
|
||||
if (!HAVE_ZSH) return;
|
||||
const sb = makeSandbox();
|
||||
try {
|
||||
expect(runBlock(sb, block!, { voyageKey: "vk_test_set", shell: "zsh" }).argc).toEqual([7]);
|
||||
} finally {
|
||||
sb.cleanup();
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe(".bak rollback contract (plan D7)", () => {
|
||||
for (const [site, block] of Object.entries(ROLLBACK_SITES)) {
|
||||
it(`${site}: failed init restores the original config and leaves partial PGLite state alone`, () => {
|
||||
const sb = makeSandbox({ initFails: true, seedConfig: true });
|
||||
try {
|
||||
const original = readFileSync(sb.configPath, "utf-8");
|
||||
const r = runBlock(sb, block!);
|
||||
expect(r.stderr).toContain("restored");
|
||||
expect(readFileSync(sb.configPath, "utf-8")).toBe(original);
|
||||
expect(backups(sb)).toEqual([]);
|
||||
expect(existsSync(join(sb.home, ".gbrain", "pglite", "partial-write.tmp"))).toBe(true);
|
||||
} finally {
|
||||
sb.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it(`${site}: successful init installs the new config and keeps the backup for audit`, () => {
|
||||
const sb = makeSandbox({ seedConfig: true });
|
||||
try {
|
||||
const r = runBlock(sb, block!);
|
||||
expect(r.status).toBe(0);
|
||||
expect(JSON.parse(readFileSync(sb.configPath, "utf-8")).engine).toBe("pglite");
|
||||
expect(backups(sb).length).toBe(1);
|
||||
} finally {
|
||||
sb.cleanup();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
it("Path 4 continues setup after a failed init; engine remediation stops with exit 1", () => {
|
||||
const path4 = makeSandbox({ initFails: true, seedConfig: true });
|
||||
const remediation = makeSandbox({ initFails: true, seedConfig: true });
|
||||
try {
|
||||
const calls = runInitWithVoyageGate(env, "vk_test_set");
|
||||
expect(calls.length).toBe(1);
|
||||
const argv = calls[0];
|
||||
expect(argv).toContain("init --pglite --json");
|
||||
expect(argv).toContain("--embedding-model voyage:voyage-code-3");
|
||||
expect(argv).toContain("--embedding-dimensions 1024");
|
||||
const p4 = runBlock(path4, PATH4_BLOCK!);
|
||||
expect(p4.status).toBe(0);
|
||||
expect(p4.stderr).toContain("Continuing setup without local code search");
|
||||
expect(runBlock(remediation, REMEDIATION_BLOCK!).status).toBe(1);
|
||||
} finally {
|
||||
env.cleanup();
|
||||
path4.cleanup();
|
||||
remediation.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it("omits voyage flags when VOYAGE_API_KEY is unset", () => {
|
||||
const env = makeFakeEnv();
|
||||
it("Path 4 with no existing config: failed init creates no backup and no config", () => {
|
||||
const sb = makeSandbox({ initFails: true });
|
||||
try {
|
||||
const calls = runInitWithVoyageGate(env, undefined);
|
||||
expect(calls.length).toBe(1);
|
||||
const argv = calls[0];
|
||||
expect(argv).toContain("init --pglite --json");
|
||||
expect(argv).not.toContain("voyage");
|
||||
expect(argv).not.toContain("--embedding-model");
|
||||
expect(argv).not.toContain("--embedding-dimensions");
|
||||
runBlock(sb, PATH4_BLOCK!);
|
||||
expect(backups(sb)).toEqual([]);
|
||||
expect(existsSync(sb.configPath)).toBe(false);
|
||||
} finally {
|
||||
env.cleanup();
|
||||
sb.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
it("zsh: flags arrive as SEPARATE argv words (#1798 — the shell that broke)", () => {
|
||||
if (!HAVE_ZSH) return; // zsh ships on macOS; skip quietly elsewhere
|
||||
const env = makeFakeEnv();
|
||||
try {
|
||||
const calls = runInitWithVoyageGate(env, "vk_test_set", "zsh");
|
||||
expect(calls.length).toBe(1);
|
||||
expect(calls[0]).toContain("--embedding-model voyage:voyage-code-3");
|
||||
// init --pglite --json + 4 flag words = 7 argv entries. The pre-#1798
|
||||
// unquoted-var shape produced 4 under zsh (the whole flag string as one
|
||||
// word), and gbrain silently fell back to its default embedding model.
|
||||
expect(lastArgc(env)).toBe(7);
|
||||
} finally {
|
||||
env.cleanup();
|
||||
}
|
||||
});
|
||||
describe("template alignment", () => {
|
||||
const tmpl = Object.values(TEMPLATES).map((p) => readFileSync(p, "utf-8")).join("\n");
|
||||
|
||||
it("demonstrates the #1798 collision: an unquoted flags var is ONE word under zsh", () => {
|
||||
if (!HAVE_ZSH) return;
|
||||
const env = makeFakeEnv();
|
||||
try {
|
||||
const brokenShape = `
|
||||
set -u
|
||||
GBRAIN_EMBED_FLAGS="--embedding-model voyage:voyage-code-3 --embedding-dimensions 1024"
|
||||
gbrain init --pglite --json $GBRAIN_EMBED_FLAGS
|
||||
`;
|
||||
const result = spawnSync("zsh", ["-c", brokenShape], {
|
||||
encoding: "utf-8",
|
||||
env: { ...process.env, HOME: env.home, PATH: `${env.bindir}:/usr/bin:/bin` },
|
||||
timeout: 30_000,
|
||||
});
|
||||
expect(result.status).toBe(0);
|
||||
expect(lastArgc(env)).toBe(4); // init, --pglite, --json, "<entire flag string>"
|
||||
} finally {
|
||||
env.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it("template uses the positional-params shape, not an unquoted flags var", () => {
|
||||
// Carved (token-reduction Phase 4): count across the tmpl UNION — one
|
||||
// PGLite init site stays in the skeleton, the Path-3/4 sites live in the
|
||||
// brain-init section.
|
||||
const tmpl = readFileSync(
|
||||
join(import.meta.dir, "..", "setup-gbrain", "SKILL.md.tmpl"),
|
||||
"utf-8",
|
||||
) + readFileSync(
|
||||
join(import.meta.dir, "..", "setup-gbrain", "sections", "brain-init.md.tmpl"),
|
||||
"utf-8",
|
||||
) + readFileSync(
|
||||
join(import.meta.dir, "..", "setup-gbrain", "sections", "engine-remediation.md.tmpl"),
|
||||
"utf-8",
|
||||
);
|
||||
it("uses the positional-params shape at all 3 init sites, never an unquoted flags var", () => {
|
||||
expect(tmpl).not.toContain("$GBRAIN_EMBED_FLAGS");
|
||||
const sites = tmpl.match(/gbrain init --pglite --json "\$@"/g) || [];
|
||||
expect(sites.length).toBe(3);
|
||||
const setSites = tmpl.match(/set -- --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024/g) || [];
|
||||
expect(setSites.length).toBe(3);
|
||||
});
|
||||
|
||||
it("treats empty-string VOYAGE_API_KEY the same as unset (no false positive)", () => {
|
||||
const env = makeFakeEnv();
|
||||
try {
|
||||
const calls = runInitWithVoyageGate(env, "");
|
||||
expect(calls.length).toBe(1);
|
||||
expect(calls[0]).not.toContain("voyage");
|
||||
} finally {
|
||||
env.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("template alignment: the .tmpl actually contains the voyage gate", () => {
|
||||
// Belt-and-suspenders: if someone edits the template and drops the
|
||||
// VOYAGE_API_KEY conditional without updating the test above, this catches
|
||||
// it. The shell snippet under test must literally appear in the .tmpl.
|
||||
// Carved union — see comment above.
|
||||
const tmpl = readFileSync(join(import.meta.dir, "..", "setup-gbrain", "SKILL.md.tmpl"), "utf-8")
|
||||
+ readFileSync(join(import.meta.dir, "..", "setup-gbrain", "sections", "brain-init.md.tmpl"), "utf-8")
|
||||
+ readFileSync(join(import.meta.dir, "..", "setup-gbrain", "sections", "engine-remediation.md.tmpl"), "utf-8");
|
||||
|
||||
it("setup-gbrain template gates the embedding-model flag on VOYAGE_API_KEY", () => {
|
||||
// Should appear at least once (currently 3 init sites use the same gate).
|
||||
expect(tmpl).toContain('if [ -n "${VOYAGE_API_KEY:-}" ]; then');
|
||||
expect(tmpl).toContain("--embedding-model voyage:voyage-code-3");
|
||||
expect(tmpl).toContain("--embedding-dimensions 1024");
|
||||
});
|
||||
|
||||
it("setup-gbrain template uses the conditional gate at all 3 PGLite init sites", () => {
|
||||
// Count the gate occurrences. If a future edit adds/removes a PGLite
|
||||
// init site, update this expectation deliberately.
|
||||
const matches = tmpl.match(/if \[ -n "\$\{VOYAGE_API_KEY:-\}" \]; then/g);
|
||||
expect(matches?.length).toBe(3);
|
||||
expect(tmpl.match(/gbrain init --pglite --json "\$@"/g)?.length).toBe(3);
|
||||
expect(tmpl.match(/set -- --embedding-model voyage:voyage-code-3 --embedding-dimensions 1024/g)?.length).toBe(3);
|
||||
expect(tmpl.match(/if \[ -n "\$\{VOYAGE_API_KEY:-\}" \]; then/g)?.length).toBe(3);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user