Files
gstack/make-pdf/test/cli-exit-codes.test.ts
ac20ef1e13 v1.91.16.0 fix: safe installs and upgrades, skills start everywhere, truthful completion (parallel wave) (#3014)
* fix(make-pdf,design): real TOC page numbers and anchors; design models move to gpt-5.5 + gpt-image-2 (#2903, #2807)

* ci: pin every action to a SHA, Scorecard, checksummed Bun, scoped Codex CLI in the CI image, macOS named-regressions job (#1948, #1706, #1997)

* fix(cso,browse): Windows/macOS lease identity, retention skips bad runs, truthful 'not assessed' reports, env.kv false positive, Windows browse spawn fixes (#2894, #2912, #2048, #2637, #2805)

* fix(outside-voice): one Codex model precedence for probe and dispatch, bounded timeouts with KILL escalation, probe cache identity, guarded mktemp, skills hidden in nested Codex (#2776, #2914, #2787, #2881, #2847)

* feat(setup): install registry, ./setup --status, staged activation, per-scope migrations, host tiers and conformance kit, OpenCode/Hermes fixes, Copilot CLI (experimental), disabled_skills, slimmer CLAUDE.md (#1925, #2879, #2347, #2338, #2626, #2629, #2825, #2015, #393, #1206, #2096)

Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com>
Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com>
Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com>

* fix(preamble): skills start in worktree-isolated Claude Code sessions — one literal skill-start line, context-recovery helper, no eval in skill blocks, $N token tripwire, honest plan-mode precedence (#2763, #2896, #2851)

* fix(setup): per-install renders for renamed, vendored and custom-config installs, space-safe alias roots, per-host /cso launcher path (#1882, #2906)

* test: free tests never write the real HOME (home-write tripwire), no orphaned browse daemons, symlinked-TMPDIR (macOS) fixes, gbrain-on-PATH isolation (#2895, #2816, #2829, #2841, #2982, #2911, #2909, #2908, #2910)

* chore: integrate wave follow-ups — regenerate all hosts, README model docs, TODOS, touchfiles, duration seeds, Bun 1.3 telemetry test fix, replay fixtures pinned to their recorded bytes

* v1.91.14.0: release metadata

* test(browse): group-kill cleanup falls back to the leader when macOS returns EPERM for a live group

* ci: Codex login steps run under bash (the CI container's default sh rejects pipefail)

* fix: full-suite repairs — private state roots for 7 real-HOME writers, truthful home-guard attribution with --attribute-home, gstack-paths shim in ship-hook fixtures, Copilot in team-init probes, test-of-test detector, cookie review fixture pinned to the approved excerpt

* fix(windows): LF for Dockerfiles, normalized home-guard exclusions, path.relative fixture containment, ship push test spawn timeouts

* test(design-detect): widen the whole-scan budget test's timing margins so a loaded macOS runner can't time out the single-batch call

* fix(evals): qa protocol names defined for the judge, async PTY judge so overlapping sessions keep their cleanup deadline, bws setup cases use the standard JUDGE_MS tier

* test(pty): the non-blocking judge case is POSIX-only (its fake judge is a shebang script, like the PTY harness it covers)

* test(evals): ship-coverage-audit names absolute skill paths, uses Read, and skips the Step 7 artifact nothing consumes (removes the mechanism behind 3 of 4 historical timeouts)

* test: deflake terminal-agent concurrent startup (write the placeholder record before spawning, like the product) and qa-evidence resumed pipe (barrier on a full high-water mark instead of a sleep)

* fix(design): daemon log stream handles open errors instead of crashing the daemon; shutdown test awaits the stubbed exit

* test(evals): skillify checks the product's resolved state root (no GSTACK_HOME bypass), negative cases check every root; tpa shims live outside the agent's cwd and answer consistently

* test(evals): coverage-audit detector splits parallel columns at │ rails and [+] groups; coverage prompts name absolute skill paths; TODOs from the gate eval census

* fix(plan-eng-review): scope gate always sends the menu (no headless 'Scope pending' exit); PTY harness ends idle turn-ends and API-error panels promptly with a precise diagnostic

* test(evals): one position-bound coverage-read rule credits read-only && chains (labels, context ranges, git displays in either order) and still rejects forged output

* fix(ship): documentation section has one saved source inserted by cat (never retyped); spawned-marker evidence is the child output; refreshed codex/factory ship goldens

* test(docsync): the byte-exact insert positive is POSIX-only (the bash-command observer refuses backslash paths by design)

* test(evals): plan-design-with-ui focus-title rule accepts the comma-less 'dimensions or focus' wording (replayed on the captured CI run, negative controls kept)

* fix(qa-callers): exploratory preflight says the scope/method Reads come after exploratory.md; design-review scan check credits an executed scan proven by one engine run over the changed files

---------

Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com>
Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com>
Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com>
2026-10-03 07:54:09 -07:00

330 lines
15 KiB
TypeScript

/**
* cli.ts error → exit code mapping, pinned through the REAL CLI process.
*
* exit 0 success / 1 bad args / 2 render error / 3 TOC page numbers failed / 4 no browser
*
* `main()` is not exported and runs only under `import.meta.main`, so the
* catch block cannot be driven in-process. Each case spawns `bun cli.ts`
* instead, two ways:
*
* 1. Stubbed orchestrator. A `--preload` file registers a Bun runtime plugin
* whose onLoad swaps make-pdf/src/orchestrator.ts for a module whose
* generate()/preview() throw a chosen error class. The stub imports
* BrowserUnavailableError from the same types.ts cli.ts uses, so the
* `instanceof` check in the catch block is exercised for real.
* 2. No stubs. Filesystem errors that need no browser (missing input, output
* dir that does not exist), bad args, and fake `aside` / `browse` shell
* scripts reached via GSTACK_BROWSE_BIN and a scrubbed PATH. The children
* never see the real PATH, so no real browser is ever probed or launched.
*
* Exit 4 through an UNSTUBBED process is not forceable from inside this
* checkout: resolveBrowseBin() always finds <repo>/browse/dist/browse (a
* BROWSE_ROOTS entry computed from import.meta.dir), so the stubbed case (1)
* is the exit-4 pin here; make-pdf/test/asideClient.test.ts covers the
* renderFailure() classification that produces the error in the first place.
*/
import { afterAll, beforeAll, describe, expect, test } from "bun:test";
import * as fs from "node:fs";
import * as os from "node:os";
import * as path from "node:path";
import { ExitCode } from "../src/types";
import { NO_BROWSER, NO_BROWSER_HELP } from "../../lib/aside-render";
const ROOT = path.resolve(import.meta.dir, "..", "..");
const CLI = path.join(ROOT, "make-pdf", "src", "cli.ts");
const TYPES = path.join(ROOT, "make-pdf", "src", "types.ts");
const isWin = process.platform === "win32";
let tmp: string;
/** PATH for children: nothing on it (no `aside`, no `browse`, no `pdftotext`). */
let emptyBin: string;
/** PATH for children: only a fake `aside` that answers the probe but cannot run a script. */
let fakeAsideBin: string;
/** Fake browse daemon CLIs (reached via GSTACK_BROWSE_BIN, never PATH). */
let browseOk: string;
let browseRefusesNewtab: string;
let inputMd: string;
beforeAll(() => {
tmp = fs.mkdtempSync(path.join(os.tmpdir(), "make-pdf-cli-exit-"));
emptyBin = path.join(tmp, "empty-bin");
fs.mkdirSync(emptyBin);
fakeAsideBin = path.join(tmp, "fake-aside-bin");
fs.mkdirSync(fakeAsideBin);
inputMd = path.join(tmp, "in.md");
fs.writeFileSync(inputMd, "# Smoke\n\nOne paragraph is enough.\n", "utf8");
if (!isWin) {
// Answers `aside --version` and the ASIDE_READY probe like a live app, then
// fails every render script the way an Aside release that renamed its
// private CDP bridge would — the exact shape render() retries on browse.
writeScript(path.join(fakeAsideBin, "aside"), [
'case "$1" in',
' --version) echo "aside 1.26.0-fake" ;;',
" repl)",
' case "$2" in',
' *ASIDE_READY*) echo "ASIDE_READY /fake/aside/session" ;;',
' *) echo "ReferenceError: openTab is not defined" ;;',
" esac ;;",
"esac",
"exit 0",
]);
browseOk = writeFakeBrowse("browse-ok", { pdf: true });
browseRefusesNewtab = writeFakeBrowse("browse-refuses-newtab", { newtabFails: true });
}
});
afterAll(() => {
fs.rmSync(tmp, { recursive: true, force: true });
});
// ─── helpers ──────────────────────────────────────────────────────────────────
function writeScript(file: string, body: string[]): string {
// Builtins only (case/echo/read/printf): the children run with PATH scrubbed.
fs.writeFileSync(file, ["#!/bin/sh", ...body, ""].join("\n"), { mode: 0o755 });
return file;
}
/**
* A stand-in for gstack's browse daemon CLI: `newtab --json` hands out a tab,
* `js` answers with `pageText`, and `pdf --from-file <payload>` writes a fake
* PDF at the payload's `output` path (JSON parsed with parameter expansion).
*/
function writeFakeBrowse(name: string, opts: { pageText?: string; pdf?: boolean; newtabFails?: boolean }): string {
const body = ['case "$1" in'];
body.push(opts.newtabFails
? ' newtab) echo "daemon refused: boom" >&2; exit 1 ;;'
: ` newtab) echo '{"tabId":7}' ;;`);
body.push(` js) echo '${opts.pageText ?? "browser-ok"}' ;;`);
if (opts.pdf) {
body.push(
" pdf)",
' read -r payload < "$3"',
' out="${payload#*\\"output\\":\\"}"',
' out="${out%%\\"*}"',
" printf '%%PDF-1.4 fake\\n' > \"$out\"",
" ;;",
);
}
body.push("esac", "exit 0");
return writeScript(path.join(tmp, name), body);
}
/** Hermetic child env: no real PATH, Aside skipped unless a test opts back in. */
function childEnv(extra: Record<string, string | undefined> = {}): Record<string, string> {
const env: Record<string, string> = {
PATH: emptyBin,
HOME: process.env.HOME ?? tmp,
TMPDIR: os.tmpdir(),
GSTACK_SKIP_ASIDE: "1",
NO_COLOR: "1",
};
for (const [k, v] of Object.entries(extra)) {
if (v === undefined) delete env[k];
else env[k] = v;
}
return env;
}
function runCli(args: string[], opts: { preload?: string; env?: Record<string, string> } = {}) {
const argv = [process.execPath, ...(opts.preload ? ["--preload", opts.preload] : []), CLI, ...args];
const r = Bun.spawnSync(argv, { cwd: ROOT, env: opts.env ?? childEnv(), stdout: "pipe", stderr: "pipe", stdin: "ignore", timeout: 120_000 });
return { code: r.exitCode, stdout: r.stdout.toString(), stderr: r.stderr.toString() };
}
const ORCHESTRATOR_FILTER = String.raw`make-pdf[/\\]src[/\\]orchestrator\.ts$`;
/**
* Write a --preload file whose Bun plugin replaces the orchestrator module with
* one whose generate()/preview() `throw <throwExpr>`. `throwExpr` may reference
* BrowserUnavailableError (imported from the real types.ts, so instanceof holds).
*/
function stubOrchestrator(name: string, throwExpr: string): string {
const module = [
`import { BrowserUnavailableError } from ${JSON.stringify(TYPES)};`,
`export const OUTPUT_TMP_DIR = ${JSON.stringify(os.tmpdir())};`,
`export async function generate() { throw ${throwExpr}; }`,
`export async function preview() { throw ${throwExpr}; }`,
"",
].join("\n");
const preload = [
"// Generated by make-pdf/test/cli-exit-codes.test.ts",
"Bun.plugin({",
' name: "make-pdf-test-stub-orchestrator",',
" setup(build) {",
` build.onLoad({ filter: new RegExp(${JSON.stringify(ORCHESTRATOR_FILTER)}) }, () => ({ loader: "ts", contents: ${JSON.stringify(module)} }));`,
" },",
"});",
"",
].join("\n");
const file = path.join(tmp, `stub-${name}.preload.ts`);
fs.writeFileSync(file, preload, "utf8");
return file;
}
// ─── 1. error class → exit code (orchestrator stubbed) ────────────────────────
describe("cli.ts maps the thrown error class to the exit code (orchestrator stubbed)", () => {
test("BrowserUnavailableError → exit 4, stderr is `$P: <message>`, stdout stays empty", () => {
const msg = `${NO_BROWSER}: ${NO_BROWSER_HELP} (NEEDS_ASIDE: stubbed)`;
const r = runCli(["generate", inputMd], {
preload: stubOrchestrator("no-browser", `new BrowserUnavailableError(${JSON.stringify(msg)})`),
});
expect(r.code).toBe(ExitCode.BrowserUnavailable);
expect(r.code).toBe(4);
expect(r.stderr.trim()).toBe(`$P: ${msg}`);
expect(r.stdout).toBe("");
});
test("a plain Error → exit 2 (render error): message only, the stack only with --verbose", () => {
const preload = stubOrchestrator("render-error", `new Error("PDF render failed: render script did not finish")`);
const quiet = runCli(["generate", inputMd], { preload });
expect(quiet.code).toBe(ExitCode.RenderError);
expect(quiet.code).toBe(2);
expect(quiet.stderr.trim()).toBe("$P: PDF render failed: render script did not finish");
expect(quiet.stdout).toBe("");
const verbose = runCli(["generate", inputMd, "--verbose"], { preload });
expect(verbose.code).toBe(ExitCode.RenderError);
const lines = verbose.stderr.trim().split("\n");
expect(lines[0]).toBe("$P: PDF render failed: render script did not finish");
expect(lines.length).toBeGreaterThan(1); // err.stack follows
expect(verbose.stderr).toMatch(/\bat\b/);
expect(verbose.stdout).toBe("");
});
test("an ENOENT-coded error → exit 1 (bad args) with `file not found: <path>`", () => {
const withPath = runCli(["generate", inputMd], {
preload: stubOrchestrator("enoent-path",
`Object.assign(new Error("ENOENT: no such file or directory, open '/x/in.md'"), { code: "ENOENT", errno: -2, syscall: "open", path: "/x/in.md" })`),
});
expect(withPath.code).toBe(ExitCode.BadArgs);
expect(withPath.code).toBe(1);
expect(withPath.stderr.trim()).toBe("$P: file not found: /x/in.md");
expect(withPath.stdout).toBe("");
// No `path` on the error: the message stands in.
const noPath = runCli(["generate", inputMd], {
preload: stubOrchestrator("enoent-nopath", `Object.assign(new Error("ENOENT: no such file or directory"), { code: "ENOENT" })`),
});
expect(noPath.code).toBe(ExitCode.BadArgs);
expect(noPath.stderr.trim()).toBe("$P: file not found: ENOENT: no such file or directory");
});
test("a TocPaginationError → exit 3, never a PDF with empty TOC page cells", () => {
const r = runCli(["generate", inputMd], {
preload: stubOrchestrator("toc-pages", `Object.assign(new Error("TOC page numbers did not settle after 3 prints"), { name: "TocPaginationError" })`),
});
expect(r.code).toBe(ExitCode.TocPagination);
expect(r.code).toBe(3);
expect(r.stderr.trim()).toBe("$P: --toc: TOC page numbers did not settle after 3 prints");
expect(r.stdout).toBe("");
});
test("a non-Error throw is still reported (String(err)) and exits 2", () => {
const r = runCli(["generate", inputMd], { preload: stubOrchestrator("string-throw", `"boom"`) });
expect(r.code).toBe(ExitCode.RenderError);
expect(r.stderr.trim()).toBe("$P: boom");
expect(r.stdout).toBe("");
});
test("preview shares the same catch block: BrowserUnavailableError → exit 4", () => {
const r = runCli(["preview", inputMd], {
preload: stubOrchestrator("preview-no-browser", `new BrowserUnavailableError("${NO_BROWSER}: stubbed")`),
});
expect(r.code).toBe(ExitCode.BrowserUnavailable);
expect(r.stderr.trim()).toBe(`$P: ${NO_BROWSER}: stubbed`);
expect(r.stdout).toBe("");
});
});
// ─── 2. real error paths that need no browser (no stubs) ──────────────────────
describe("cli.ts exit codes end to end (no stubs, no browser touched)", () => {
test("a missing input is the orchestrator's own guard: exit 2 `input file not found` — NOT the ENOENT branch", () => {
// generate() checks existsSync before any fs call can throw ENOENT, so a
// typo'd input reads as a render error (2), not bad args (1). Pinned as-is.
const missing = path.join(tmp, "does-not-exist.md");
const r = runCli(["generate", missing, "--quiet"]);
expect(r.code).toBe(ExitCode.RenderError);
expect(r.stderr.trim()).toBe(`$P: input file not found: ${missing}`);
expect(r.stdout).toBe("");
});
test("a real ENOENT (output directory does not exist; --to html needs no browser) → exit 1 `file not found`", () => {
const out = path.join(tmp, "no-such-dir", "out.html");
const r = runCli(["generate", inputMd, out, "--to", "html", "--quiet"]);
expect(r.code).toBe(ExitCode.BadArgs);
expect(r.stderr.trim()).toBe(`$P: file not found: ${out}`);
expect(r.stdout).toBe("");
expect(fs.existsSync(out)).toBe(false);
});
test("bad args → exit 1: no input, unknown command (with usage), invalid --to", () => {
const noInput = runCli(["generate"]);
expect(noInput.code).toBe(ExitCode.BadArgs);
expect(noInput.stderr).toContain("$P generate: missing <input.md>");
expect(noInput.stdout).toBe("");
const unknown = runCli(["frobnicate"]);
expect(unknown.code).toBe(ExitCode.BadArgs);
expect(unknown.stderr).toContain("$P: unknown command: frobnicate");
expect(unknown.stderr).toContain("Usage:");
expect(unknown.stdout).toBe("");
const badTo = runCli(["generate", inputMd, "--to", "xml"]);
expect(badTo.code).toBe(ExitCode.BadArgs);
expect(badTo.stderr).toContain("invalid --to 'xml'");
expect(badTo.stdout).toBe("");
});
test("exit 0 paths: no arguments prints usage (stderr); version prints VERSION on stdout only", () => {
const usage = runCli([]);
expect(usage.code).toBe(ExitCode.Success);
expect(usage.stderr).toContain("Usage:");
expect(usage.stdout).toBe("");
const version = runCli(["version"]);
expect(version.code).toBe(ExitCode.Success);
expect(version.stdout.trim()).toBe(fs.readFileSync(path.join(ROOT, "VERSION"), "utf8").trim());
expect(version.stderr).toBe("");
});
});
// ─── 3. fake browsers: success contract, render error vs no browser, mid-run note ─
describe.skipIf(isWin)("cli.ts through fake browsers (GSTACK_BROWSE_BIN + a scrubbed PATH)", () => {
test("gstack's own browser renders: exit 0, stdout is the output path and nothing else, --quiet keeps stderr empty", () => {
const out = path.join(tmp, "steady", "out.pdf");
const r = runCli(["generate", inputMd, out, "--quiet"], { env: childEnv({ GSTACK_BROWSE_BIN: browseOk }) });
expect(r.code).toBe(ExitCode.Success);
expect(r.stdout).toBe(`${out}\n`);
expect(r.stderr).toBe("");
expect(fs.readFileSync(out, "utf8")).toBe("%PDF-1.4 fake\n");
});
test("a browser that is up but cannot render is a render error (exit 2), never `no browser` (exit 4)", () => {
const out = path.join(tmp, "refused", "out.pdf");
const r = runCli(["generate", inputMd, out, "--quiet"], { env: childEnv({ GSTACK_BROWSE_BIN: browseRefusesNewtab }) });
expect(r.code).toBe(ExitCode.RenderError);
expect(r.stderr.trim()).toBe("$P: PDF render failed: browse newtab failed: daemon refused: boom");
expect(r.stderr).not.toContain(NO_BROWSER);
expect(r.stdout).toBe("");
expect(fs.existsSync(out)).toBe(false);
});
test("Aside answers the probe but cannot run the script: the PDF comes from gstack's browser, stderr says so, exit 0", () => {
const out = path.join(tmp, "mid-run", "out.pdf");
const r = runCli(["generate", inputMd, out, "--quiet"], {
env: childEnv({ PATH: fakeAsideBin, GSTACK_SKIP_ASIDE: undefined, GSTACK_BROWSE_BIN: browseOk }),
});
expect(r.code).toBe(ExitCode.Success);
expect(r.stdout).toBe(`${out}\n`);
expect(r.stderr).toContain("Aside was unavailable mid-run; the PDF was rendered through gstack's own browser.");
expect(fs.readFileSync(out, "utf8")).toBe("%PDF-1.4 fake\n");
});
});