mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-31 10:20:42 +02:00
* fix(ci): free-tests lane actually runs the make-pdf e2e gates The 9 make-pdf/test/e2e gate tests probe make-pdf/dist/pdf, browse/dist/browse, and the diagram-render bundle, then self-skip when absent. The required free-tests lane never built any of them, so the gates silently skipped on Linux for their entire life (verified: 9 of 14 skip, exit 0). make-pdf-gate.yml's justification for deleting its Linux leg claimed the free lane covered this — it didn't. - new build:gates script: exactly the three artifacts the gates probe (full bun run build compiles five binaries; ~60-90s tax on the only required check is not warranted) - free-tests.yml: build:gates step + poppler-utils + fonts-noto-color-emoji (fonts must precede the first browse daemon launch — Chromium snapshots fontconfig at startup; verified live: a warm daemon renders tofu, a fresh one embeds NotoColorEmoji) - make-pdf/test/e2e/ci-prereqs.test.ts: GSTACK_EXPECT_BINARIES=1 (set by the workflow) inverts the skip polarity in CI — dropping the build step or poppler fails the lane instead of re-opening the silent-skip hole Pre-flight: all 9 gates green on Linux locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): kill the three zero-test eval jobs (hollow green) - delete the vestigial e2e-codex / e2e-gemini matrix rows: both files are whole-file periodic-tier, so with no row tier: they ran ZERO tests and reported green on every PR (~2 min of runner each, pure false confidence; the periodic lane owns those suites) - e2e-pty-plan-smoke gains tier: gate — its two files are whole-file describeE2ETier('gate'), so the job burned ~7 min of container setup then skipped every describe - KNOWN_TIER_UNSET burned down to empty; the ratchet stays armed so a future row/file tier mismatch fails the suite instead of shipping hollow green Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): least-privilege permissions + fork-safe concurrency keys - evals.yml / evals-periodic.yml evals jobs: explicit contents:read + packages:read (container-image pull) and persist-credentials:false — the jobs that execute PR-authored code with three provider API keys ran on the repo-default token grant with the token written into .git/config - permissions blocks for the 4 workflows that had none (skill-docs, make-pdf-gate, windows-free-tests, windows-setup-e2e) - fork-safe concurrency keys: actionlint, skill-docs, make-pdf-gate, windows-setup-e2e switch from head_ref to PR-number keying — a bare branch name carries no fork prefix, so same-name branches from two forks shared one group and cancelled each other's runs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): one bun version everywhere + drift tripwire Lanes disagreed four ways: 1.3.13 (free-tests, windows, Dockerfile.ci), latest (quality-gate, make-pdf-gate), unpinned (skill-docs, version-gate — setup-bun installs latest), 1.3.10 (.gitlab-ci.yml). Different Bun versions change the runner output shapes the strict classifiers regex-match, spawn semantics, and shell parsing — a lane on a different Bun tests a different product; Dockerfile.ci's own comment records this class biting once already (silent 1.3.13/1.3.14 drift). All surfaces pinned to 1.3.13; test/bun-version-drift.test.ts scans every workflow setup-bun stanza + Dockerfile.ci + .gitlab-ci.yml and fails on any mismatch or unpinned stanza. skill-docs also gains --frozen-lockfile (was bare bun install). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ci): bind the three-way image-tag hashFiles() expressions evals.yml, evals-periodic.yml, and ci-image.yml each compute the CI image tag from hashFiles('.github/docker/Dockerfile.ci', 'bun.lock', 'patches/**') — synced by comment only (TODOS.md 'CI three-way image-tag drift'). If one input list drifts, that workflow computes a different tag for the same content: eval lanes silently rebuild the image every run, or ci-image prebuilds a tag nobody looks up. The test extracts each tag-computation site and fails on any mismatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): ci-image stops rebuilding the identical image every ship - package.json out of the trigger paths: the tag hash deliberately excludes it (version bumps every ship), so every merge rebuilt and re-pushed the IDENTICAL tag (~2m26s for zero content change); patches/** added (it IS a tag input) - manifest existence check (mirrors evals.yml): tag already exists → skip the build - concurrency group: two rapid main pushes raced pushing the same :latest/:buildcache tags - cron staggered 06:00→04:00 Monday: it shared the exact minute with evals-periodic, which could race a half-pushed tag or duplicate the build - timeout-minutes: 30 (was unbounded → 360-min default for a hung docker build) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): quality-gate drops the 74s full-history checkout fetch-depth:0 cost 74 of the job's 92 seconds; the three gates it feeds take ~12s combined. Shallow checkout + exact-SHA fetches for the diff's base/head (an exact-SHA fetch, not a guessed depth — long-lived branches and merge queues still resolve), with a --deepen fallback for push events whose 'before' is unusable. timeout right-sized 20→10 min. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): small-lane batch — timeouts, right-sizing, windows cache warm-start - timeout-minutes on the 6 remaining unbounded jobs (actionlint 5, skill-docs 10, version-gate 10, make-pdf-gate 15, pr-title-sync 5, evals build-image 15) — a hung step sat on GitHub's 360-min default - right-size measured-over-long timeouts: dependency-review 10→5, windows-setup-e2e 15→10 - dependency-review: 2-core runner (28s API call on an 8-core box) and drop .github/workflows/** from its trigger paths (workflow edits have no dependencies to review) - windows caches gain restore-keys: a lockfile bump paid the 26s/43s restore for a guaranteed cold miss Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): scope GSTACK_HOME to each file's execution window Five files assigned process.env.GSTACK_HOME at module scope. Shard processes evaluate sibling modules before running their tests, so the assignment leaked into every other file in the shard — the damage was already visible in defensive workarounds (relink.test.ts:28 'fresh install test saw a neighbor's skill_prefix'; cdp-e2e's own comment documents a sibling's temp dir baked into artifacts). Pattern: save original, assign in beforeAll, restore in afterAll (cdp-e2e already restored but still assigned at load — its window now matches the others). GSTACK_TELEMETRY_OFF and GSTACK_PROJECT_SLUG get the same treatment where they rode along. Victim files' defenses stay in place (cheap insurance). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: tripwire against module-scope GSTACK_HOME assignments Column-0 assignment of GSTACK_HOME / GSTACK_STATE_ROOT in any tracked *.test.ts fails with the file:line and the fix (beforeAll + afterAll restore). Kills the cross-file env-leak class the previous commit swept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): e2e-harness-audit derives its skill census from disk The hand-maintained 39-name SKILL_GLOBS list had drifted to 39 of 54 SKILL.md.tmpl on disk. No live gap today (none of the 15 unlisted skills is interactive), but the next interactive skill would have landed unguarded with zero signal. The audit now walks top-level dirs for SKILL.md.tmpl (statSync so symlinked dirs like connect-chrome count), so new skills are in scope the commit they appear. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(evals): judges honor the eval-model resolution chain + real 429 backoff callJudge inlined GSTACK_EVAL_MODEL_JUDGE || sonnet, silently ignoring the global GSTACK_EVAL_MODEL override every other eval call site honors via lib/eval-model.ts. New 'judge' kind in DEFAULTS (sonnet — the D1a pin-on-regressors calibration stands; model CHOICE unchanged) and callJudge resolves through it: explicit arg > GSTACK_EVAL_MODEL_JUDGE > GSTACK_EVAL_MODEL > default. 429 handling upgraded from one fixed 1s retry (reliably lost races at CI concurrency) to three jittered exponential retries (~1s/4s/16s), honoring the server's retry-after when present. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): the two expect(true) paid stubs become test.todo skill-e2e-spec-execute (600s budget) and skill-llm-eval-spec (300s) reported PASS on every periodic run while asserting nothing. Deleting them would remove the periodic-tier selector surface they exist to register (diff-based selection for spec/ changes), so they become test.todo — reported as todo/skip, never pass — with the v1.1 implementation specs kept in-file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): reactivate 5 quarantined browse tests (2 security) extension-sender-auth's two privileged-message denial tests (content script + missing sender.url — the extension's security boundary) and snapshot's three skips were quarantined 'pre-existing' failures. Root cause: machine-local state on the quarantining dev machines — the test and gate code are byte-identical between the quarantining commit (410b4928) and HEAD, and all five pass deterministically on a clean checkout (68/68 across both files, multiple runs). No assertions weakened, no product changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(evals): activate the 4 paid test files that could never run anywhere carve-section-loading, codex-e2e-plan-format, codex-e2e-recommendation-substance, and llm-judge-recommendation gated on EVALS/tier (free suite loads them as describe.skip) but their names fell outside PAID_TEST_GLOBS, so no paid lane ever selected them — net execution zero, forever. The existing matrix tripwire filtered on isPaidTestFile() first, so it was blind to exactly this class (the same bug that hid the pre-split monolith's gate tests for ~8 releases). - PAID_TEST_GLOBS: codex-e2e* + skill-llm-eval* wildcards (replacing exact names) + llm-judge-recommendation + carve-section-loading; package.json's six test-script glob lists mirrored - codex-e2e-plan-format gains the explicit periodic tier gate its siblings carry (external-service rule) — without it the sharded runner's no-guard default would spawn Codex in the gate tier per PR - eval:bg:periodic --timeout 32400→37800: the census growth pushed the periodic worst case to 35910s; the old value had 270s of headroom BEFORE this change and would now kill healthy runs mid-flight - new test/paid-orphan-tripwire.test.ts: any EVALS/tier-gated test file outside the globs fails the free suite (reasoned SCANNER_EXEMPT for the gate helpers + meta-tests) — the class-killer - paid-shards pins updated: the four orphans now assert INSIDE the census Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(browse): restrictDirectoryPermissions warns and skips symlinked dirs Closes the Windows Free Tests red: recent lane failures showed a platform-unguarded POSIX mode-bit assertion ('Expected: 493' — a symlink-skip test) from PR-branch variants; the KNOWN_WINDOWS_SAFE force-include reason ('mode-bitmask hits are POSIX-branch only') did not hold for that shape, and main had neither the guard nor the behavior. - product: lstat first; a symlinked dir gets a warning and a skip on both platforms — chmod AND icacls dereference the link, so restricting through a symlink hardens an unvetted target (and /inheritance:r could lock out its real owner). All callers already treat hardening as best-effort (try/catch). - test: the symlink regression test, platform-aware — symlinkSync in the house try/catch skip pattern (Windows runners without Developer Mode can't create symlinks), mode-bit assertion guarded off win32, behavior assertions (no throw, warning text, target readable) everywhere; POSIX still proves the skip (0o755 unchanged, not 0o700) - KNOWN_WINDOWS_SAFE reason updated to the now-true premise 20/20 pass on Linux. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): unique tmp dirs for plan artifacts + audited live-repo cwd sites Six paid PTY tests wrote their expected plan artifact to a FIXED shared /tmp path ('/tmp/gstack-test-plan-<mode>.md') and rmSync'd it in finally — under --retry 1, EVALS_JOBS>1, or two concurrent worktrees, a sibling's cleanup deletes this run's artifact and the D19 'agent did not produce expected plan file' assertion fires spuriously. Each test now mkdtemps its own dir, interpolates the unique path into the agent prompt (fixture-sourced prompts get a replaceAll + drift guard that throws if the fixture's literal ever moves), and cleans up its own dir. The 18 cwd:-into-the-live-repo sites were audited: all deliberate (skill registry + hermetic pre-trusted dir, in-repo gen renders, git history reads, slug resolution) — each now carries a '// LIVE-REPO CWD: <reason>' comment so the next audit can tell deliberate from accidental. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): trim the seven over-wall 1700s timeouts to the 1500s physical ceiling 1,700,000ms (28.3 min) exceeded every wall these tests run inside: the 25-min CI job timeout and the 1800s sharded-runner wall (which also leaves --retry 1 zero room for a second attempt). Budget above the wall is fiction, not headroom — a test that actually used it produced a job-level kill (no bun summary, no artifact) instead of a clean per-test timeout. No recorded p95 exists for this family (they are being retiered to periodic in the re-platform wave); the trim stops at the physical ceiling rather than guessing lower. Final policy lands in the Wave-2 eval-budgets constants module. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(gen): main() guard — importing gen-skill-docs no longer regenerates the tree The generator's whole body executed at module load, so any import of it (test/gen-skill-docs.test.ts pulls assertSinglePreamble via require(); test/catalog-trim.test.ts imports helpers) regenerated all 71 SKILL.md in place — the root cause of half the TREE_MUTATING serial-shard entries (hazard class #2532). The body now lives in an exported main(): number behind if (import.meta.main). Semantics preserved exactly: failure exits are immediate (matching the old top-level process.exit), success leaves the event loop to drain so the llms.txt fire-and-forget IIFE finishes its write, and the module stays synchronous/require()-able. Proofs: byte-identical --host all output (git status clean), --dry-run stale-tree still exits 1 (the skill-docs freshness lane depends on it), and the new test/gen-skill-docs-import-purity.test.ts pins load-time purity via a subprocess probe (mtime-based, so a dirty worktree can't false-fail). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(gen): --out-dir renders every host, outputs-only --out-dir was Claude-host-only (gen-skill-docs.ts:842), which forced the codex/factory-regenerating tests (gen-skill-docs, skill-validation, host-config) to mutate the live tree — the reason they sit in the TREE_MUTATING serial shard. The flag now mirrors ALL outputs into the out-dir: external-host trees (.agents/.factory/... via processExternalHost), external section files, openclaw docs, and gstack/llms.txt (a catalog-mode render must never rewrite the tracked index). OUTPUTS ONLY — inputs (templates, sections/, host configs) are always read from ROOT, so an empty out-dir can never feed the render. rewriteSectionBase stays Claude-only (external hosts have their own path grammar). Proofs: in-place --host all is byte-identical (tree clean); --host all --out-dir <mkdtemp> renders the full multi-host tree with ROOT untouched; gen-skill-docs-out-dir tests + 415/415 gen-skill-docs.test.ts green (bin/dev-setup's claude rendering byte-compat). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(evals): every E2E key's dep list names its own declaring test file 129-of-177 keys omitted their own test file, so editing only a test's prompt or assertions selected NOTHING — the changed test never ran on the change that changed it. 135 keys self-registered (110 E2E + 25 LLM-judge), resolved by strict declaration evidence (testName:/ testIfSelected/judge call sites), with skill-name false positives excluded. e2e-tier-alignment's warn-only branch for unregistered files is now a hard failure with a 4-entry KNOWN_UNREGISTERED ratchet (template- literal testNames, fail-open-safe) + a burn-down test so the set only shrinks. Selection sanity: a one-file diff on skill-e2e-qa-workflow now selects its 4 tests (was 0); skill-llm-eval 0 → 25. Known follow-ups (filed): 15 E2E + 2 judge PHANTOM keys select tests that exist nowhere; codex-e2e-plan-format's testIfSelected names have no map keys (run-all only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(evals): ratchet the 8 newly-visible gate-matrix gaps The self-registration sweep made these eight files' gate-tier keys visible to the census for the first time — their gate tests run in NO CI lane today (pre-existing hole, newly measurable). Ratcheted into KNOWN_MATRIX_GAPS with the burn-down note: the paid-lane re-platform runs every gate file by construction and retires this ratchet class. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(test): duration-aware LPT shard packing for the free suite Hash sharding balances file COUNTS (1.15x spread) but not cost — the Playwright-launching files landed 4/3/4/1/2/1 across 6 shards, giving a measured 28s–97s shard spread and ~40s of idle tail on every run. Full-suite mode now packs by recorded per-file durations (longest-processing-time-first) when the committed seed scripts/free-test-durations.json exists. - ONE store, no overlay: the seed is refreshed occasionally via the new --record-durations mode (each file timed in its own child — exact, and immune to bun's stream buffering, where silent passers print no header to timestamp); GSTACK_FREE_TEST_DURATIONS overrides the path for experiments; CI never records - seed is a hint: missing → silent hash-shard fallback; corrupt (bad merge) → one warning + fallback; unknown files → 75th-percentile pessimism so a surprise long-runner can't recreate the tail - packed shards get duration-aware walls (max(base, predicted x 3)) — LPT decouples count from cost BY DESIGN, so the 5s/file heuristic would undersize a shard holding few expensive files - one log line per shard (files + predicted seconds) so packing regressions are diagnosable from any run log - the --shard CI-matrix path is untouched: stable hash indices are its contract - successor note in-code: bun >=1.3.14 ships native --timings/--shard LPT — swap this packer when the repo unpins 1.3.13 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): decouple slop:diff from bun run test; quality-gate runs it per PR 'bun run test' silently appended up to two 120s npx slop-scan runs plus a git worktree add/remove after the suite (2>/dev/null || true) — invisible in the documented '~90-100s' timing and pure friction in the pre-commit loop. Decoupling is not coverage removal: quality-gate.yml now runs slop:diff on every PR (advisory, matching its in-repo 'never blocking' contract), and /review already invokes it explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(test): eval-budgets timeout tiers + fit/ceiling policy test Five named tiers (JUDGE 120s / CAPTURE 300s / CAPTURE_LONG 600s / PTY 900s / PTY_LONG 1200s) replace hand-ratcheted sprawl (46x300s, 46x120s, 44x360s, 44x180s, 27x240s, 19x150s, 13x420s, 12x600s...), much of it inflated to paper over the old 40-way in-shard concurrency that the sharded runner's 1-file-per-shard model kills. Policy test pins: every tier fits the shard wall minus 120s overhead (the structural fix for budgets-above-the-wall fiction), tiers stay ordered, and no paid literal exceeds PTY_LONG x1.25 — oversized tests get split, not budgeted past the wall. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(test): shared runBin helper for bin-script unit tests ~36 free test files each carry a near-identical local run() (spawnSync + utf-8 + {status, stdout, stderr}) differing only in env composition, cwd, and timeout. runBin absorbs the invariant core; options carry the variance (gstackHome sets BOTH GSTACK_HOME and GSTACK_STATE_DIR — the config-precedence trap several locals rediscovered independently; home for $HOME-anchored bins; input/trim/timeout/maxBuffer). Free-test-only by design so it never becomes a de facto global touchfile. Migration of the 36 call sites lands separately (mechanical batches). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): runBin trim assertion — trim shapes stream ends, not interior Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(test): mechanical sweep — 298 paid-test timeouts onto eval-budget tiers 69 files, both shapes (trailing bun-test budgets and runner timeout/timeoutMs options), ROUND-UP ONLY so nothing that passed can start failing: 75 → JUDGE_MS, 137 → CAPTURE_MS, 74 → CAPTURE_LONG_MS, 9 → PTY_MS, 3 → PTY_LONG_MS. Raw >=60s literal count in the paid scope: 395 → 97, of which 51 are non-timeout noise (fixture dates, run IDs) and 46 are enumerated justified holds (comment-carrying calibrated budgets, poll-loop constants, utility spawn waits, and the seven physical-ceiling 1_500_000 sites). The eval-budgets policy ratchet keeps the residue from regrowing. Known collapse: where an inner runner budget and its enclosing test budget now share a tier, the old stagger is gone — an overrun surfaces as a bun test timeout instead of a graceful runner timeout (diagnosability trade, not a correctness one). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: coverage fill — 95 tests for six zero-coverage surfaces - eval CLI family (eval-list/compare/summary + eval-select smoke): the primary interface to eval results had no tests; isolation via a fake gstack-slug under a mkdtemp HOME (the scripts' real resolution path — they do NOT honor GSTACK_EVAL_DIR; only EvalCollector does). Pinned current behavior: eval-list does NOT exclude _partial runs (documented improvement candidate) - slop-diff (runs on every /review + quality-gate): fixture git repo + first-on-PATH npx stub (never downloads real slop-scan); no-diff early exit, missing-scanner fallback, fingerprint line-insensitivity, merge-base worktree scan - bin/gstack-code-intelligence CLI arg surface (lib was covered, the 284-line CLI wasn't): select/consent/suggest/index/search gating; pinned: --help routes to usage failure exit 1 (no handler) - browse media-extract: the page.evaluate callback exercised in-process against a mock DOM (no exports added) — lazy-src fallback chain, HLS/DASH detection, bg-image url() parsing, 500-element cap - browse session-cookie-store: factory contract (cookieName/ttlMs/ maxSessions eviction, cross-store isolation, mint→validate round-trip); store is in-memory — no fs cases exist - lib/version-source direct unit tests (gstack-version-bump.test.ts spawns the bin, never imports the lib): parse/format/cmp/bump coercion, npm 4→3 translation, #2501 mangled-JSON regression class All hermetic (mkdtemp homes, runBin child isolation); windows curation correctly partitions the six. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(test): first runBin migration batch (3 of ~36 run() duplicates) explain-level-config, benchmark-cli, evidence move onto the shared helper; each file's remaining special-case spawnSync sites (raw-buffer probes, env-scrub probes) stay put deliberately. 55/55 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(evals): paid shards spool to disk + shared runShardChild lifecycle - runPaidShard no longer buffers whole 30-min stream-json streams in RAM (x concurrent jobs): every byte tees to a per-shard log file (slug-named, path printed at START for mid-run inspection and on the FAILED terminal line); failures print a 64KiB tail read back from disk; passing shards stay quiet (the file is the record) — the free runner's proven contract. Classification unchanged: the strict classifier still sees every byte first. - the ~35 duplicated spawn/group-kill/wall-timer/finally-reap lines move into runShardChild in test-strict-output.ts (detached-per- platform spawn, signal forwarding, SIGKILL group kill at the wall, drain-before-verdict); designed so the free runner can migrate later - expectedFiles drift fixed toward ENFORCEMENT: the injected-command exemption is gone — a fake command exiting 0 without bun's terminal summary now reads FAILED (pinned: silent-pass → failed) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(evals): parent-computed selection propagates to shard children The sharded runner computed diff selection once, then each of its 48-73 children recomputed it at module load — including, on touchfiles-diff branches, a per-child bun subprocess evaluating the old data file (20s timeout each). The parent now serializes {version, selected, reason} as EVALS_SELECTION_JSON into the shard env; e2e-helpers adopts it at load. Fail-open preserved: any parse/shape violation → ONE stderr warning + local recompute; absent env → silent local compute (non-sharded entrypoints unchanged). Drift test pins parent→child round-trip to identical selection decisions plus the malformed/absent cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): kill the four worst fixed sleeps (300s/30s/30s/20s) - watchdog.test: the 20s blind wait for one production parent-watchdog tick becomes BROWSE_PARENT_WATCHDOG_INTERVAL_MS=250 (new env knob in server.ts, NaN-safe, production default unchanged) + polls for the boot line and the tick's stay-alive log — strictly stronger (the old form never proved a tick observed the parent death). 24s → 3.6s. - stop-dead-daemon / terminal-agent-owner-watchdog: the 300s/30s stand-in child lifetimes become stdin-EOF-bound — the child can never self-exit mid-test on a slow runner (spurious-failure class) and self-reaps instantly if the test dies (no 300s orphans). Node-compat stdin APIs (owner-watchdog runs on the Windows lane). - browser-skill-commands: the sleeper fixture's 30s self-time becomes 8s (no stdin pipe exists in runToFiles) — far above the 1s product timeout it must outlive, below the test ceiling, so a timeout-kill regression fails on clean assertions instead of an opaque bun timeout; added: stdout must NOT contain 'done'. 45/45 green across the four files + server tripwires. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): gen-skill-docs + catalog-trim leave the serial mutator shard gen-skill-docs.test.ts's 15 in-place generator spawns now render into mkdtemp out-dirs (gitignored-artifact reads repointed; the handshake scan's silent console.warn degrade became a hard assertion); its tracked-tree reads (freshness dry-run, SKILL.md content pins) stay reads. catalog-trim needed no change beyond the earlier main() guard — its import is now side-effect-free (pinned by the import-purity test). Both TREE_MUTATING entries deleted in this commit, per the transition rule: an entry leaves in the same commit as the file's last in-place write. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): skill-validation renders codex host into an out-dir Its 3 in-place --host codex regeneration sites collapse into one module-level --out-dir render; assertions untouched. TREE_MUTATING entry deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): host-config self-provisions goldens (ordering dependency severed) Its goldens were 'produced by gen-skill-docs.test.ts' with a when-missing beforeAll fallback that wrote the live tree — an inter-test ordering dependency the serial shard hid. It now renders codex+factory UNCONDITIONALLY into its own out-dir and reads goldens only from there (the Claude golden deliberately keeps reading tracked ship/SKILL.md — a read; out-dir claude renders repoint section-base paths by design). TREE_MUTATING entry deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): gbrain-detection-override drops mutate-then-git-restore regenAndSnapshot renders --host claude --out-dir <mkdtemp> (+ --respect-detection) and snapshots probes from the out-dir. The git-restore machinery is deleted outright — it restored only PROBE_FILES of the 71 files each call wrote, so a stale tree kept the other 68 dirty (the partial-restore bug), and its 'no output-path arg' comment had been false since --out-dir landed. TREE_MUTATING entry deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): catalog-mode-full renders to out-dir; restore machinery deleted The full-catalog smoke no longer rewrites all 71 SKILL.md then regenerates to restore (with its 'CRITICAL: failed to restore' prayer path) — it renders into a mkdtemp and additionally asserts tracked ship/SKILL.md is byte-unchanged. TREE_MUTATING entry deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): idempotency proof strengthens to two-out-dir recursive diff Two renders into two separate out-dirs, EVERY file diffed byte-for-byte (claude-only and --host all; normalization only for each dir's own sanctioned section-base repoint; presence-sanity lists guard against a vacuous empty-dir pass) — strictly stronger than the old in-place double-regen that sampled 5 files. TREE_MUTATING entry deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): spec-template-sync compares an out-dir render, not an in-place one TREE_MUTATING entry deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(test): the serial tree-mutating shard dissolves — TREE_MUTATING is empty Zero mutators remain (all eight render into out-dirs now), so the four ratchet READERS (parity caps, size budgets, carve parity/ordering) get a quiet tree by construction in any shard and rejoin the parallel phase. The ~35-40s serial tail on every full-suite run is gone. The mechanism stays: a future test that genuinely must write shared artifacts in place earns an entry with a reason and is serialized again; the census pin still fails on renamed keys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(gen): out-dir byte-identity + tree-clean pins for external hosts codex render: porcelain unchanged AND out-dir gstack-ship/SKILL.md byte-identical to a fresh in-place render (+openai.yaml presence); --host all render: exit 0, porcelain unchanged, claude + .agents + .factory + llms.txt + openclaw docs all present in the out-dir. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(test): commit the initial free-test durations seed (496 files) Recorded via --record-durations on a quiescent tree: 479s serial total, p50 92ms / p90 1.8s / max 31.4s — the top-heavy cost shape LPT packing exists for. A hint, not a contract: refresh opportunistically with bun run test:free --record-durations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(evals): planner/executor/report modes — the CI re-platform surface One PLANNER computes diff selection + the slice plan ONCE and writes a manifest (--emit-plan <path> --slices K); K executors consume it (--plan <path> --slice i), never self-selecting, and write slice-result artifacts; a REPORT reconciles results against the manifest (--report <dir>) fail-closed: a slice whose artifact never landed is a FAILURE, a planned shard nobody reported fails, wrong-slice/duplicate/cross-tier results fail. Kills per-slice selector divergence and hollow-lane aggregation at the root. - hollow-shard guard: under EVALS_ALL, exit 0 with ZERO executed tests (bun's 'Ran N tests' now captured by the classifier — additive) is 'passed-empty' and fails the run; selective runs keep it 'passed' with one warning (in-file diff/tier self-skips are legitimate there); unknown counts are never guessed hollow - retry parity: --retry 1 default + RETRY_OVERRIDES literals for the three files whose old matrix rows earned retries: 2 (stale entries pinned against disk) - live smoke: gate plan = 48 shards across 6 slices; report mode exits 1 on a fabricated missing slice, 0 when complete Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(ci): sliced paid lane (planner -> 6 executors -> fail-closed report) The parity-phase re-platform: evals.yml gains a second, sliced lane driven by scripts/test-paid-shards.ts — the SAME engine local eval:bg:gate uses, so CI and local share one selection engine. - plan-slices: ONE planner (fetch-depth 0 — the only job needing history) emits the manifest; selection fails open to run-all, never per-slice (the divergence class is structurally dead) - eval-slices: 6-way matrix consuming the manifest; PTY seed + skill-registration steps run unconditionally (idempotent — a sliced lane cannot key them on suite names); aggregate spawn budget 6 x EVALS_JOBS=2 x EVALS_CONCURRENCY=2 = 24 lane-wide (the matrix's 40-way per row queued session startup behind 39 siblings — the timeout-flake family root); slice results + spooled shard logs uploaded as artifacts - slices-report: reconciles slice artifacts against the manifest FAIL-CLOSED via --report — a slice whose artifact never landed, or a planned shard nobody reported, is a failure, not an absence - sequenced needs: evals so provider concurrency never doubles while both lanes coexist; the matrix + its ratchets are deleted after demonstrated parity (intersection + expected-additions comparison) - workflow_dispatch gains evals_all (default true) for parity runs and post-merge smokes — a dispatch can never silently select zero Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(ci): weekly periodic lane runs EVERY periodic test + gate census backstop evals-periodic.yml re-platforms onto the sharded runner: planner manifest → 6 executor slices → FAIL-CLOSED report. This IS the coverage contract: all ~70 periodic-tier files weekly (EVALS_ALL=1), killing the silent-rot class where a hard-coded 9-file matrix left ~57 files running NOWHERE (the autoplan E2E rotted invisibly for months). - test/helpers/periodic-exclude-data.ts: reasoned exclusions in their OWN literals file (deliberately not touchfiles-data — map-diff evaluates old versions of that file standalone). Every entry carries reason + tracking with a re-entry condition; the runner surfaces each exclusion per run; policy test pins real-file + non-empty fields. Initial: ship-idempotency + brain-privacy-gate (documented-red, never green) and skill-e2e-ios (manual hardware). The TODOS 'sidebar E2E trio' turned out already deleted — only tombstone tests remain. - gate-census job: weekly EVALS_ALL gate-tier run — PR lanes are diff-billed, so without this the full gate census might never execute anywhere; with the hollow-shard guard it is a census-health check (exit 0 + zero executed tests fails), not just a test run. - failure notification is a concrete gh issue UPSERT (one tracking issue, commented per red week — never issue-per-week spam), with issues:write scoped to the report job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: TESTING_INTERNALS covers the 2026-08 runner overhaul LPT-packed free suite + --record-durations, the emptied TREE_MUTATING mechanism, the sharded paid runner as the single selection engine, CI planner/executor/report with the fail-closed report and hollow-shard guard, the weekly coverage contract + exclusions policy, and the eval-budgets timeout tiers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(CLAUDE.md): testing prose matches the overhauled runners - bun run test: duration-packed shards + --record-durations; the trailing serial tree-mutating shard no longer exists - two-tier system: the sliced CI lanes (one engine local+CI), the weekly all-periodic coverage contract + exclusions, the gate census - periodic detach timeout 32400 → 37800 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(TODOS): close the absorbed test-infra items, file the overhaul follow-ups Closed with receipts: the periodic coverage contract (implemented as full weekly coverage + exclusions), the eval-harness observability P1 (verified already landed: heartbeat, incremental _partial persistence, live stderr + eval-watch), and the sidebar trio (already deleted — tombstones remain). Filed: matrix deletion after parity, the required-check maintainer decision, browse /tmp-namespace hardening, PTY boot-readiness waits, the single typed test registry, bun-native LPT swap, runBin/free-runner migrations, eval-list partial exclusion, phantom key cleanup, duration-weighted slicing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * v1.73.0.0: test/CI overhaul — green means green, suites restructured for speed Version + release notes for the audit-and-overhaul branch: every silently-skipping or never-running test class fixed and tripwired, the free suite duration-packed with the serial mutator shard dissolved, the paid lane re-platformed onto the sharded runner (planner/slices/ fail-closed report, parity phase), the weekly all-periodic coverage contract, eval-budget timeout tiers, and 95 new coverage tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): first-live-run fixes — executor history + two environment-blind assertions The sliced lane's first run (PR #2721) did its job: the planner and report worked, the manifest governed, and every failure had a name. Three were fixable on the spot: - executor + gate-census checkouts get fetch-depth: 0 — files with SELF-derived selection (the LLM-judge map, routing) walk git at module load, and selection is deliberately fail-closed on git errors, so the shallow checkout crashed those shards ('ambiguous argument main...HEAD'). The manifest still governs WHICH shards run. - landscape --toc gate: the exact toBe(3) landscape-page count was font-metric-dependent (3 on Amazon Linux, 2 on ubuntu CI — the same disease the file's own page-index comment warns about). Now a comparative invariant: --toc must not CHANGE the landscape count vs a baseline render. - paid-run-manifest parse test builds its manifest under EVALS_ALL so it never walks git (proven with GIT_DIR=/nonexistent). Remaining first-run failures are newly-exposed rot in gate files that had never executed in CI (skillify D1 refusal, session-intelligence context-restore, one tpa-apple-ban retry flake) — being probed separately; they are the lane WORKING, not the lane failing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(TODOS): file the three first-execution findings from the sliced lane's live run Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * v1.74.0.0: queue-advance — #2722 claims the v1.73.0.0 slot The version gate caught a live queue collision (its whole job); same MINOR bump level, next free slot per bin/gstack-next-version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): per-shard CHROMIUM_PROFILE — the collision class duration packing exposed Nine test files launch in-process persistent contexts or daemons that default to the SHARED ~/.gstack/chromium-profile. Two concurrent shard processes on one profile dir kill each other's browser — observed live on CI once duration packing recomposed shards: handoff's launchPersistentContext died 'Target page, context or browser has been closed' (--user-data-dir=~/.gstack/chromium-profile in the call log) while a sibling shard's daemon logged 'Chromium process crashed'. Hash sharding had masked the collision by chance placement; handoff passes standalone everywhere. Fix at the runner, not per file: each shard child gets CHROMIUM_PROFILE=<shard-state>/chromium-profile (the documented env knob, same isolation idea as the existing per-shard TMPDIR). Files within a shard run serially, so sharing the per-shard profile is safe; config.test's resolution-order tests save/restore the env around their assertions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): landscape --toc gate asserts promotion PRESENCE, not counts Two rounds of CI receipts: the exact toBe(3) was font-metric-coupled (3 on Amazon Linux, 2 on ubuntu), and the baseline-comparison repair then failed 2-vs-3 across renders SECONDS apart in one CI job while the sibling no-toc test saw 3 — per-render image-promotion timing makes any count assertion here a coin flip. The sibling test owns exact promotion counts; this test's actual invariant is that --toc does not break the promotion machinery: >=1 landscape page + the TOC rendered. Also drops the second render (halves the test's runtime). Flaky per-render image promotion itself is worth its own look — noted in TODOS with these receipts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(TODOS): file the per-render image-promotion nondeterminism (receipts from PR #2721) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): per-FILE Chromium profiles for the nine in-process launcher files Completes the profile-isolation work: the per-shard CHROMIUM_PROFILE stopped cross-shard kills; these nine files launch in-process persistent contexts and could still collide with a lingering daemon a sibling file spawned on the SAME shard profile. Each now scopes a mkdtemp profile via beforeAll/afterAll (the module-scope-tripwire-safe pattern), cleaned up per file. All nine green solo and in combined runs, except the pre-existing commands+snapshot pairing — proven identical WITH and WITHOUT these edits (baseline receipts) — which is the daemon-lifecycle follow-up now extended in TODOS with this session's receipts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(browse): Chromium-crash exit is daemon-only — embedded launches never kill their host handleChromiumDisconnect unconditionally process.exit()ed. Correct for the standalone daemon (its supervisor/user must notice); suicidal when a TEST launches BrowserManager in-process: a mid-suite Chromium death exited the whole bun shard with no terminal summary — the exact truncation class the strict runner flags (observed live: CI shard 1 oneb233299died at cache-concurrent-refresh right after a daemon-spawning gate test; with this fix the same pairing runs to completion and REPORTS instead of dying). The standalone entrypoint opts in via markDaemonProcess() under server.ts's import.meta.main gate — the same embedder contract its signal handlers already use (gbrowser phoenix keeps its own handlers). Embedded contexts now get the disconnect log line and continue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): context-restore assertion is evidence-based, not prose-matching The test failed twice per run in TWO CI cycles while passing locally 4/4: the prompt said 'present the content' and the check grepped the FINAL message for exact phrases — local runs quoted the file, CI runs paraphrased ('the most recent context is from branch-b...') and the substring check lost the coin flip. - prompt now demands machine-checkable output: the newest file's '## Working on:' heading VERBATIM + a literal 'RESTORED: <filename>' marker (the mtime-scramble and cross-branch subject matter untouched) - assertion ordered strongest-first: RESTORED marker → legacy content phrases → tool-call corroboration (Read/Bash input naming the newer file, credited ONLY when the older file was never read — a both-files run must still present the right one) - the older-file negative got STRONGER: an explicit RESTORED marker naming the older file fails even if wintermute words appear elsewhere - sibling scan: context-recovery-artifacts got the additive prompt-side treatment only (quote the matched literals verbatim); its lenient 1-of-6 assertion deliberately unchanged 3/3 consecutive local green with all evidence classes firing (marker=true, content=true, toolNewer=true, toolOlder=false). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): skillify family — HOME==cwd broke project-skill registration Root cause (forensically pinned from stream-json init events + a kill-after-init probe): with HOME set EQUAL to the child's cwd, claude resolves <cwd>/.claude/skills as the PERSONAL skills directory and the seeded project-tier skills never register — the Skill tool returned 'Unknown skill'. The provenance-refusal test then improvised a refusal whose wording missed the regex (the deterministic CI+local red); the happy-path and approval-reject siblings passed only because their agents self-recovered by Reading SKILL.md manually — silently not exercising the Skill-tool path at all. All three tests now use HOME=<workDir>/home (a fresh subdir keeps the override's intent: child ~/.gstack writes land in the assertable sandbox, without the cwd collision). Refusal test additionally: a 'not registered/unknown skill' tripwire (a not-loaded skill can never pass as a refusal) and the refusal regex now matches assistant text only — the skill BODY echoed into the transcript contains the exact refusal message, so the old full-surface match could pass vacuously once the skill loaded. Sibling disk assertions sweep both $HOME/.gstack and cwd .gstack roots (positives and negatives). Verified paid: refusal 2x consecutive green with the skill's EXACT message rendered ('Launching skill: skillify' in-transcript), then the full file 5/5 green (~$1.35) with both siblings driving real Skill calls (25-27 turns each). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(TODOS): two of three first-execution findings fixed (skillify family, context-restore) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): context-restore gets a private home — the REAL root cause was fixture sharing The evidence-based assertion fix was treating a symptom. The slice artifact's embedded transcript showed the CI agent restoring 20260829-context-save-skill-test.md — the checkpoint the SIBLING context-save test wrote into the SHARED gstackHome checkpoints dir, which by filename-prefix ordering genuinely IS the newest. The agent behaved CORRECTLY; the test's fixture set was open to concurrent sibling writes, and bun --concurrent ordering differs between CI (save finished first) and local (restore listed first) — the entire local-green/CI-red split explained. The restore test now uses its own .gstack-restore-home (the whole home moves, not just the handed path — an agent deriving the dir from GSTACK_HOME/projects/<slug> must land in the closed set too). Full file 4/4 paid green with all evidence flags firing. Also: the on-failure shard-log artifact glob uploaded nothing — the Fix-bun-temp step points TMPDIR at /home/runner/.cache, so the spool lands there, not /tmp. Both eval workflows now glob both locations (this gap is why diagnosing THIS failure required digging transcripts out of the slice-results artifact). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(evidence): carry the real index mtime onto gstack-wtree's temp copy The stat-cache seed (cp of the real index) stamped the temp index "now", which defeats git's racy-git protection: an entry is only re-hashed when its cached mtime is not older than the index file itself, so a same-size rewrite landing in the same second as the last real index write looked non-racy, kept its stale stat-cache entry, and vanished from the fingerprint — evidence stayed FRESH after a source change. This is the CI flake in test/evidence.test.ts "allow-paths carve-out" (sub-second alignment on fast runners: expected STALE exit 1, got FRESH exit 0). touch -r restores the original index timestamp, reinstating the exact racy window git itself uses. Deterministic regression pin in test/review-log.test.ts reproduces the miss with pinned zero-nsec timestamps (fails on the old script, passes now); receipts: manual probe shows the fresh-stamped copy returning the clean tree for a same-size 'hello'→'howdy' rewrite while the mtime-carried copy detects it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): landscape gate bounds the promotion count instead of pinning 3 The alt-hinted image promotion rides the per-render measurement race already filed in TODOS (2-vs-3 landscape pages on renders seconds apart — CI receipts from PR #2721, now reproduced locally). Pin the two deterministic promotions as the floor and the three promotable blocks as the ceiling (anything above 3 means the veto leaked); the veto/portrait assertions remain exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Test <test@test.com>
3861 lines
174 KiB
TypeScript
3861 lines
174 KiB
TypeScript
import { describe, test, expect, afterAll } from 'bun:test';
|
|
import { assertSinglePreamble } from '../scripts/gen-skill-docs';
|
|
import { COMMAND_DESCRIPTIONS } from '../browse/src/commands';
|
|
import { SNAPSHOT_FLAGS } from '../browse/src/snapshot';
|
|
import * as fs from 'fs';
|
|
import * as path from 'path';
|
|
import * as os from 'os';
|
|
import { spawnSync } from 'child_process';
|
|
|
|
const ROOT = path.resolve(import.meta.dir, '..');
|
|
const MAX_SKILL_DESCRIPTION_LENGTH = 1024;
|
|
|
|
// Carved-skill aware (v2 plan T9): ship is now a skeleton SKILL.md + sections/*.md.
|
|
// Read the union so assertions about content that MOVED into a section still pass.
|
|
// The skeleton is a subset of the union, so skeleton-only assertions also hold,
|
|
// and negative assertions stay safe (the absent phrases live in neither file).
|
|
function readSkillUnion(skill: string): string {
|
|
let t = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
const secDir = path.join(ROOT, skill, 'sections');
|
|
if (fs.existsSync(secDir)) {
|
|
for (const f of fs.readdirSync(secDir).sort()) {
|
|
if (f.endsWith('.md')) t += '\n' + fs.readFileSync(path.join(secDir, f), 'utf-8');
|
|
}
|
|
}
|
|
return t;
|
|
}
|
|
function readShipUnion(): string {
|
|
return readSkillUnion('ship');
|
|
}
|
|
|
|
// Token-reduction Phase 1: the preamble's inline bash (session bookkeeping,
|
|
// config echoes, telemetry producers, artifacts sync) moved into
|
|
// bin/gstack-skill-start / bin/gstack-skill-end. The render carries a one-line
|
|
// invocation fence + interpretation prose. Assertions that pinned inline-bash
|
|
// internals now pin the scripts (the new home); render-side assertions pin the
|
|
// fence + prose. Script behavior is pinned by test/gstack-skill-start.test.ts.
|
|
const SKILL_START_SCRIPT = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
|
|
const SKILL_END_SCRIPT = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-end'), 'utf-8');
|
|
|
|
function extractDescription(content: string): string {
|
|
const fmEnd = content.indexOf('\n---', 4);
|
|
expect(fmEnd).toBeGreaterThan(0);
|
|
const frontmatter = content.slice(4, fmEnd);
|
|
const lines = frontmatter.split('\n');
|
|
let description = '';
|
|
let inDescription = false;
|
|
const descLines: string[] = [];
|
|
|
|
for (const line of lines) {
|
|
if (line.match(/^description:\s*\|?\s*$/)) {
|
|
inDescription = true;
|
|
continue;
|
|
}
|
|
if (line.match(/^description:\s*\S/)) {
|
|
return line.replace(/^description:\s*/, '').trim();
|
|
}
|
|
if (inDescription) {
|
|
if (line === '' || line.match(/^\s/)) {
|
|
descLines.push(line.replace(/^ /, ''));
|
|
} else {
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
|
|
if (descLines.length > 0) {
|
|
description = descLines.join('\n').trim();
|
|
}
|
|
return description;
|
|
}
|
|
|
|
function extractMarkdownSection(content: string, heading: string): string {
|
|
const escaped = heading.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
|
const startMatch = content.match(new RegExp(`^${escaped}.*$`, 'm'));
|
|
expect(startMatch?.index).toBeDefined();
|
|
const start = startMatch!.index!;
|
|
const afterHeading = start + startMatch![0].length;
|
|
const nextSection = content.slice(afterHeading).match(/\n## /);
|
|
const end = nextSection?.index === undefined
|
|
? content.length
|
|
: afterHeading + nextSection.index;
|
|
return content.slice(start, end).trim();
|
|
}
|
|
|
|
function extractPreambleBeforeWorkflow(content: string, workflowMarkers: string[]): string {
|
|
const markerIndexes = workflowMarkers
|
|
.map(marker => content.indexOf(marker))
|
|
.filter(index => index >= 0);
|
|
expect(markerIndexes.length).toBeGreaterThan(0);
|
|
return content.slice(0, Math.min(...markerIndexes));
|
|
}
|
|
|
|
function isRepoRootSymlink(candidateDir: string): boolean {
|
|
try {
|
|
return fs.realpathSync(candidateDir) === fs.realpathSync(ROOT);
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
// Dynamic template discovery — matches the generator's findTemplates() behavior.
|
|
// New skills automatically get test coverage without updating a static list.
|
|
const ALL_SKILLS = (() => {
|
|
const skills: Array<{ dir: string; name: string }> = [];
|
|
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
|
skills.push({ dir: '.', name: 'root gstack' });
|
|
}
|
|
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
|
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
|
if (fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) {
|
|
skills.push({ dir: entry.name, name: entry.name });
|
|
}
|
|
}
|
|
return skills;
|
|
})();
|
|
|
|
// hosts/claude.ts generation.skipSkills entries would filter here; the set is
|
|
// currently empty (the /claude outside-voice template was removed).
|
|
// The claude host deliberately skips some skills (skipSkills — e.g. the
|
|
// /claude outside-voice skill exists only for non-Claude hosts), so those
|
|
// dirs have a SKILL.md.tmpl but no generated claude-host SKILL.md on a fresh
|
|
// checkout. Every generated-file assertion must exclude them or it is red on
|
|
// every clean clone (it was, invisibly, until the free suite ran in CI).
|
|
import { getHostConfig as __getHostConfig } from '../hosts/index';
|
|
const CLAUDE_SKIPPED = new Set(__getHostConfig('claude').generation.skipSkills ?? []);
|
|
const CLAUDE_GENERATED_SKILLS = ALL_SKILLS.filter(s => !CLAUDE_SKIPPED.has(s.dir));
|
|
|
|
// ─── Out-dir render isolation ────────────────────────────────
|
|
// Every generator invocation in this file that used to regenerate the live
|
|
// tree (the gitignored .agents/.factory/... host dirs included) now renders
|
|
// into this module-level out-dir: ONE `--host all` render covers the claude
|
|
// host plus every external host, and all golden-artifact reads plus the
|
|
// per-host `--dry-run` determinism checks point here. The tracked tree is
|
|
// only ever READ (the `generated files are fresh` dry-run deliberately
|
|
// compares against the committed files — that is a read, not a write).
|
|
// Out-dir renders of external hosts are byte-identical to in-place renders
|
|
// (pinned by test/gen-skill-docs-out-dir.test.ts).
|
|
const EXTERNAL_OUT = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-gen-docs-out-'));
|
|
{
|
|
const render = Bun.spawnSync(
|
|
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--out-dir', EXTERNAL_OUT],
|
|
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe' },
|
|
);
|
|
if (render.exitCode !== 0) {
|
|
throw new Error(
|
|
`gen-skill-docs --host all --out-dir failed (exit ${render.exitCode}):\n${render.stderr.toString()}`,
|
|
);
|
|
}
|
|
}
|
|
afterAll(() => {
|
|
fs.rmSync(EXTERNAL_OUT, { recursive: true, force: true });
|
|
});
|
|
|
|
describe('gen-skill-docs', () => {
|
|
// Browse carve (token-reduction Phase 4): the command reference + snapshot
|
|
// flags render into browse/sections/command-list.md now — read the
|
|
// skeleton+sections union so these pins hold across the carve.
|
|
test('generated SKILL.md contains all command categories', () => {
|
|
const content = readSkillUnion('browse');
|
|
const categories = new Set(Object.values(COMMAND_DESCRIPTIONS).map(d => d.category));
|
|
for (const cat of categories) {
|
|
expect(content).toContain(`### ${cat}`);
|
|
}
|
|
});
|
|
|
|
test('generated SKILL.md contains all commands', () => {
|
|
const content = readSkillUnion('browse');
|
|
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
|
const display = meta.usage || cmd;
|
|
expect(content).toContain(display);
|
|
}
|
|
});
|
|
|
|
test('command table is sorted alphabetically within categories', () => {
|
|
const content = readSkillUnion('browse');
|
|
// Extract command names from the Navigation section as a test
|
|
const navSection = content.match(/### Navigation\n\|.*\n\|.*\n([\s\S]*?)(?=\n###|\n## )/);
|
|
expect(navSection).not.toBeNull();
|
|
const rows = navSection![1].trim().split('\n');
|
|
const commands = rows.map(r => {
|
|
const match = r.match(/\| `(\w+)/);
|
|
return match ? match[1] : '';
|
|
}).filter(Boolean);
|
|
const sorted = [...commands].sort();
|
|
expect(commands).toEqual(sorted);
|
|
});
|
|
|
|
test('generated header is present in SKILL.md', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('AUTO-GENERATED from SKILL.md.tmpl');
|
|
expect(content).toContain('Regenerate: bun run gen:skill-docs');
|
|
});
|
|
|
|
test('generated header is present in browse/SKILL.md', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('AUTO-GENERATED from SKILL.md.tmpl');
|
|
});
|
|
|
|
test('snapshot flags section contains all flags', () => {
|
|
const content = readSkillUnion('browse');
|
|
for (const flag of SNAPSHOT_FLAGS) {
|
|
expect(content).toContain(flag.short);
|
|
expect(content).toContain(flag.description);
|
|
}
|
|
});
|
|
|
|
test('every skill has a SKILL.md.tmpl template', () => {
|
|
for (const skill of ALL_SKILLS) {
|
|
const tmplPath = path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
|
expect(fs.existsSync(tmplPath)).toBe(true);
|
|
}
|
|
});
|
|
|
|
test('every skill has a generated SKILL.md with auto-generated header', () => {
|
|
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
|
const mdPath = path.join(ROOT, skill.dir, 'SKILL.md');
|
|
expect(fs.existsSync(mdPath)).toBe(true);
|
|
const content = fs.readFileSync(mdPath, 'utf-8');
|
|
expect(content).toContain('AUTO-GENERATED from SKILL.md.tmpl');
|
|
expect(content).toContain('Regenerate: bun run gen:skill-docs');
|
|
}
|
|
});
|
|
|
|
// #1778: strict YAML parsers (Codex/OpenAI skill loading) reject frontmatter
|
|
// whose plain `description:` scalar contains an interior ": " (read as a nested
|
|
// mapping). Parse EVERY generated frontmatter block with a strict YAML parser,
|
|
// not just string-check that name:/description: exist.
|
|
function frontmatterBlock(content: string): string {
|
|
expect(content.startsWith('---\n')).toBe(true);
|
|
const end = content.indexOf('\n---', 4);
|
|
expect(end).toBeGreaterThan(0);
|
|
return content.slice(4, end);
|
|
}
|
|
|
|
test('every generated SKILL.md frontmatter parses as strict YAML', () => {
|
|
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
|
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
|
const fm = frontmatterBlock(content);
|
|
let parsed: any;
|
|
expect(() => { parsed = Bun.YAML.parse(fm); },
|
|
`frontmatter for ${skill.dir} must be valid YAML`).not.toThrow();
|
|
expect(typeof parsed?.name).toBe('string');
|
|
expect(typeof parsed?.description).toBe('string');
|
|
}
|
|
});
|
|
|
|
test('every generated Codex (.agents/skills) frontmatter parses as strict YAML', () => {
|
|
// Reads the module-level out-dir render (guaranteed present — the render
|
|
// throws at module load if it fails), never the live gitignored tree.
|
|
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
|
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
|
if (!entry.isDirectory()) continue;
|
|
const mdPath = path.join(agentsDir, entry.name, 'SKILL.md');
|
|
if (!fs.existsSync(mdPath)) continue;
|
|
const fm = frontmatterBlock(fs.readFileSync(mdPath, 'utf-8'));
|
|
expect(() => Bun.YAML.parse(fm),
|
|
`Codex frontmatter for ${entry.name} must be valid YAML`).not.toThrow();
|
|
}
|
|
});
|
|
|
|
test(`every generated SKILL.md description stays within ${MAX_SKILL_DESCRIPTION_LENGTH} chars`, () => {
|
|
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
|
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
|
const description = extractDescription(content);
|
|
expect(description.length).toBeLessThanOrEqual(MAX_SKILL_DESCRIPTION_LENGTH);
|
|
}
|
|
});
|
|
|
|
test(`every Codex SKILL.md description stays within ${MAX_SKILL_DESCRIPTION_LENGTH} chars`, () => {
|
|
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
|
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
|
if (!entry.isDirectory()) continue;
|
|
const skillMd = path.join(agentsDir, entry.name, 'SKILL.md');
|
|
if (!fs.existsSync(skillMd)) continue;
|
|
const content = fs.readFileSync(skillMd, 'utf-8');
|
|
const description = extractDescription(content);
|
|
expect(description.length).toBeLessThanOrEqual(MAX_SKILL_DESCRIPTION_LENGTH);
|
|
}
|
|
});
|
|
|
|
test('every Codex SKILL.md description stays under 900-char warning threshold', () => {
|
|
const WARN_THRESHOLD = 900;
|
|
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
|
const violations: string[] = [];
|
|
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
|
if (!entry.isDirectory()) continue;
|
|
const skillMd = path.join(agentsDir, entry.name, 'SKILL.md');
|
|
if (!fs.existsSync(skillMd)) continue;
|
|
const content = fs.readFileSync(skillMd, 'utf-8');
|
|
const description = extractDescription(content);
|
|
if (description.length > WARN_THRESHOLD) {
|
|
violations.push(`${entry.name}: ${description.length} chars (limit ${MAX_SKILL_DESCRIPTION_LENGTH}, ${MAX_SKILL_DESCRIPTION_LENGTH - description.length} remaining)`);
|
|
}
|
|
}
|
|
expect(violations).toEqual([]);
|
|
});
|
|
|
|
test('package.json version matches VERSION file (npm-valid translation)', () => {
|
|
// Decision 11 (v1.67 wave): VERSION stays the 4-digit source of truth;
|
|
// package.json carries the npm-valid 3-digit translation (npm rejects a
|
|
// fourth component). The pre-v1.67 1:1 four-digit mirror is also accepted
|
|
// (grandfathered until the next write), matching gstack-version-bump's
|
|
// own drift contract.
|
|
const pkg = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf-8'));
|
|
const version = fs.readFileSync(path.join(ROOT, 'VERSION'), 'utf-8').trim();
|
|
const npmTranslation = version.split('.').slice(0, 3).join('.');
|
|
expect([npmTranslation, version]).toContain(pkg.version);
|
|
});
|
|
|
|
test('generated files are fresh (match --dry-run)', () => {
|
|
// Deliberately compares against the LIVE TRACKED SKILL.md files (no
|
|
// --out-dir): this is the freshness gate for the committed tree. Dry-run
|
|
// writes nothing — it is a read.
|
|
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--dry-run'], {
|
|
cwd: ROOT,
|
|
stdout: 'pipe',
|
|
stderr: 'pipe',
|
|
});
|
|
expect(result.exitCode).toBe(0);
|
|
const output = result.stdout.toString();
|
|
// Every skill should be FRESH
|
|
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
|
const file = skill.dir === '.' ? 'SKILL.md' : `${skill.dir}/SKILL.md`;
|
|
expect(output).toContain(`FRESH: ${file}`);
|
|
}
|
|
expect(output).not.toContain('STALE');
|
|
});
|
|
|
|
test('no generated SKILL.md contains unresolved placeholders', () => {
|
|
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
|
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
|
const unresolved = content.match(/\{\{[A-Z_]+\}\}/g);
|
|
expect(unresolved).toBeNull();
|
|
}
|
|
});
|
|
|
|
test('templates contain placeholders', () => {
|
|
// P2 (v1.2.0): the root template is a pure router — only {{PREAMBLE}}.
|
|
// The browse command/snapshot placeholders live in browse/SKILL.md.tmpl now.
|
|
const rootTmpl = fs.readFileSync(path.join(ROOT, 'SKILL.md.tmpl'), 'utf-8');
|
|
expect(rootTmpl).toContain('{{PREAMBLE}}');
|
|
expect(rootTmpl).not.toContain('{{COMMAND_REFERENCE}}');
|
|
expect(rootTmpl).not.toContain('{{SNAPSHOT_FLAGS}}');
|
|
|
|
// Browse carve: the reference resolvers moved into the on-demand section
|
|
// template (so gen-skill-docs keeps them fresh from browse/src); the
|
|
// skeleton points at the section instead of inlining the reference.
|
|
const browseTmpl = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md.tmpl'), 'utf-8');
|
|
expect(browseTmpl).not.toContain('{{COMMAND_REFERENCE}}');
|
|
expect(browseTmpl).not.toContain('{{SNAPSHOT_FLAGS}}');
|
|
expect(browseTmpl).toContain('{{SECTION:command-list}}');
|
|
expect(browseTmpl).toContain('{{PREAMBLE}}');
|
|
|
|
const browseSectionTmpl = fs.readFileSync(
|
|
path.join(ROOT, 'browse', 'sections', 'command-list.md.tmpl'), 'utf-8');
|
|
expect(browseSectionTmpl).toContain('{{COMMAND_REFERENCE}}');
|
|
expect(browseSectionTmpl).toContain('{{SNAPSHOT_FLAGS}}');
|
|
});
|
|
|
|
test('generated SKILL.md contains operational self-improvement (replaced contributor mode)', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
|
expect(content).not.toContain('Contributor Mode');
|
|
expect(content).not.toContain('gstack_contributor');
|
|
expect(content).not.toContain('contributor-logs');
|
|
expect(content).toContain('Operational Self-Improvement');
|
|
expect(content).toContain('gstack-learnings-log');
|
|
// The learnings-resurface call moved from the inline preamble bash into
|
|
// the skill-start script (Phase 1) — same command, new home.
|
|
expect(SKILL_START_SCRIPT).toContain('gstack-learnings-search" --limit 3');
|
|
});
|
|
|
|
test('generated SKILL.md with LEARNINGS_LOG contains operational type', () => {
|
|
// Check a skill that has LEARNINGS_LOG (e.g., review)
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('operational');
|
|
});
|
|
|
|
test('session awareness lives in gstack-skill-start (registry touch + stale cleanup)', () => {
|
|
// The sessions registry moved from inline preamble bash into the script:
|
|
// it records the harness pid (--parent-pid identity) and expires entries
|
|
// older than 120 minutes.
|
|
expect(SKILL_START_SCRIPT).toContain('sessions/$PARENT_PID');
|
|
expect(SKILL_START_SCRIPT).toContain('-mmin +120');
|
|
// The render keeps the completion-status protocol the sessions feed into.
|
|
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('RECOMMENDATION');
|
|
});
|
|
|
|
test('branch detection lives in gstack-skill-start and is echoed as BRANCH', () => {
|
|
expect(SKILL_START_SCRIPT).toContain('_BRANCH=$(git branch --show-current');
|
|
expect(SKILL_START_SCRIPT).toContain('echo "BRANCH: $_BRANCH"');
|
|
});
|
|
|
|
// #2001: update_check: false silences the binary but the upgrade-handling
|
|
// instruction prose used to ship unconditionally. Token-reduction Phase 2
|
|
// made the gate STRUCTURAL: the prose left the renders entirely (absence is
|
|
// pinned by test/onboarding-moved-literals.test.ts) and now emits from
|
|
// gstack-skill-start's instruction layer ONLY when the update-check binary
|
|
// produced output — and that binary silences itself on update_check=false.
|
|
// Opted-out installs can never see the prose, by construction.
|
|
test('update_check opt-out gates the update binary and upgrade-flow emission (issue #2001)', () => {
|
|
// The config-echo cluster lives in gstack-skill-start: the flag is still
|
|
// read and echoed as a STATUS line for the model.
|
|
expect(SKILL_START_SCRIPT, 'script must read update_check config').toContain('_UPDATE_CHECK=$(');
|
|
expect(SKILL_START_SCRIPT, 'script must echo UPDATE_CHECK').toContain('echo "UPDATE_CHECK: $_UPDATE_CHECK"');
|
|
// Gate half 1: the update-check binary exits silently when opted out.
|
|
const updateCheck = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-update-check'), 'utf-8');
|
|
expect(updateCheck, 'binary must read update_check config').toContain('get update_check');
|
|
expect(updateCheck, 'binary must exit silently on update_check=false')
|
|
.toMatch(/if \[ "\$_UC" = "false" \]; then\n\s*exit 0/);
|
|
// Gate half 2: the upgrade-flow instruction block emits only when the
|
|
// binary emitted something (empty when opted out, cached, or up to date).
|
|
expect(SKILL_START_SCRIPT, 'upgrade-flow must be gated on update-check output')
|
|
.toMatch(/if \[ -n "\$_UPD" \]; then\n\s*_emit_block upgrade-flow/);
|
|
});
|
|
|
|
test('tier 2+ skills contain ELI10 simplification rules (AskUserQuestion format)', () => {
|
|
// Root SKILL.md is tier 1 (no AskUserQuestion format). Check a tier 2+ skill instead.
|
|
// v1.7.0.0 Pros/Cons format uses "ELI10 (ALWAYS)" rather than "Simplify (ELI10".
|
|
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('ELI10');
|
|
expect(content).toContain('plain English');
|
|
expect(content).toContain('not function names');
|
|
});
|
|
|
|
test('tier 1 skills do NOT contain AskUserQuestion format', () => {
|
|
// Use benchmark (tier 1) instead of root — root SKILL.md gets overwritten by Codex test setup
|
|
const content = fs.readFileSync(path.join(ROOT, 'benchmark', 'SKILL.md'), 'utf-8');
|
|
expect(content).not.toContain('## AskUserQuestion Format');
|
|
expect(content).not.toContain('## Completeness Principle');
|
|
});
|
|
|
|
test('telemetry producer lives in the scripts; render documents the analytics sink', () => {
|
|
// The skill-usage.jsonl producers moved into the scripts (Phase 1).
|
|
expect(SKILL_START_SCRIPT).toContain('analytics/skill-usage.jsonl');
|
|
expect(SKILL_END_SCRIPT).toContain('analytics/skill-usage.jsonl');
|
|
// The render still tells the model where telemetry lands.
|
|
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('~/.gstack/analytics');
|
|
});
|
|
|
|
test('plan-review generated preambles stay under the Option A budget', () => {
|
|
const reviewSkills = [
|
|
{
|
|
path: path.join(ROOT, 'plan-ceo-review', 'SKILL.md'),
|
|
markers: ['# Mega Plan Review Mode', '## Step 0: Detect platform and base branch'],
|
|
},
|
|
{
|
|
path: path.join(ROOT, 'plan-eng-review', 'SKILL.md'),
|
|
markers: ['# Plan Review Mode'],
|
|
},
|
|
];
|
|
|
|
// Plan skills carry the same preamble surface as other tier-≥2 skills
|
|
// (Artifacts Sync, Context Recovery, Routing Injection are load-bearing
|
|
// functionality, not optional). Budget is set to current size + small
|
|
// headroom; ratchet down if a future slim trims real bytes.
|
|
// Ratcheted from 33000 → 35000 when the gbrain context-load block was
|
|
// added (per /sync-gbrain plan §4). Ratcheted 35000 → 36500 in v1.27.0.0
|
|
// when generate-brain-sync-block.ts gained the gbrain_mcp_mode probe +
|
|
// remote-mode ARTIFACTS_SYNC status line (Path 4 of /setup-gbrain).
|
|
// Ratcheted 36500 → 39000 in the contributor wave when #1205 added the
|
|
// \\u-escape CJK rule (rule 12 + self-check item) to the AskUserQuestion
|
|
// preamble.
|
|
// Ratcheted 39000 → 40000 in plan-tune cathedral T14: question-tuning
|
|
// resolver gained the <gstack-qid:...> marker convention + the
|
|
// (recommended) label requirement (D2 + D18 — both load-bearing for
|
|
// hook enforcement). Adds ~700 bytes.
|
|
// Ratcheted 40000 → 60000 in v1.52.0.0 cap audit: ~20K headroom so
|
|
// future preamble adds don't trip the gate on each PR. Real runaway
|
|
// (preamble doubling) still trips; normal scope growth doesn't.
|
|
for (const skill of reviewSkills) {
|
|
const content = fs.readFileSync(skill.path, 'utf-8');
|
|
const preamble = extractPreambleBeforeWorkflow(content, skill.markers);
|
|
expect(Buffer.byteLength(preamble, 'utf-8')).toBeLessThan(60_000);
|
|
}
|
|
});
|
|
|
|
test('voice and writing-style preamble sections stay compact', () => {
|
|
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
|
const voice = extractMarkdownSection(content, '## Voice');
|
|
const writingStyle = extractMarkdownSection(content, '## Writing Style');
|
|
|
|
expect(Buffer.byteLength(voice, 'utf-8')).toBeLessThan(3_000);
|
|
expect(Buffer.byteLength(writingStyle, 'utf-8')).toBeLessThan(2_000);
|
|
});
|
|
|
|
test('slim voice section preserves the gstack voice contract', () => {
|
|
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
|
const voice = extractMarkdownSection(content, '## Voice');
|
|
|
|
expect(voice).toMatch(/lead with the point|direct/i);
|
|
expect(voice).toMatch(/file|function|line|command|real numbers/i);
|
|
expect(voice).toMatch(/user.*outcome|user.*experience|real user/i);
|
|
expect(voice).toMatch(/corporate|academic|PR|hype/i);
|
|
expect(voice).toMatch(/AI vocabulary|delve|crucial|robust/i);
|
|
expect(voice).toMatch(/user decides|user.*context|sovereignty|recommendation, not a decision/i);
|
|
});
|
|
|
|
test('preamble .pending-* glob is zsh-safe (uses find, not shell glob)', () => {
|
|
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
|
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
|
if (!content.includes('.pending-')) continue;
|
|
// Must NOT have a bare shell glob ".pending-*" outside of find's -name argument
|
|
expect(content).not.toMatch(/for _PF in [^\n]*\/\.pending-\*/);
|
|
// Must use find to avoid zsh NOMATCH error on glob expansion
|
|
expect(content).toContain("find ~/.gstack/analytics -maxdepth 1 -name '.pending-*'");
|
|
}
|
|
});
|
|
|
|
test('bash blocks with shell globs are zsh-safe (setopt guard or find)', () => {
|
|
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
|
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
|
const bashBlocks = [...content.matchAll(/```bash\n([\s\S]*?)```/g)].map(m => m[1]);
|
|
|
|
for (const block of bashBlocks) {
|
|
const lines = block.split('\n');
|
|
|
|
for (const line of lines) {
|
|
const trimmed = line.trimStart();
|
|
if (trimmed.startsWith('#')) continue;
|
|
if (!trimmed.includes('*')) continue;
|
|
// Skip lines where * is inside find -name, git pathspecs, or $(find)
|
|
if (/\bfind\b/.test(trimmed)) continue;
|
|
if (/\bgit\b/.test(trimmed)) continue;
|
|
if (/\$\(find\b/.test(trimmed)) continue;
|
|
|
|
// Check 1: "for VAR in <glob>" must use $(find ...) — caught above by the
|
|
// $(find check, so any surviving for-in with a glob pattern is a violation
|
|
if (/\bfor\s+\w+\s+in\b/.test(trimmed) && /\*\./.test(trimmed)) {
|
|
throw new Error(
|
|
`Unsafe for-in glob in ${skill.dir}/SKILL.md: "${trimmed}". ` +
|
|
`Use \`for f in $(find ... -name '*.ext')\` for zsh compatibility.`
|
|
);
|
|
}
|
|
|
|
// Check 2: ls/cat/rm/grep with glob file args must have setopt guard
|
|
const isGlobCmd = /\b(?:ls|cat|rm|grep)\b/.test(trimmed) &&
|
|
/(?:\/\*[a-z.*]|\*\.[a-z])/.test(trimmed);
|
|
if (isGlobCmd) {
|
|
expect(block).toContain('setopt +o nomatch');
|
|
}
|
|
}
|
|
}
|
|
}
|
|
});
|
|
|
|
test('preamble-using skills have correct skill name in telemetry', () => {
|
|
const PREAMBLE_SKILLS = [
|
|
{ dir: '.', name: 'gstack' },
|
|
{ dir: 'ship', name: 'ship' },
|
|
{ dir: 'review', name: 'review' },
|
|
{ dir: 'qa', name: 'qa' },
|
|
{ dir: 'retro', name: 'retro' },
|
|
];
|
|
for (const skill of PREAMBLE_SKILLS) {
|
|
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
|
// The skill name now travels as --skill into gstack-skill-start (the
|
|
// preamble fence) and gstack-skill-end (the telemetry epilogue) — the
|
|
// scripts write it into the JSONL events.
|
|
expect(content, `${skill.dir} preamble fence must pass its own name`)
|
|
.toMatch(new RegExp(`--skill "${skill.name}" --model`));
|
|
expect(content, `${skill.dir} epilogue must pass its own name`)
|
|
.toContain(`gstack-skill-end --skill "${skill.name}"`);
|
|
}
|
|
});
|
|
|
|
test('qa and qa-only templates use QA_METHODOLOGY placeholder', () => {
|
|
// qa carve: the macro moved into the section template (the skeleton
|
|
// carries the STOP-Read pointer); qa-only remains an inline monolith.
|
|
const qaSkeletonTmpl = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md.tmpl'), 'utf-8');
|
|
expect(qaSkeletonTmpl).toContain('{{SECTION:qa-patterns}}');
|
|
expect(qaSkeletonTmpl).not.toContain('{{QA_METHODOLOGY}}');
|
|
const qaSectionTmpl = fs.readFileSync(path.join(ROOT, 'qa', 'sections', 'qa-patterns.md.tmpl'), 'utf-8');
|
|
expect(qaSectionTmpl).toContain('{{QA_METHODOLOGY}}');
|
|
|
|
const qaOnlyTmpl = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md.tmpl'), 'utf-8');
|
|
expect(qaOnlyTmpl).toContain('{{QA_METHODOLOGY}}');
|
|
});
|
|
|
|
test('QA_METHODOLOGY appears expanded in both qa and qa-only generated files', () => {
|
|
const qaContent = readSkillUnion('qa'); // carved: methodology lives in qa/sections/qa-patterns.md
|
|
const qaOnlyContent = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md'), 'utf-8');
|
|
|
|
// Both should contain the health score rubric
|
|
expect(qaContent).toContain('Health Score Rubric');
|
|
expect(qaOnlyContent).toContain('Health Score Rubric');
|
|
|
|
// Both should contain framework guidance
|
|
expect(qaContent).toContain('Framework-Specific Guidance');
|
|
expect(qaOnlyContent).toContain('Framework-Specific Guidance');
|
|
|
|
// Both should contain the important rules
|
|
expect(qaContent).toContain('Important Rules');
|
|
expect(qaOnlyContent).toContain('Important Rules');
|
|
|
|
// Both should contain the 6 phases
|
|
expect(qaContent).toContain('Phase 1');
|
|
expect(qaOnlyContent).toContain('Phase 1');
|
|
expect(qaContent).toContain('Phase 6');
|
|
expect(qaOnlyContent).toContain('Phase 6');
|
|
});
|
|
|
|
test('qa-only has no-fix guardrails', () => {
|
|
const qaOnlyContent = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md'), 'utf-8');
|
|
expect(qaOnlyContent).toContain('Never fix bugs');
|
|
expect(qaOnlyContent).toContain('NEVER fix anything');
|
|
// Should not have Edit, Glob, or Grep in allowed-tools.
|
|
// Scope to frontmatter (between the first two --- lines) — the body can
|
|
// legitimately mention these tool names in prose (e.g., Claude model
|
|
// overlay says "prefer Read, Edit, Write, Glob, Grep over Bash").
|
|
const fmMatch = qaOnlyContent.match(/^---\n([\s\S]*?)\n---/);
|
|
expect(fmMatch).not.toBeNull();
|
|
const frontmatter = fmMatch![1];
|
|
expect(frontmatter).toMatch(/allowed-tools:/);
|
|
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Edit/);
|
|
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Glob/);
|
|
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Grep/);
|
|
});
|
|
|
|
test('qa has fix-loop tools and phases', () => {
|
|
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
|
|
// Should have Edit, Glob, Grep in allowed-tools
|
|
expect(qaContent).toContain('Edit');
|
|
expect(qaContent).toContain('Glob');
|
|
expect(qaContent).toContain('Grep');
|
|
// Should have fix-loop phases
|
|
expect(qaContent).toContain('Phase 7');
|
|
expect(qaContent).toContain('Phase 8');
|
|
expect(qaContent).toContain('Fix Loop');
|
|
expect(qaContent).toContain('Triage');
|
|
expect(qaContent).toContain('WTF');
|
|
});
|
|
});
|
|
|
|
describe('BASE_BRANCH_DETECT resolver', () => {
|
|
// Find a generated SKILL.md that uses the placeholder (ship is guaranteed to)
|
|
const shipContent = readShipUnion();
|
|
|
|
test('resolver output contains PR base detection command', () => {
|
|
expect(shipContent).toContain('gh pr view --json baseRefName');
|
|
});
|
|
|
|
test('resolver output contains repo default branch detection command', () => {
|
|
expect(shipContent).toContain('gh repo view --json defaultBranchRef');
|
|
});
|
|
|
|
test('resolver output contains fallback to main', () => {
|
|
expect(shipContent).toMatch(/fall\s*back\s+to\s+`main`/i);
|
|
});
|
|
|
|
test('resolver output uses "the base branch" phrasing', () => {
|
|
expect(shipContent).toContain('the base branch');
|
|
});
|
|
|
|
test('resolver output contains GitLab CLI commands', () => {
|
|
expect(shipContent).toContain('glab');
|
|
});
|
|
|
|
test('resolver output contains git-native fallback', () => {
|
|
expect(shipContent).toContain('git symbolic-ref');
|
|
});
|
|
|
|
test('resolver output mentions GitLab platform', () => {
|
|
expect(shipContent).toMatch(/gitlab/i);
|
|
});
|
|
});
|
|
|
|
describe('GitLab support in generated skills', () => {
|
|
const retroContent = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
|
|
const shipSkillContent = readShipUnion();
|
|
|
|
test('retro contains GitLab MR number extraction', () => {
|
|
expect(retroContent).toContain('[#!]');
|
|
});
|
|
|
|
test('retro uses BASE_BRANCH_DETECT (contains glab)', () => {
|
|
expect(retroContent).toContain('glab');
|
|
});
|
|
|
|
test('ship contains glab mr create', () => {
|
|
expect(shipSkillContent).toContain('glab mr create');
|
|
});
|
|
|
|
test('ship checks .gitlab-ci.yml', () => {
|
|
expect(shipSkillContent).toContain('.gitlab-ci.yml');
|
|
});
|
|
});
|
|
|
|
/**
|
|
* Quality evals — catch description regressions.
|
|
*
|
|
* These test that generated output is *useful for an AI agent*,
|
|
* not just structurally valid. Each test targets a specific
|
|
* regression we actually shipped and caught in review.
|
|
*/
|
|
describe('description quality evals', () => {
|
|
// Regression: snapshot flags lost value hints (-d <N>, -s <sel>, -o <path>)
|
|
// Browse carve: the flag reference renders into browse/sections/command-list.md.
|
|
test('snapshot flags with values include value hints in output', () => {
|
|
const content = readSkillUnion('browse');
|
|
for (const flag of SNAPSHOT_FLAGS) {
|
|
if (flag.takesValue) {
|
|
expect(flag.valueHint).toBeDefined();
|
|
expect(content).toContain(`${flag.short} ${flag.valueHint}`);
|
|
}
|
|
}
|
|
});
|
|
|
|
// Regression: "is" lost the valid states enum
|
|
test('is command lists valid state values', () => {
|
|
const desc = COMMAND_DESCRIPTIONS['is'].description;
|
|
for (const state of ['visible', 'hidden', 'enabled', 'disabled', 'checked', 'editable', 'focused']) {
|
|
expect(desc).toContain(state);
|
|
}
|
|
});
|
|
|
|
// Regression: "press" lost common key examples
|
|
test('press command lists example keys', () => {
|
|
const desc = COMMAND_DESCRIPTIONS['press'].description;
|
|
expect(desc).toContain('Enter');
|
|
expect(desc).toContain('Tab');
|
|
expect(desc).toContain('Escape');
|
|
});
|
|
|
|
// Regression: "console" lost --errors filter note
|
|
test('console command describes --errors behavior', () => {
|
|
const desc = COMMAND_DESCRIPTIONS['console'].description;
|
|
expect(desc).toContain('--errors');
|
|
});
|
|
|
|
// Regression: snapshot -i lost "@e refs" context
|
|
test('snapshot -i mentions @e refs', () => {
|
|
const flag = SNAPSHOT_FLAGS.find(f => f.short === '-i')!;
|
|
expect(flag.description).toContain('@e');
|
|
});
|
|
|
|
// Regression: snapshot -C lost "@c refs" context
|
|
test('snapshot -C mentions @c refs', () => {
|
|
const flag = SNAPSHOT_FLAGS.find(f => f.short === '-C')!;
|
|
expect(flag.description).toContain('@c');
|
|
});
|
|
|
|
// Guard: every description must be at least 8 chars (catches empty or stub descriptions)
|
|
test('all command descriptions have meaningful length', () => {
|
|
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
|
expect(meta.description.length).toBeGreaterThanOrEqual(8);
|
|
}
|
|
});
|
|
|
|
// Guard: snapshot flag descriptions must be at least 10 chars
|
|
test('all snapshot flag descriptions have meaningful length', () => {
|
|
for (const flag of SNAPSHOT_FLAGS) {
|
|
expect(flag.description.length).toBeGreaterThanOrEqual(10);
|
|
}
|
|
});
|
|
|
|
// Guard: descriptions must not contain pipe (breaks markdown table cells)
|
|
// Usage strings are backtick-wrapped in the table so pipes there are safe.
|
|
test('no command description contains pipe character', () => {
|
|
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
|
expect(meta.description).not.toContain('|');
|
|
}
|
|
});
|
|
|
|
// Guard: generated output uses → not ->
|
|
test('generated SKILL.md uses unicode arrows', () => {
|
|
// P2 (v1.2.0): the browse body moved out of the top-level router into
|
|
// browse/SKILL.md. Guard arrow style on the browse body (sliced from its
|
|
// H1 so the auto-generated `-->` header comments are excluded).
|
|
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
|
const body = content.slice(content.indexOf('# browse: QA Testing'));
|
|
expect(body).toContain('→');
|
|
expect(body).not.toContain('->');
|
|
});
|
|
});
|
|
|
|
describe('REVIEW_DASHBOARD resolver', () => {
|
|
const REVIEW_SKILLS = ['plan-ceo-review', 'plan-eng-review', 'plan-design-review'];
|
|
|
|
for (const skill of REVIEW_SKILLS) {
|
|
test(`review dashboard appears in ${skill} generated file`, () => {
|
|
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
|
expect(content).toContain('gstack-review');
|
|
expect(content).toContain('REVIEW READINESS DASHBOARD');
|
|
});
|
|
}
|
|
|
|
test('review dashboard appears in ship generated file', () => {
|
|
const content = readShipUnion();
|
|
expect(content).toContain('reviews.jsonl');
|
|
expect(content).toContain('REVIEW READINESS DASHBOARD');
|
|
});
|
|
|
|
test('dashboard treats review as a valid Eng Review source', () => {
|
|
const content = readShipUnion();
|
|
expect(content).toContain('plan-eng-review, review, plan-design-review');
|
|
expect(content).toContain('`review` (diff-scoped pre-landing review)');
|
|
expect(content).toContain('`plan-eng-review` (plan-stage architecture review)');
|
|
expect(content).toContain('from either \\`review\\` or \\`plan-eng-review\\`');
|
|
});
|
|
|
|
test('shared dashboard propagates review source to plan-eng-review', () => {
|
|
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
|
expect(content).toContain('plan-eng-review, review, plan-design-review');
|
|
expect(content).toContain('`review` (diff-scoped pre-landing review)');
|
|
});
|
|
|
|
test('resolver output contains key dashboard elements', () => {
|
|
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
|
expect(content).toContain('VERDICT');
|
|
expect(content).toContain('CLEARED');
|
|
expect(content).toContain('Eng Review');
|
|
expect(content).toContain('7 days');
|
|
expect(content).toContain('Design Review');
|
|
expect(content).toContain('skip_eng_review');
|
|
});
|
|
|
|
test('dashboard bash block includes git HEAD for staleness detection', () => {
|
|
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
|
expect(content).toContain('git rev-parse --short HEAD');
|
|
expect(content).toContain('---HEAD---');
|
|
});
|
|
|
|
test('dashboard includes staleness detection prose', () => {
|
|
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
|
expect(content).toContain('Staleness detection');
|
|
expect(content).toContain('commit');
|
|
});
|
|
|
|
for (const skill of REVIEW_SKILLS) {
|
|
test(`${skill} contains review chaining section`, () => {
|
|
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
|
expect(content).toContain('Review Chaining');
|
|
});
|
|
|
|
test(`${skill} Review Log includes commit field`, () => {
|
|
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
|
expect(content).toContain('"commit"');
|
|
});
|
|
}
|
|
|
|
test('plan-ceo-review chaining mentions eng and design reviews', () => {
|
|
// Carved skill: the chaining prose lives in sections/*.md. (It used to
|
|
// pass against the skeleton only because the preamble's routing-injection
|
|
// rules incidentally named these skills — that prose moved into
|
|
// bin/gstack-skill-start in token-reduction Phase 2.)
|
|
const content = readSkillUnion('plan-ceo-review');
|
|
expect(content).toContain('/plan-eng-review');
|
|
expect(content).toContain('/plan-design-review');
|
|
});
|
|
|
|
test('plan-eng-review chaining mentions design and ceo reviews', () => {
|
|
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
|
expect(content).toContain('/plan-design-review');
|
|
expect(content).toContain('/plan-ceo-review');
|
|
});
|
|
|
|
test('plan-design-review chaining mentions eng, ceo, and design skills', () => {
|
|
const content = readSkillUnion('plan-design-review');
|
|
expect(content).toContain('/plan-eng-review');
|
|
expect(content).toContain('/plan-ceo-review');
|
|
expect(content).toContain('/design-shotgun');
|
|
expect(content).toContain('/design-html');
|
|
});
|
|
|
|
test('ship does NOT contain review chaining', () => {
|
|
const content = readShipUnion();
|
|
expect(content).not.toContain('Review Chaining');
|
|
});
|
|
});
|
|
|
|
// ─── Test Coverage Audit Resolver Tests ─────────────────────
|
|
|
|
describe('TEST_COVERAGE_AUDIT placeholders', () => {
|
|
const planSkill = readSkillUnion('plan-eng-review'); // carved
|
|
const shipSkill = readShipUnion();
|
|
const reviewSkill = readSkillUnion('review'); // carved: Review Army moved to sections/review-army.md
|
|
|
|
test('plan and ship modes share codepath tracing methodology', () => {
|
|
// Review mode delegates test coverage to the Testing specialist subagent (Review Army)
|
|
const sharedPhrases = [
|
|
'Trace data flow',
|
|
'Diagram the execution',
|
|
'Quality scoring rubric',
|
|
'★★★',
|
|
'★★',
|
|
'GAP',
|
|
];
|
|
for (const phrase of sharedPhrases) {
|
|
expect(planSkill).toContain(phrase);
|
|
expect(shipSkill).toContain(phrase);
|
|
}
|
|
// Plan mode traces the plan, not a git diff
|
|
expect(planSkill).toContain('Trace every codepath in the plan');
|
|
expect(planSkill).not.toContain('git diff origin');
|
|
// Ship mode traces the diff
|
|
expect(shipSkill).toContain('Trace every codepath changed');
|
|
});
|
|
|
|
test('review mode uses Review Army for specialist dispatch', () => {
|
|
expect(reviewSkill).toContain('Review Army');
|
|
expect(reviewSkill).toContain('Specialist Dispatch');
|
|
expect(reviewSkill).toContain('testing.md');
|
|
});
|
|
|
|
test('plan and ship modes include E2E decision matrix', () => {
|
|
// Review mode delegates to Testing specialist
|
|
for (const skill of [planSkill, shipSkill]) {
|
|
expect(skill).toContain('E2E Test Decision Matrix');
|
|
expect(skill).toContain('→E2E');
|
|
expect(skill).toContain('→EVAL');
|
|
}
|
|
});
|
|
|
|
test('plan and ship modes include regression rule', () => {
|
|
// Review mode delegates to Testing specialist
|
|
for (const skill of [planSkill, shipSkill]) {
|
|
expect(skill).toContain('REGRESSION RULE');
|
|
expect(skill).toContain('IRON RULE');
|
|
}
|
|
});
|
|
|
|
test('plan and ship modes include test framework detection', () => {
|
|
// Review mode delegates to Testing specialist
|
|
for (const skill of [planSkill, shipSkill]) {
|
|
expect(skill).toContain('Test Framework Detection');
|
|
expect(skill).toContain('CLAUDE.md');
|
|
}
|
|
});
|
|
|
|
test('plan mode adds tests to plan + includes test plan artifact', () => {
|
|
expect(planSkill).toContain('Add missing tests to the plan');
|
|
expect(planSkill).toContain('eng-review-test-plan');
|
|
expect(planSkill).toContain('Test Plan Artifact');
|
|
});
|
|
|
|
test('ship mode auto-generates tests + includes before/after count', () => {
|
|
expect(shipSkill).toContain('Generate tests for uncovered paths');
|
|
expect(shipSkill).toContain('Before/after test count');
|
|
expect(shipSkill).toContain('30 code paths max');
|
|
expect(shipSkill).toContain('ship-test-plan');
|
|
});
|
|
|
|
test('review mode uses Fix-First + Review Army for specialist coverage', () => {
|
|
expect(reviewSkill).toContain('Fix-First');
|
|
expect(reviewSkill).toContain('INFORMATIONAL');
|
|
// Review Army handles test coverage via Testing specialist subagent
|
|
expect(reviewSkill).toContain('Review Army');
|
|
expect(reviewSkill).toContain('Testing');
|
|
});
|
|
|
|
test('plan mode does NOT include ship-specific content', () => {
|
|
expect(planSkill).not.toContain('Before/after test count');
|
|
expect(planSkill).not.toContain('30 code paths max');
|
|
expect(planSkill).not.toContain('ship-test-plan');
|
|
});
|
|
|
|
test('review mode does NOT include test plan artifact', () => {
|
|
expect(reviewSkill).not.toContain('Test Plan Artifact');
|
|
expect(reviewSkill).not.toContain('eng-review-test-plan');
|
|
expect(reviewSkill).not.toContain('ship-test-plan');
|
|
});
|
|
|
|
test('review/specialists/ directory has all expected checklist files', () => {
|
|
const specDir = path.join(ROOT, 'review', 'specialists');
|
|
const expected = [
|
|
'testing.md',
|
|
'maintainability.md',
|
|
'security.md',
|
|
'performance.md',
|
|
'data-migration.md',
|
|
'api-contract.md',
|
|
'red-team.md',
|
|
];
|
|
for (const f of expected) {
|
|
expect(fs.existsSync(path.join(specDir, f))).toBe(true);
|
|
}
|
|
});
|
|
|
|
test('each specialist file has standard header with scope and output format', () => {
|
|
const specDir = path.join(ROOT, 'review', 'specialists');
|
|
const files = fs.readdirSync(specDir).filter(f => f.endsWith('.md'));
|
|
for (const f of files) {
|
|
const content = fs.readFileSync(path.join(specDir, f), 'utf-8');
|
|
// All specialist files must have Scope and Output/JSON in header
|
|
expect(content).toContain('Scope:');
|
|
expect(content.toLowerCase()).toMatch(/output|json/);
|
|
// Must define NO FINDINGS behavior
|
|
expect(content).toContain('NO FINDINGS');
|
|
}
|
|
});
|
|
|
|
// Regression guard: ship output contains key phrases from before the refactor
|
|
test('ship SKILL.md regression guard — key phrases preserved', () => {
|
|
const regressionPhrases = [
|
|
'100% coverage is the goal',
|
|
'ASCII coverage diagram',
|
|
'processPayment',
|
|
'refundPayment',
|
|
'billing.test.ts',
|
|
'checkout.e2e.ts',
|
|
'COVERAGE:',
|
|
'QUALITY:',
|
|
'GAPS:',
|
|
'Code paths:',
|
|
'User flows:',
|
|
];
|
|
for (const phrase of regressionPhrases) {
|
|
expect(shipSkill).toContain(phrase);
|
|
}
|
|
});
|
|
|
|
test('ship SKILL.md contains review army specialist dispatch', () => {
|
|
expect(shipSkill).toContain('Specialist Dispatch');
|
|
expect(shipSkill).toContain('Step 9.1');
|
|
expect(shipSkill).toContain('Step 9.2');
|
|
});
|
|
|
|
test('ship SKILL.md contains cross-review finding dedup', () => {
|
|
expect(shipSkill).toContain('Cross-review finding dedup');
|
|
expect(shipSkill).toContain('Step 9.3');
|
|
});
|
|
|
|
test('ship SKILL.md contains re-run idempotency behavior', () => {
|
|
expect(shipSkill).toContain('Re-run behavior (idempotency)');
|
|
expect(shipSkill).toContain('Never skip a verification step');
|
|
});
|
|
});
|
|
|
|
// --- {{TEST_FAILURE_TRIAGE}} resolver tests ---
|
|
|
|
describe('TEST_FAILURE_TRIAGE resolver', () => {
|
|
const shipSkill = readShipUnion();
|
|
|
|
test('contains all 4 triage steps', () => {
|
|
expect(shipSkill).toContain('Step T1: Classify each failure');
|
|
expect(shipSkill).toContain('Step T2: Handle in-branch failures');
|
|
expect(shipSkill).toContain('Step T3: Handle pre-existing failures');
|
|
expect(shipSkill).toContain('Step T4: Execute the chosen action');
|
|
});
|
|
|
|
test('T1 includes classification criteria (in-branch vs pre-existing)', () => {
|
|
expect(shipSkill).toContain('In-branch');
|
|
expect(shipSkill).toContain('Likely pre-existing');
|
|
expect(shipSkill).toContain('git diff origin/');
|
|
});
|
|
|
|
test('T3 branches on REPO_MODE (solo vs collaborative)', () => {
|
|
expect(shipSkill).toContain('REPO_MODE');
|
|
expect(shipSkill).toContain('solo');
|
|
expect(shipSkill).toContain('collaborative');
|
|
});
|
|
|
|
test('solo mode offers fix-now, TODO, and skip options', () => {
|
|
expect(shipSkill).toContain('Investigate and fix now');
|
|
expect(shipSkill).toContain('Add as P0 TODO');
|
|
expect(shipSkill).toContain('Skip');
|
|
});
|
|
|
|
test('collaborative mode offers blame + assign option', () => {
|
|
expect(shipSkill).toContain('Blame + assign GitHub issue');
|
|
expect(shipSkill).toContain('gh issue create');
|
|
});
|
|
|
|
test('defaults ambiguous failures to in-branch (safety)', () => {
|
|
expect(shipSkill).toContain('When ambiguous, default to in-branch');
|
|
});
|
|
});
|
|
|
|
// --- {{PLAN_FILE_REVIEW_REPORT}} resolver tests ---
|
|
|
|
describe('PLAN_FILE_REVIEW_REPORT resolver', () => {
|
|
const REVIEW_SKILLS = ['plan-ceo-review', 'plan-eng-review', 'plan-design-review', 'codex'];
|
|
|
|
for (const skill of REVIEW_SKILLS) {
|
|
test(`plan file review report appears in ${skill} generated file`, () => {
|
|
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('GSTACK REVIEW REPORT');
|
|
});
|
|
}
|
|
|
|
test('resolver output contains key report elements', () => {
|
|
const content = readSkillUnion('plan-ceo-review'); // carved: report writer moved to section
|
|
expect(content).toContain('Trigger');
|
|
expect(content).toContain('Findings');
|
|
expect(content).toContain('VERDICT');
|
|
expect(content).toContain('/plan-ceo-review');
|
|
expect(content).toContain('/plan-eng-review');
|
|
expect(content).toContain('/plan-design-review');
|
|
expect(content).toContain('/codex review');
|
|
});
|
|
});
|
|
|
|
// --- {{PLAN_COMPLETION_AUDIT}} resolver tests ---
|
|
|
|
describe('PLAN_COMPLETION_AUDIT placeholders', () => {
|
|
const shipSkill = readShipUnion();
|
|
const reviewSkill = readSkillUnion('review'); // carved: plan-completion audit moved to sections/plan-completion.md
|
|
|
|
test('ship SKILL.md contains plan completion audit step', () => {
|
|
expect(shipSkill).toContain('Plan Completion Audit');
|
|
expect(shipSkill).toContain('Step 8');
|
|
});
|
|
|
|
test('review SKILL.md contains plan completion in scope drift', () => {
|
|
expect(reviewSkill).toContain('Plan File Discovery');
|
|
expect(reviewSkill).toContain('Actionable Item Extraction');
|
|
expect(reviewSkill).toContain('Integration with Scope Drift Detection');
|
|
});
|
|
|
|
test('both modes share plan file discovery methodology', () => {
|
|
expect(shipSkill).toContain('Plan File Discovery');
|
|
expect(reviewSkill).toContain('Plan File Discovery');
|
|
// Both should have conversation context first
|
|
expect(shipSkill).toContain('Conversation context (primary)');
|
|
expect(reviewSkill).toContain('Conversation context (primary)');
|
|
// Both should have grep fallback
|
|
expect(shipSkill).toContain('Content-based search (fallback)');
|
|
expect(reviewSkill).toContain('Content-based search (fallback)');
|
|
});
|
|
|
|
test('ship mode has gate logic for NOT DONE items', () => {
|
|
expect(shipSkill).toContain('NOT DONE');
|
|
expect(shipSkill).toContain('Stop — implement the missing items');
|
|
expect(shipSkill).toContain('Ship anyway — defer');
|
|
expect(shipSkill).toContain('intentionally dropped');
|
|
});
|
|
|
|
test('review mode is INFORMATIONAL only', () => {
|
|
expect(reviewSkill).toContain('INFORMATIONAL');
|
|
expect(reviewSkill).toContain('MISSING REQUIREMENTS');
|
|
expect(reviewSkill).toContain('SCOPE CREEP');
|
|
});
|
|
|
|
test('item extraction has 50-item cap', () => {
|
|
expect(shipSkill).toContain('at most 50 items');
|
|
});
|
|
|
|
test('uses file-level traceability (not commit-level)', () => {
|
|
expect(shipSkill).toContain('Cite the specific file');
|
|
expect(shipSkill).not.toContain('commit-level traceability');
|
|
});
|
|
});
|
|
|
|
// --- {{PLAN_VERIFICATION_EXEC}} resolver tests ---
|
|
|
|
describe('PLAN_VERIFICATION_EXEC placeholder', () => {
|
|
const shipSkill = readShipUnion();
|
|
|
|
test('ship SKILL.md contains plan verification step', () => {
|
|
expect(shipSkill).toContain('Step 8.1');
|
|
expect(shipSkill).toContain('Plan Verification');
|
|
});
|
|
|
|
test('references /qa-only invocation', () => {
|
|
expect(shipSkill).toContain('qa-only/SKILL.md');
|
|
expect(shipSkill).toContain('qa-only');
|
|
});
|
|
|
|
test('contains dev-server discovery (CLAUDE.md first, then a port probe)', () => {
|
|
// Fork port wave 2: the hardcoded 4-port list became read-CLAUDE.md-or-
|
|
// probe; the probe loops common ports instead of naming each once.
|
|
expect(shipSkill).toContain('CLAUDE.md first');
|
|
expect(shipSkill).toContain('http://localhost:$_p');
|
|
expect(shipSkill).toContain('NO_SERVER');
|
|
});
|
|
|
|
test('skips gracefully when no verification section', () => {
|
|
expect(shipSkill).toContain('No verification steps found in plan');
|
|
});
|
|
|
|
test('skips gracefully when no dev server', () => {
|
|
expect(shipSkill).toContain('No dev server detected');
|
|
});
|
|
});
|
|
|
|
// --- Coverage gate tests ---
|
|
|
|
describe('Coverage gate in ship', () => {
|
|
const shipSkill = readShipUnion();
|
|
const reviewSkill = readSkillUnion('review'); // carved: testing.md specialist ref lives in sections/review-army.md
|
|
|
|
test('ship SKILL.md contains coverage gate with thresholds', () => {
|
|
expect(shipSkill).toContain('Coverage gate');
|
|
expect(shipSkill).toContain('>= target');
|
|
expect(shipSkill).toContain('< minimum');
|
|
});
|
|
|
|
test('ship SKILL.md supports configurable thresholds via CLAUDE.md', () => {
|
|
expect(shipSkill).toContain('## Test Coverage');
|
|
expect(shipSkill).toContain('Minimum:');
|
|
expect(shipSkill).toContain('Target:');
|
|
});
|
|
|
|
test('coverage gate skips on parse failure (not block)', () => {
|
|
expect(shipSkill).toContain('could not determine percentage — skipping');
|
|
});
|
|
|
|
test('review SKILL.md delegates coverage to Testing specialist', () => {
|
|
// Coverage audit moved to Testing specialist subagent in Review Army
|
|
expect(reviewSkill).toContain('testing.md');
|
|
expect(reviewSkill).toContain('INFORMATIONAL');
|
|
});
|
|
});
|
|
|
|
// --- Ship metrics logging ---
|
|
|
|
describe('Ship metrics logging', () => {
|
|
const shipSkill = readShipUnion();
|
|
|
|
test('ship SKILL.md contains metrics persistence step', () => {
|
|
expect(shipSkill).toContain('Step 20');
|
|
expect(shipSkill).toContain('coverage_pct');
|
|
expect(shipSkill).toContain('plan_items_total');
|
|
expect(shipSkill).toContain('plan_items_done');
|
|
expect(shipSkill).toContain('verification_result');
|
|
});
|
|
});
|
|
|
|
// --- Plan file discovery shared helper ---
|
|
|
|
describe('Plan file discovery shared helper', () => {
|
|
// The shared helper should appear in ship (via PLAN_COMPLETION_AUDIT_SHIP)
|
|
// and in review (via PLAN_COMPLETION_AUDIT_REVIEW)
|
|
const shipSkill = readShipUnion();
|
|
const reviewSkill = readSkillUnion('review'); // carved: plan-completion audit moved to sections/plan-completion.md
|
|
|
|
test('plan file discovery appears in both ship and review', () => {
|
|
expect(shipSkill).toContain('Plan File Discovery');
|
|
expect(reviewSkill).toContain('Plan File Discovery');
|
|
});
|
|
|
|
test('both include conversation context first', () => {
|
|
expect(shipSkill).toContain('Conversation context (primary)');
|
|
expect(reviewSkill).toContain('Conversation context (primary)');
|
|
});
|
|
|
|
test('both include content-based fallback', () => {
|
|
expect(shipSkill).toContain('Content-based search (fallback)');
|
|
expect(reviewSkill).toContain('Content-based search (fallback)');
|
|
});
|
|
});
|
|
|
|
// --- Retro plan completion ---
|
|
|
|
describe('Retro plan completion section', () => {
|
|
// Carved: the narrative report format (incl. Plan Completion) lives in
|
|
// retro/sections/report-format.md — read the skeleton+sections union.
|
|
const retroSkill = readSkillUnion('retro');
|
|
|
|
test('retro SKILL.md contains plan completion section', () => {
|
|
expect(retroSkill).toContain('### Plan Completion');
|
|
expect(retroSkill).toContain('plan_items_total');
|
|
expect(retroSkill).toContain('Plan Completion This Period');
|
|
});
|
|
});
|
|
|
|
// --- Plan status footer in preamble ---
|
|
|
|
describe('Plan status footer in preamble', () => {
|
|
test('preamble contains plan status footer as neutral forward reference to EXIT PLAN MODE GATE', () => {
|
|
// Read any skill that uses PREAMBLE
|
|
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
|
expect(content).toContain('Plan Status Footer');
|
|
expect(content).toContain('GSTACK REVIEW REPORT');
|
|
expect(content).toContain('ExitPlanMode');
|
|
expect(content).toContain('EXIT PLAN MODE GATE');
|
|
// The preamble must NOT impose review-report rules on operational skills
|
|
// that have no review report. It's a forward reference, not enforcement.
|
|
expect(content).not.toContain('NO REVIEWS YET');
|
|
});
|
|
});
|
|
|
|
// --- make-pdf setup ordering ---
|
|
|
|
describe('make-pdf setup ordering', () => {
|
|
test('MAKE-PDF SETUP appears before generic preamble footer sections', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'make-pdf', 'SKILL.md'), 'utf-8');
|
|
const preambleIdx = content.indexOf('## Preamble (run first)');
|
|
const setupIdx = content.indexOf('## MAKE-PDF SETUP');
|
|
const planModeIdx = content.indexOf('## Plan Mode Safe Operations');
|
|
const telemetryIdx = content.indexOf('## Telemetry (run last)');
|
|
const workflowIdx = content.indexOf('# make-pdf: publication-quality PDFs from markdown');
|
|
|
|
expect(preambleIdx).toBeGreaterThanOrEqual(0);
|
|
expect(setupIdx).toBeGreaterThan(preambleIdx);
|
|
expect(setupIdx).toBeLessThan(planModeIdx);
|
|
expect(setupIdx).toBeLessThan(telemetryIdx);
|
|
expect(setupIdx).toBeLessThan(workflowIdx);
|
|
expect(content.match(/^## MAKE-PDF SETUP/gm)?.length ?? 0).toBe(1);
|
|
});
|
|
});
|
|
|
|
// --- Skill invocation during plan mode in preamble ---
|
|
|
|
describe('Skill invocation during plan mode in preamble', () => {
|
|
test('preamble contains skill invocation plan mode section', () => {
|
|
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
|
expect(content).toContain('Skill Invocation During Plan Mode');
|
|
expect(content).toContain('precedence over generic plan mode behavior');
|
|
expect(content).toContain('Do not continue the workflow');
|
|
expect(content).toContain('cancel the skill or leave plan mode');
|
|
});
|
|
});
|
|
|
|
// --- {{SPEC_REVIEW_LOOP}} resolver tests ---
|
|
|
|
describe('SPEC_REVIEW_LOOP resolver', () => {
|
|
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
|
|
|
test('contains all 5 review dimensions', () => {
|
|
for (const dim of ['Completeness', 'Consistency', 'Clarity', 'Scope', 'Feasibility']) {
|
|
expect(content).toContain(dim);
|
|
}
|
|
});
|
|
|
|
test('references Agent tool for subagent dispatch', () => {
|
|
expect(content).toMatch(/Agent.*tool/i);
|
|
});
|
|
|
|
test('specifies max 3 iterations', () => {
|
|
expect(content).toMatch(/3.*iteration|maximum.*3/i);
|
|
});
|
|
|
|
test('includes quality score', () => {
|
|
expect(content).toContain('quality score');
|
|
});
|
|
|
|
test('includes metrics path', () => {
|
|
expect(content).toContain('spec-review.jsonl');
|
|
});
|
|
|
|
test('includes convergence guard', () => {
|
|
expect(content).toMatch(/[Cc]onvergence/);
|
|
});
|
|
|
|
test('includes graceful failure handling', () => {
|
|
expect(content).toMatch(/skip.*review|unavailable/i);
|
|
});
|
|
});
|
|
|
|
// --- {{DESIGN_SKETCH}} resolver tests ---
|
|
|
|
describe('DESIGN_SKETCH resolver', () => {
|
|
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
|
|
|
test('references DESIGN.md for design system constraints', () => {
|
|
expect(content).toContain('DESIGN.md');
|
|
});
|
|
|
|
test('contains wireframe or sketch terminology', () => {
|
|
expect(content).toMatch(/wireframe|sketch/i);
|
|
});
|
|
|
|
test('references browse binary for rendering', () => {
|
|
expect(content).toContain('$B goto');
|
|
});
|
|
|
|
test('references screenshot capture', () => {
|
|
expect(content).toContain('$B screenshot');
|
|
});
|
|
|
|
test('specifies rough aesthetic', () => {
|
|
expect(content).toMatch(/[Rr]ough|hand-drawn/);
|
|
});
|
|
|
|
test('includes skip conditions', () => {
|
|
expect(content).toMatch(/no UI component|skip/i);
|
|
});
|
|
});
|
|
|
|
// --- {{CODEX_SECOND_OPINION}} resolver tests ---
|
|
|
|
describe('CODEX_SECOND_OPINION resolver', () => {
|
|
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
|
const codexContent = fs.readFileSync(path.join(EXTERNAL_OUT, '.agents', 'skills', 'gstack-office-hours', 'SKILL.md'), 'utf-8');
|
|
|
|
test('Phase 3.5 section appears in office-hours SKILL.md', () => {
|
|
expect(content).toContain('Phase 3.5: Cross-Model Second Opinion');
|
|
});
|
|
|
|
test('contains codex exec invocation', () => {
|
|
expect(content).toContain('codex exec');
|
|
});
|
|
|
|
test('contains opt-in AskUserQuestion text', () => {
|
|
expect(content).toContain('second opinion from an independent AI perspective');
|
|
});
|
|
|
|
test('contains cross-model synthesis instructions', () => {
|
|
expect(content).toMatch(/[Ss]ynthesis/);
|
|
expect(content).toContain('Where Claude agrees with the second opinion');
|
|
});
|
|
|
|
test('contains Claude subagent fallback', () => {
|
|
expect(content).toContain('CODEX_NOT_AVAILABLE');
|
|
expect(content).toContain('Agent tool');
|
|
expect(content).toContain('SECOND OPINION (Claude subagent)');
|
|
});
|
|
|
|
test('contains premise revision check', () => {
|
|
expect(content).toContain('Codex challenged premise');
|
|
});
|
|
|
|
test('contains error handling for auth, timeout, and empty', () => {
|
|
expect(content).toMatch(/[Aa]uth.*fail/);
|
|
expect(content).toMatch(/[Tt]imeout/);
|
|
expect(content).toMatch(/[Ee]mpty response/);
|
|
});
|
|
|
|
test('Codex host variant does NOT contain the Phase 3.5 resolver output', () => {
|
|
// The resolver returns '' for codex host, so the interactive section is stripped.
|
|
// Static template references to "Phase 3.5" in prose/conditionals are fine.
|
|
// Other resolvers (design review lite) may contain CODEX_NOT_AVAILABLE, so we
|
|
// check for Phase 3.5-specific markers only.
|
|
expect(codexContent).not.toContain('Phase 3.5: Cross-Model Second Opinion');
|
|
expect(codexContent).not.toContain('TMPERR_OH');
|
|
expect(codexContent).not.toContain('gstack-codex-oh-');
|
|
});
|
|
});
|
|
|
|
// --- Codex filesystem boundary tests ---
|
|
|
|
describe('Codex filesystem boundary', () => {
|
|
// Skills that call codex exec/review and should contain boundary text
|
|
const CODEX_CALLING_SKILLS = [
|
|
'codex', // /codex skill — 3 modes
|
|
'autoplan', // /autoplan — CEO/design/eng voices
|
|
'review', // /review — adversarial step resolver
|
|
'ship', // /ship — adversarial step resolver
|
|
'plan-eng-review', // outside voice resolver
|
|
'plan-ceo-review', // outside voice resolver
|
|
'office-hours', // second opinion resolver
|
|
];
|
|
|
|
const BOUNDARY_MARKER = 'Do NOT read or execute any';
|
|
|
|
test('boundary instruction appears in all skills that call codex', () => {
|
|
for (const skill of CODEX_CALLING_SKILLS) {
|
|
// Union: ship's codex call lives in sections/adversarial.md after the carve.
|
|
const content = readSkillUnion(skill);
|
|
expect(content).toContain(BOUNDARY_MARKER);
|
|
}
|
|
});
|
|
|
|
test('codex skill has Filesystem Boundary section', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('## Filesystem Boundary');
|
|
expect(content).toContain('skill definitions meant for a different AI system');
|
|
});
|
|
|
|
test('codex skill has rabbit-hole detection rule', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Detect skill-file rabbit holes');
|
|
expect(content).toContain('gstack-update-check');
|
|
expect(content).toContain('Consider retrying');
|
|
});
|
|
|
|
test('review.ts CODEX_BOUNDARY constant is interpolated into resolver output', () => {
|
|
// The adversarial step resolver should include boundary text in codex exec
|
|
// prompts. Carved: the adversarial step lives in sections/adversarial.md.
|
|
const reviewContent = readSkillUnion('review');
|
|
// Boundary should appear near codex exec invocations
|
|
const boundaryIdx = reviewContent.indexOf(BOUNDARY_MARKER);
|
|
const codexExecIdx = reviewContent.indexOf('codex exec');
|
|
// Both must exist and boundary must come before a codex exec call
|
|
expect(boundaryIdx).toBeGreaterThan(-1);
|
|
expect(codexExecIdx).toBeGreaterThan(-1);
|
|
});
|
|
|
|
test('autoplan boundary text avoids host-specific paths for cross-host compatibility', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'autoplan', 'SKILL.md.tmpl'), 'utf-8');
|
|
// autoplan template uses generic 'skills/gstack' pattern instead of host-specific
|
|
// paths like ~/.claude/ or .agents/skills (which break Codex/Claude output tests)
|
|
const boundaryStart = content.indexOf('Filesystem Boundary');
|
|
const boundaryEnd = content.indexOf('---', boundaryStart + 1);
|
|
const boundarySection = content.slice(boundaryStart, boundaryEnd);
|
|
expect(boundarySection).not.toContain('~/.claude/');
|
|
expect(boundarySection).not.toContain('.agents/skills');
|
|
expect(boundarySection).toContain('skills/gstack');
|
|
expect(boundarySection).toContain(BOUNDARY_MARKER);
|
|
});
|
|
});
|
|
|
|
// --- {{BENEFITS_FROM}} resolver tests ---
|
|
|
|
describe('BENEFITS_FROM resolver', () => {
|
|
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
|
const engContent = readSkillUnion('plan-eng-review'); // carved
|
|
|
|
test('plan-ceo-review contains prerequisite skill offer', () => {
|
|
expect(ceoContent).toContain('Prerequisite Skill Offer');
|
|
expect(ceoContent).toContain('/office-hours');
|
|
});
|
|
|
|
test('plan-eng-review contains prerequisite skill offer', () => {
|
|
expect(engContent).toContain('Prerequisite Skill Offer');
|
|
expect(engContent).toContain('/office-hours');
|
|
});
|
|
|
|
test('offer includes graceful decline', () => {
|
|
expect(ceoContent).toContain('No worries');
|
|
});
|
|
|
|
test('skills without benefits-from do NOT have prerequisite offer', () => {
|
|
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
|
|
expect(qaContent).not.toContain('Prerequisite Skill Offer');
|
|
});
|
|
|
|
test('inline invocation — no "another window" language', () => {
|
|
expect(ceoContent).not.toContain('another window');
|
|
expect(engContent).not.toContain('another window');
|
|
});
|
|
|
|
test('inline invocation — read-and-follow path present', () => {
|
|
expect(ceoContent).toContain('office-hours/SKILL.md');
|
|
expect(engContent).toContain('office-hours/SKILL.md');
|
|
});
|
|
|
|
test('BENEFITS_FROM delegates to INVOKE_SKILL pattern', () => {
|
|
// Should contain the INVOKE_SKILL-style loading prose (not the old manual skip list)
|
|
expect(engContent).toContain('Follow its instructions from top to bottom');
|
|
expect(engContent).toContain('skipping these sections');
|
|
expect(ceoContent).toContain('Follow its instructions from top to bottom');
|
|
});
|
|
});
|
|
|
|
// --- {{INVOKE_SKILL}} resolver tests ---
|
|
|
|
describe('INVOKE_SKILL resolver', () => {
|
|
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
|
|
|
test('plan-ceo-review uses INVOKE_SKILL for mid-session office-hours fallback', () => {
|
|
// The mid-session detection path should use INVOKE_SKILL-generated prose
|
|
expect(ceoContent).toContain('office-hours/SKILL.md');
|
|
expect(ceoContent).toContain('Follow its instructions from top to bottom');
|
|
});
|
|
|
|
test('INVOKE_SKILL output includes default skip list', () => {
|
|
expect(ceoContent).toContain('Preamble (run first)');
|
|
expect(ceoContent).toContain('Telemetry (run last)');
|
|
expect(ceoContent).toContain('AskUserQuestion Format');
|
|
});
|
|
|
|
test('INVOKE_SKILL output includes error handling', () => {
|
|
expect(ceoContent).toContain('If unreadable');
|
|
expect(ceoContent).toContain('Could not load');
|
|
});
|
|
|
|
test('template uses {{INVOKE_SKILL:office-hours}} placeholder', () => {
|
|
const tmpl = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf-8');
|
|
expect(tmpl).toContain('{{INVOKE_SKILL:office-hours}}');
|
|
});
|
|
});
|
|
|
|
// --- {{CHANGELOG_WORKFLOW}} resolver tests ---
|
|
|
|
describe('CHANGELOG_WORKFLOW resolver', () => {
|
|
const shipContent = readShipUnion();
|
|
|
|
test('ship SKILL.md contains changelog workflow', () => {
|
|
expect(shipContent).toContain('CHANGELOG (auto-generate)');
|
|
expect(shipContent).toContain('git log <base>..HEAD --oneline');
|
|
});
|
|
|
|
test('changelog workflow includes cross-check step', () => {
|
|
expect(shipContent).toContain('Cross-check');
|
|
expect(shipContent).toContain('Every commit must map to at least one bullet point');
|
|
});
|
|
|
|
test('changelog workflow includes voice guidance', () => {
|
|
expect(shipContent).toContain('Lead with what the user can now **do**');
|
|
});
|
|
|
|
test('template uses {{CHANGELOG_WORKFLOW}} placeholder', () => {
|
|
// Post-carve (T9): the skeleton points to the changelog section, which carries
|
|
// the resolver. Neither should inline the old changelog content.
|
|
const skel = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md.tmpl'), 'utf-8');
|
|
const changelogSection = fs.readFileSync(path.join(ROOT, 'ship', 'sections', 'changelog.md.tmpl'), 'utf-8');
|
|
expect(skel).toContain('{{SECTION:changelog}}');
|
|
expect(changelogSection).toContain('{{CHANGELOG_WORKFLOW}}');
|
|
expect(skel + changelogSection).not.toContain('Group commits by theme');
|
|
});
|
|
|
|
test('changelog workflow includes keep-changelog format', () => {
|
|
expect(shipContent).toContain('### Added');
|
|
expect(shipContent).toContain('### Fixed');
|
|
});
|
|
});
|
|
|
|
// --- Duplicate {{PREAMBLE}} guard (#2508/#2362) ---
|
|
|
|
describe('assertSinglePreamble', () => {
|
|
test('one {{PREAMBLE}} passes', () => {
|
|
expect(() => assertSinglePreamble('a\n{{PREAMBLE}}\nb', 'x/SKILL.md.tmpl')).not.toThrow();
|
|
});
|
|
|
|
test('zero {{PREAMBLE}} passes (sections have none)', () => {
|
|
expect(() => assertSinglePreamble('no macro here', 'x/sections/y.md.tmpl')).not.toThrow();
|
|
});
|
|
|
|
test('a second occurrence throws with the template path — even in prose', () => {
|
|
// The original #2508 bug WAS a prose mention: "emitted by {{PREAMBLE}}'s
|
|
// preamble bash". Resolution is context-blind, so the guard must be too.
|
|
const tmpl = '{{PREAMBLE}}\n\n...later: emitted by {{PREAMBLE}}\'s preamble bash';
|
|
expect(() => assertSinglePreamble(tmpl, 'spec/SKILL.md.tmpl')).toThrow(/spec\/SKILL\.md\.tmpl.*2 times/);
|
|
});
|
|
});
|
|
|
|
// --- Parameterized resolver infrastructure tests ---
|
|
|
|
describe('parameterized resolver support', () => {
|
|
test('gen-skill-docs regex handles colon-separated args', () => {
|
|
// Verify the template containing {{INVOKE_SKILL:office-hours}} was processed
|
|
// without leaving unresolved placeholders
|
|
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
|
expect(ceoContent).not.toMatch(/\{\{INVOKE_SKILL:[^}]+\}\}/);
|
|
});
|
|
|
|
test('templates with parameterized resolvers pass unresolved check', () => {
|
|
// All generated SKILL.md files should have no unresolved {{...}} placeholders
|
|
const skillDirs = fs.readdirSync(ROOT).filter(d =>
|
|
fs.existsSync(path.join(ROOT, d, 'SKILL.md'))
|
|
);
|
|
for (const dir of skillDirs) {
|
|
const content = fs.readFileSync(path.join(ROOT, dir, 'SKILL.md'), 'utf-8');
|
|
const unresolved = content.match(/\{\{[A-Z_]+(?::[^}]*)?\}\}/g);
|
|
if (unresolved) {
|
|
throw new Error(`${dir}/SKILL.md has unresolved placeholders: ${unresolved.join(', ')}`);
|
|
}
|
|
}
|
|
});
|
|
});
|
|
|
|
// --- Preamble routing injection tests ---
|
|
|
|
describe('preamble routing injection (bin/gstack-skill-start emission layer)', () => {
|
|
// Token-reduction Phase 2: the routing-injection prose left the rendered
|
|
// preamble entirely — bin/gstack-skill-start probes, gates, and emits the
|
|
// whole flow as a GSTACK_INSTRUCTION block (with the AUQ, the routing rules
|
|
// to append, and the decline ack all INSIDE the block). Absence from the
|
|
// renders is pinned by test/onboarding-moved-literals.test.ts (tombstone);
|
|
// this suite pins the gate structure and the emitted block's content.
|
|
const routingBlock = (() => {
|
|
const start = SKILL_START_SCRIPT.indexOf('_emit_block routing-injection');
|
|
expect(start).toBeGreaterThan(0);
|
|
return SKILL_START_SCRIPT.slice(start, SKILL_START_SCRIPT.indexOf('\nEOI', start));
|
|
})();
|
|
|
|
test('routing probe checks CLAUDE.md and AGENTS.md (now in gstack-skill-start)', () => {
|
|
// #2500: the probe iterates CLAUDE.md AND AGENTS.md — non-Claude hosts
|
|
// route skills via AGENTS.md, the cross-harness convention file.
|
|
expect(SKILL_START_SCRIPT).toContain('for _RF in CLAUDE.md AGENTS.md');
|
|
expect(SKILL_START_SCRIPT).toContain('grep -q "## Skill routing" "$_RF"');
|
|
expect(SKILL_START_SCRIPT).toContain('echo "HAS_ROUTING: $_HAS_ROUTING"');
|
|
});
|
|
|
|
test('script reads and echoes routing_declined config', () => {
|
|
expect(SKILL_START_SCRIPT).toMatch(/_ROUTING_DECLINED=\$\("\$_BIN\/gstack-config" get routing_declined/);
|
|
expect(SKILL_START_SCRIPT).toContain('echo "ROUTING_DECLINED: $_ROUTING_DECLINED"');
|
|
});
|
|
|
|
test('emitted block carries the routing injection AskUserQuestion', () => {
|
|
expect(routingBlock).toContain('Add routing rules to CLAUDE.md');
|
|
expect(routingBlock).toContain("I'll invoke skills manually");
|
|
});
|
|
|
|
test('routing injection respects prior decline (gate + in-block ack)', () => {
|
|
expect(SKILL_START_SCRIPT).toContain('[ "$_ROUTING_DECLINED" = "false" ]');
|
|
expect(routingBlock).toMatch(/routing_declined.*true/);
|
|
expect(routingBlock).toContain('re-enable with `__BIN__/gstack-config set routing_declined false`');
|
|
});
|
|
|
|
test('routing injection only fires when all conditions met', () => {
|
|
// Must be: HAS_ROUTING=no AND ROUTING_DECLINED=false AND PROACTIVE_PROMPTED=yes
|
|
expect(SKILL_START_SCRIPT).toContain(
|
|
'if [ "$_HAS_ROUTING" = "no" ] && [ "$_ROUTING_DECLINED" = "false" ] && [ "$_PROACTIVE_PROMPTED" = "yes" ]; then',
|
|
);
|
|
});
|
|
|
|
test('routing section content includes key routing rules', () => {
|
|
expect(routingBlock).toContain('invoke /office-hours');
|
|
expect(routingBlock).toContain('invoke /investigate');
|
|
expect(routingBlock).toContain('invoke /ship');
|
|
expect(routingBlock).toContain('invoke /qa');
|
|
});
|
|
|
|
test('routing section uses renamed checkpoint skills (not stale /checkpoint)', () => {
|
|
expect(routingBlock).toContain('invoke /context-save');
|
|
expect(routingBlock).toContain('invoke /context-restore');
|
|
expect(routingBlock).not.toContain('invoke checkpoint');
|
|
});
|
|
|
|
test('routing section uses soft "when in doubt" policy, not hard "ALWAYS invoke"', () => {
|
|
expect(routingBlock).toContain('When in doubt, invoke the skill');
|
|
expect(routingBlock).not.toContain('Do NOT answer directly');
|
|
});
|
|
});
|
|
|
|
// --- {{DESIGN_OUTSIDE_VOICES}} resolver tests ---
|
|
|
|
describe('DESIGN_OUTSIDE_VOICES resolver', () => {
|
|
test('plan-design-review contains outside voices section', () => {
|
|
const content = readSkillUnion('plan-design-review');
|
|
expect(content).toContain('Design Outside Voices');
|
|
expect(content).toContain('CODEX_AVAILABLE');
|
|
expect(content).toContain('LITMUS SCORECARD');
|
|
});
|
|
|
|
test('design-review contains outside voices section', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Design Outside Voices');
|
|
expect(content).toContain('source audit');
|
|
});
|
|
|
|
test('design-consultation contains outside voices section', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'design-consultation', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Design Outside Voices');
|
|
expect(content).toContain('design direction');
|
|
});
|
|
|
|
test('branches correctly per skillName — different prompts', () => {
|
|
const planContent = readSkillUnion('plan-design-review');
|
|
const consultContent = fs.readFileSync(path.join(ROOT, 'design-consultation', 'SKILL.md'), 'utf-8');
|
|
// plan-design-review uses analytical prompt (high reasoning)
|
|
expect(planContent).toContain('model_reasoning_effort="high"');
|
|
// design-consultation uses creative prompt (medium reasoning)
|
|
expect(consultContent).toContain('model_reasoning_effort="medium"');
|
|
});
|
|
});
|
|
|
|
// --- {{DESIGN_HARD_RULES}} resolver tests ---
|
|
|
|
describe('DESIGN_HARD_RULES resolver', () => {
|
|
test('plan-design-review Pass 4 contains hard rules', () => {
|
|
const content = readSkillUnion('plan-design-review');
|
|
expect(content).toContain('Design Hard Rules');
|
|
expect(content).toContain('Classifier');
|
|
expect(content).toContain('MARKETING/LANDING PAGE');
|
|
expect(content).toContain('APP UI');
|
|
});
|
|
|
|
test('design-review contains hard rules', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Design Hard Rules');
|
|
});
|
|
|
|
test('includes all 3 rule sets', () => {
|
|
const content = readSkillUnion('plan-design-review');
|
|
expect(content).toContain('Landing page rules');
|
|
expect(content).toContain('App UI rules');
|
|
expect(content).toContain('Universal rules');
|
|
});
|
|
|
|
test('references shared AI slop blacklist items', () => {
|
|
const content = readSkillUnion('plan-design-review');
|
|
expect(content).toContain('3-column feature grid');
|
|
expect(content).toContain('Purple/violet/indigo');
|
|
});
|
|
|
|
test('includes OpenAI hard rejection criteria', () => {
|
|
const content = readSkillUnion('plan-design-review');
|
|
expect(content).toContain('Generic SaaS card grid');
|
|
expect(content).toContain('Carousel with no narrative purpose');
|
|
});
|
|
|
|
test('includes OpenAI litmus checks', () => {
|
|
const content = readSkillUnion('plan-design-review');
|
|
expect(content).toContain('Brand/product unmistakable');
|
|
expect(content).toContain('premium with all decorative shadows removed');
|
|
});
|
|
});
|
|
|
|
// --- Extended DESIGN_SKETCH resolver tests ---
|
|
|
|
describe('DESIGN_SKETCH extended with outside voices', () => {
|
|
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
|
|
|
test('contains outside design voices step', () => {
|
|
expect(content).toContain('Outside design voices');
|
|
});
|
|
|
|
test('offers opt-in via AskUserQuestion', () => {
|
|
expect(content).toContain('outside design perspectives');
|
|
});
|
|
|
|
test('still contains original wireframe steps', () => {
|
|
expect(content).toContain('wireframe');
|
|
expect(content).toContain('$B goto');
|
|
});
|
|
});
|
|
|
|
// --- Extended DESIGN_REVIEW_LITE resolver tests ---
|
|
|
|
describe('DESIGN_REVIEW_LITE extended with Codex', () => {
|
|
const content = readShipUnion();
|
|
|
|
test('contains Codex design voice block', () => {
|
|
expect(content).toContain('Codex design voice');
|
|
expect(content).toContain('CODEX (design)');
|
|
});
|
|
|
|
test('still contains original checklist steps', () => {
|
|
expect(content).toContain('design-checklist.md');
|
|
expect(content).toContain('SCOPE_FRONTEND');
|
|
});
|
|
|
|
});
|
|
|
|
// ─── Codex Generation Tests ─────────────────────────────────
|
|
|
|
describe('Codex generation (--host codex)', () => {
|
|
// .agents/ is gitignored (v0.11.2.0) — read the module-level out-dir render
|
|
// (--host all covers codex) instead of regenerating the live tree in place.
|
|
const AGENTS_DIR = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
|
|
|
// Dynamic discovery of expected Codex skills: all templates except /codex.
|
|
// The out-dir is a fresh mkdtemp, so the vendored-dev-mode symlink loop
|
|
// (.agents/skills/{name} → repo root) that made the generator skip skills
|
|
// in-place can never occur here — every template renders.
|
|
const CODEX_SKILLS = (() => {
|
|
const skills: Array<{ dir: string; codexName: string }> = [];
|
|
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
|
skills.push({ dir: '.', codexName: 'gstack' });
|
|
}
|
|
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
|
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
|
if (entry.name === 'codex') continue; // /codex is excluded from Codex output
|
|
if (!fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) continue;
|
|
const codexName = entry.name.startsWith('gstack-') ? entry.name : `gstack-${entry.name}`;
|
|
skills.push({ dir: entry.name, codexName });
|
|
}
|
|
return skills;
|
|
})();
|
|
|
|
test('--host codex generates correct output paths', () => {
|
|
for (const skill of CODEX_SKILLS) {
|
|
const skillMd = path.join(AGENTS_DIR, skill.codexName, 'SKILL.md');
|
|
expect(fs.existsSync(skillMd)).toBe(true);
|
|
}
|
|
});
|
|
|
|
test('root gstack bundle has OpenAI metadata for Codex skill browsing', () => {
|
|
const rootMetadata = path.join(ROOT, 'agents', 'openai.yaml');
|
|
expect(fs.existsSync(rootMetadata)).toBe(true);
|
|
const content = fs.readFileSync(rootMetadata, 'utf-8');
|
|
expect(content).toContain('display_name: "gstack"');
|
|
expect(content).toContain('Use $gstack to locate the bundled gstack skills.');
|
|
expect(content).toContain('allow_implicit_invocation: true');
|
|
});
|
|
|
|
test('externalSkillName mapping: root is gstack, others are gstack-{dir}', () => {
|
|
// Root → gstack
|
|
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack', 'SKILL.md'))).toBe(true);
|
|
// Subdirectories → gstack-{dir}
|
|
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'))).toBe(true);
|
|
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'))).toBe(true);
|
|
// gstack-upgrade doesn't double-prefix
|
|
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-upgrade', 'SKILL.md'))).toBe(true);
|
|
// No double-prefix: gstack-gstack-upgrade must NOT exist
|
|
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-gstack-upgrade', 'SKILL.md'))).toBe(false);
|
|
});
|
|
|
|
test('Codex frontmatter has ONLY name + description', () => {
|
|
for (const skill of CODEX_SKILLS) {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
|
expect(content.startsWith('---\n')).toBe(true);
|
|
const fmEnd = content.indexOf('\n---', 4);
|
|
expect(fmEnd).toBeGreaterThan(0);
|
|
const frontmatter = content.slice(4, fmEnd);
|
|
// Must have name and description
|
|
expect(frontmatter).toContain('name:');
|
|
expect(frontmatter).toContain('description:');
|
|
// Must NOT have allowed-tools, version, or hooks
|
|
expect(frontmatter).not.toContain('allowed-tools:');
|
|
expect(frontmatter).not.toContain('version:');
|
|
expect(frontmatter).not.toContain('hooks:');
|
|
}
|
|
});
|
|
|
|
test('all Codex skills have agents/openai.yaml metadata', () => {
|
|
for (const skill of CODEX_SKILLS) {
|
|
const metadata = path.join(AGENTS_DIR, skill.codexName, 'agents', 'openai.yaml');
|
|
expect(fs.existsSync(metadata)).toBe(true);
|
|
const content = fs.readFileSync(metadata, 'utf-8');
|
|
expect(content).toContain(`display_name: "${skill.codexName}"`);
|
|
expect(content).toContain('short_description:');
|
|
expect(content).toContain('allow_implicit_invocation: true');
|
|
}
|
|
});
|
|
|
|
test('no .claude/skills/ in Codex output', () => {
|
|
for (const skill of CODEX_SKILLS) {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
|
expect(content).not.toContain('.claude/skills');
|
|
}
|
|
});
|
|
|
|
test('no ~/.claude/ paths in Codex output', () => {
|
|
for (const skill of CODEX_SKILLS) {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
|
expect(content).not.toContain('~/.claude/');
|
|
}
|
|
});
|
|
|
|
test('/codex skill excluded from Codex output', () => {
|
|
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-codex', 'SKILL.md'))).toBe(false);
|
|
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-codex'))).toBe(false);
|
|
});
|
|
|
|
test('Codex output includes Claude outside-voice skill with read-only boundary', () => {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-claude', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('claude -p');
|
|
expect(content).toContain('mktemp /tmp/gstack-claude-prompt-');
|
|
expect(content).toContain('mktemp /tmp/gstack-claude-response-XXXXXX');
|
|
expect(content).toContain('mktemp /tmp/gstack-claude-error-XXXXXX');
|
|
expect(content).toContain('mktemp /tmp/gstack-claude-diff-');
|
|
expect(content).not.toMatch(/gstack-claude-(?:prompt|response|error|diff)-X{6,}\.\w+/);
|
|
expect(content).not.toContain('/tmp/gstack-claude-diff-$$');
|
|
expect(content).toContain('cat "$PROMPT_FILE" | "$CLAUDE_BIN" -p');
|
|
expect(content).toContain('Resolve the binary and invoke it in the same host execution context');
|
|
expect(content).toContain('--disable-slash-commands');
|
|
expect(content).toContain('--tools ""');
|
|
expect(content).toContain('--allowedTools Read,Grep,Glob');
|
|
expect(content).toContain('--disallowedTools Bash,Edit,Write');
|
|
expect(content).toContain('Do not infer authentication state from credential files');
|
|
expect(content).toContain('run the actual `claude -p`');
|
|
expect(content).not.toContain('AUTH_MISSING');
|
|
expect(content).not.toContain('$HOME/.claude/.credentials.json');
|
|
expect(content).toContain('is_error');
|
|
});
|
|
|
|
test('Claude temp file templates are accepted by host mktemp', () => {
|
|
for (const template of [
|
|
'/tmp/gstack-claude-prompt-XXXXXX',
|
|
'/tmp/gstack-claude-response-XXXXXX',
|
|
'/tmp/gstack-claude-error-XXXXXX',
|
|
'/tmp/gstack-claude-diff-XXXXXX',
|
|
]) {
|
|
const result = spawnSync('mktemp', [template], { encoding: 'utf-8' });
|
|
expect(result.status).toBe(0);
|
|
const created = result.stdout.trim();
|
|
expect(created.startsWith(template.replace('XXXXXX', ''))).toBe(true);
|
|
fs.unlinkSync(created);
|
|
}
|
|
});
|
|
|
|
test('Codex review step stripped from Codex-host ship and review', () => {
|
|
const shipContent = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
|
expect(shipContent).not.toContain('codex review --base');
|
|
expect(shipContent).not.toContain('CODEX_REVIEWS');
|
|
|
|
const reviewContent = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
|
expect(reviewContent).not.toContain('codex review --base');
|
|
expect(reviewContent).not.toContain('CODEX_REVIEWS');
|
|
});
|
|
|
|
test('--host codex --dry-run freshness', () => {
|
|
// Dry-run against the out-dir render: determinism/idempotency check
|
|
// (regenerating produces the same bytes the module-level render did).
|
|
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
|
cwd: ROOT,
|
|
stdout: 'pipe',
|
|
stderr: 'pipe',
|
|
});
|
|
expect(result.exitCode).toBe(0);
|
|
const output = result.stdout.toString();
|
|
// Every Codex skill should be FRESH
|
|
for (const skill of CODEX_SKILLS) {
|
|
expect(output).toContain(`FRESH: .agents/skills/${skill.codexName}/SKILL.md`);
|
|
}
|
|
expect(output).not.toContain('STALE');
|
|
});
|
|
|
|
test('--host agents alias produces same output as --host codex', () => {
|
|
const codexResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
|
cwd: ROOT,
|
|
stdout: 'pipe',
|
|
stderr: 'pipe',
|
|
});
|
|
const agentsResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'agents', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
|
cwd: ROOT,
|
|
stdout: 'pipe',
|
|
stderr: 'pipe',
|
|
});
|
|
expect(codexResult.exitCode).toBe(0);
|
|
expect(agentsResult.exitCode).toBe(0);
|
|
// Both should produce the same output (same FRESH lines)
|
|
expect(codexResult.stdout.toString()).toBe(agentsResult.stdout.toString());
|
|
});
|
|
|
|
test('multiline descriptions preserved in Codex output', () => {
|
|
// office-hours has a multiline description — verify it survives the frontmatter transform
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-office-hours', 'SKILL.md'), 'utf-8');
|
|
const fmEnd = content.indexOf('\n---', 4);
|
|
const frontmatter = content.slice(4, fmEnd);
|
|
// Description should span multiple lines (block scalar)
|
|
const descLines = frontmatter.split('\n').filter(l => l.startsWith(' '));
|
|
expect(descLines.length).toBeGreaterThan(1);
|
|
// Verify key phrases survived
|
|
expect(frontmatter).toContain('YC Office Hours');
|
|
});
|
|
|
|
test('hook skills have safety prose and no hooks: in frontmatter', () => {
|
|
const HOOK_SKILLS = ['gstack-careful', 'gstack-freeze', 'gstack-guard'];
|
|
for (const skillName of HOOK_SKILLS) {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, skillName, 'SKILL.md'), 'utf-8');
|
|
// Must have safety advisory prose
|
|
expect(content).toContain('Safety Advisory');
|
|
// Must NOT have hooks: in frontmatter
|
|
const fmEnd = content.indexOf('\n---', 4);
|
|
const frontmatter = content.slice(4, fmEnd);
|
|
expect(frontmatter).not.toContain('hooks:');
|
|
}
|
|
});
|
|
|
|
test('all Codex SKILL.md files have auto-generated header', () => {
|
|
for (const skill of CODEX_SKILLS) {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('AUTO-GENERATED from SKILL.md.tmpl');
|
|
expect(content).toContain('Regenerate: bun run gen:skill-docs');
|
|
}
|
|
});
|
|
|
|
test('Codex preamble resolves runtime assets from repo-local or global gstack roots', () => {
|
|
// Check a skill that has a preamble (review is a good candidate)
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('GSTACK_ROOT');
|
|
expect(content).toContain('$_ROOT/.agents/skills/gstack');
|
|
// Phase 1/2: config reads moved into gstack-skill-start — the fence itself
|
|
// is the bin asset the preamble must resolve through $GSTACK_BIN, and the
|
|
// question-preference runtime call still resolves the same way.
|
|
expect(content).toContain('$GSTACK_BIN/gstack-skill-start');
|
|
expect(content).toContain('$GSTACK_BIN/gstack-question-preference');
|
|
// The upgrade-skill doc reference moved into the script's upgrade-flow
|
|
// block, resolved $0-relative ($_ROOT_DIR) — host-neutral by construction,
|
|
// so the Codex render no longer needs its own copy.
|
|
expect(SKILL_START_SCRIPT).toContain('$_ROOT_DIR/gstack-upgrade/SKILL.md');
|
|
expect(SKILL_START_SCRIPT).toContain('_ROOT_DIR=$(dirname "$_BIN")');
|
|
expect(content).not.toContain('~/.codex/skills/gstack/bin/gstack-config get telemetry');
|
|
});
|
|
|
|
// ─── Path rewriting regression tests ─────────────────────────
|
|
|
|
test('sidecar paths resolve through $GSTACK_ROOT (not gstack-review/)', () => {
|
|
// #2518: templates now anchor sidecars at the installed skill root
|
|
// (~/.claude/skills/gstack/review/...), which the codex path rewrite turns
|
|
// into $GSTACK_ROOT/review/... — resolved by the preamble against the
|
|
// repo-local .agents root or the global install. The old repo-relative
|
|
// form (.claude/skills/review/) only resolved inside gstack's own checkout.
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('$GSTACK_ROOT/review/checklist.md');
|
|
// design-checklist.md is now referenced via Review Army specialist (Claude only, stripped for Codex)
|
|
// Wrong: must NOT reference gstack-review/checklist.md (file doesn't exist there)
|
|
expect(content).not.toContain('.agents/skills/gstack-review/checklist.md');
|
|
});
|
|
|
|
test('sidecar paths in ship skill point to gstack/review/ for pre-landing review', () => {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
|
// Ship references the review checklist in its pre-landing review step
|
|
if (content.includes('checklist.md')) {
|
|
expect(content).toContain('.agents/skills/gstack/review/');
|
|
expect(content).not.toContain('.agents/skills/gstack-review/checklist');
|
|
}
|
|
});
|
|
|
|
test('greptile-triage sidecar path is correct', () => {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
|
if (content.includes('greptile-triage')) {
|
|
expect(content).toContain('$GSTACK_ROOT/review/greptile-triage.md');
|
|
expect(content).not.toContain('.agents/skills/gstack-review/greptile-triage');
|
|
}
|
|
});
|
|
|
|
test('all four path rewrite rules produce correct output', () => {
|
|
// Test each of the 4 path rewrite rules individually
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
|
|
|
// Rule 1: ~/.claude/skills/gstack → $GSTACK_ROOT
|
|
expect(content).not.toContain('~/.claude/skills/gstack');
|
|
expect(content).toContain('$GSTACK_ROOT');
|
|
|
|
// Rule 2: .claude/skills/gstack → .agents/skills/gstack
|
|
expect(content).not.toContain('.claude/skills/gstack');
|
|
|
|
// Rule 3: .claude/skills/review → .agents/skills/gstack/review
|
|
expect(content).not.toContain('.claude/skills/review');
|
|
|
|
// Rule 4: .claude/skills → .agents/skills (catch-all)
|
|
expect(content).not.toContain('.claude/skills');
|
|
});
|
|
|
|
test('path rewrite rules apply to all Codex skills with sidecar references', () => {
|
|
// Verify across ALL generated skills, not just review
|
|
for (const skill of CODEX_SKILLS) {
|
|
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
|
// No skill should reference Claude paths
|
|
expect(content).not.toContain('~/.claude/skills');
|
|
expect(content).not.toContain('.claude/skills');
|
|
if (content.includes('gstack-config') || content.includes('gstack-update-check') || content.includes('gstack-telemetry-log')) {
|
|
expect(content).toContain('$GSTACK_ROOT');
|
|
}
|
|
// If a skill references checklist.md, it must use the correct sidecar path
|
|
if (content.includes('checklist.md') && !content.includes('design-checklist.md')) {
|
|
expect(content).not.toContain('gstack-review/checklist.md');
|
|
}
|
|
}
|
|
});
|
|
|
|
// ─── Claude output regression guard ─────────────────────────
|
|
|
|
test('Claude output uses installed-root review paths (#2518)', () => {
|
|
// Codex changes must NOT affect Claude output; the Claude form is the
|
|
// installed-root anchor, not the old repo-relative path that only
|
|
// resolved inside gstack's own checkout.
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('~/.claude/skills/gstack/review/checklist.md');
|
|
expect(content).toContain('~/.claude/skills/gstack');
|
|
// Must NOT contain Codex HOST paths. `~/.codex/sessions/` is exempt: the
|
|
// timeout-wrapper guidance documents the Codex CLI's own rollout-log
|
|
// location (a user-facing CLI path, same class as ~/.codex/logs/ in the
|
|
// codex skill), not the gstack Codex host install path.
|
|
// `~/.codex/config.toml` is the same user-facing class: the shared
|
|
// codexPreflight's model_unusable branch (#2477) points at the CLI's own
|
|
// config file, where the rejected `model =` pin lives.
|
|
expect(content).not.toContain('.agents/skills');
|
|
expect(
|
|
content
|
|
.replaceAll('~/.codex/sessions/', '')
|
|
.replaceAll('~/.codex/config.toml', ''),
|
|
).not.toContain('~/.codex/');
|
|
});
|
|
|
|
test('Claude output unchanged: ship skill still uses .claude/skills/ paths', () => {
|
|
const content = readShipUnion();
|
|
expect(content).toContain('~/.claude/skills/gstack');
|
|
expect(content).not.toContain('.agents/skills');
|
|
// ~/.codex/sessions/ is the Codex CLI's rollout-log path (user-facing),
|
|
// documented by the adversarial-pass timeout guidance; ~/.codex/config.toml
|
|
// is the CLI's own config file (model_unusable guidance, #2477) — see the
|
|
// review test above.
|
|
expect(
|
|
content
|
|
.replaceAll('~/.codex/sessions/', '')
|
|
.replaceAll('~/.codex/config.toml', ''),
|
|
).not.toContain('~/.codex/');
|
|
});
|
|
|
|
test('Claude output unchanged: all Claude skills have zero Codex paths', () => {
|
|
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
|
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
|
// pair-agent legitimately documents how Codex agents store credentials.
|
|
// codex + autoplan document the Codex CLI auth file (~/.codex/auth.json)
|
|
// and log path (~/.codex/logs/) — those are user-facing Codex CLI paths,
|
|
// not the gstack Codex host install path. ~/.codex/sessions/ (rollout
|
|
// logs, referenced by the review/ship timeout guidance) and
|
|
// ~/.codex/config.toml (the model_unusable guidance in the shared
|
|
// codexPreflight, #2477) are the same user-facing class, so they are
|
|
// scrubbed before the ban.
|
|
if (skill.dir !== 'pair-agent' && skill.dir !== 'codex' && skill.dir !== 'autoplan') {
|
|
expect(
|
|
content
|
|
.replaceAll('~/.codex/sessions/', '')
|
|
.replaceAll('~/.codex/config.toml', ''),
|
|
).not.toContain('~/.codex/');
|
|
}
|
|
// gstack-upgrade legitimately references .agents/skills for cross-platform detection
|
|
if (skill.dir !== 'gstack-upgrade') {
|
|
expect(content).not.toContain('.agents/skills');
|
|
}
|
|
}
|
|
});
|
|
|
|
// ─── Design outside voices: Codex host guard ─────────────────
|
|
|
|
test('codex host produces empty outside voices in design-review', () => {
|
|
const codexContent = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-design-review', 'SKILL.md'), 'utf-8');
|
|
expect(codexContent).not.toContain('Design Outside Voices');
|
|
});
|
|
|
|
test('codex host does not include Codex design block in ship', () => {
|
|
const codexContent = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
|
expect(codexContent).not.toContain('Codex design voice');
|
|
});
|
|
|
|
// ─── Explicit --model override wins over the host default ────
|
|
// Without --model the codex host renders its defaultModel (gpt) — pinned by
|
|
// the golden test. This pins the OTHER direction through the real CLI:
|
|
// `./setup --host codex --model <id>` depends on it. The override renders
|
|
// into its OWN out-dir, so no restore pass is needed — the host-default
|
|
// render (EXTERNAL_OUT) is untouched and asserted directly.
|
|
test('explicit --model overrides the codex host default', () => {
|
|
const overrideOut = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-model-override-'));
|
|
try {
|
|
const override = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--model', 'claude', '--out-dir', overrideOut], {
|
|
cwd: ROOT,
|
|
stdout: 'pipe',
|
|
stderr: 'pipe',
|
|
});
|
|
expect(override.exitCode).toBe(0);
|
|
const content = fs.readFileSync(path.join(overrideOut, '.agents', 'skills', 'gstack-ship', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Model-Specific Behavioral Patch (claude)');
|
|
// The overlay now travels as --model into gstack-skill-start, which
|
|
// echoes MODEL_OVERLAY at runtime.
|
|
expect(content).toContain('--model "claude"');
|
|
} finally {
|
|
fs.rmSync(overrideOut, { recursive: true, force: true });
|
|
}
|
|
// Host-default direction: the untouched EXTERNAL_OUT render carries gpt.
|
|
const hostDefault = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
|
expect(hostDefault).toContain('Model-Specific Behavioral Patch (gpt)');
|
|
expect(hostDefault).toContain('--model "gpt"');
|
|
});
|
|
});
|
|
|
|
// ─── Factory generation tests ────────────────────────────────
|
|
|
|
describe('Factory generation (--host factory)', () => {
|
|
// .factory/ is gitignored — read the module-level out-dir render
|
|
// (--host all covers factory) instead of regenerating in place.
|
|
const FACTORY_DIR = path.join(EXTERNAL_OUT, '.factory', 'skills');
|
|
|
|
// Fresh out-dir → the vendored-dev-mode symlink loop can never occur, so
|
|
// every template renders (see the Codex discovery note above).
|
|
const FACTORY_SKILLS = (() => {
|
|
const skills: Array<{ dir: string; factoryName: string }> = [];
|
|
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
|
skills.push({ dir: '.', factoryName: 'gstack' });
|
|
}
|
|
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
|
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
|
if (entry.name === 'codex') continue;
|
|
if (!fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) continue;
|
|
const factoryName = entry.name.startsWith('gstack-') ? entry.name : `gstack-${entry.name}`;
|
|
skills.push({ dir: entry.name, factoryName });
|
|
}
|
|
return skills;
|
|
})();
|
|
|
|
test('--host factory generates correct output paths', () => {
|
|
for (const skill of FACTORY_SKILLS) {
|
|
const skillMd = path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md');
|
|
expect(fs.existsSync(skillMd)).toBe(true);
|
|
}
|
|
});
|
|
|
|
test('Factory frontmatter has name + description + user-invocable', () => {
|
|
for (const skill of FACTORY_SKILLS) {
|
|
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
|
const fmEnd = content.indexOf('\n---', 4);
|
|
const frontmatter = content.slice(4, fmEnd);
|
|
expect(frontmatter).toContain('name:');
|
|
expect(frontmatter).toContain('description:');
|
|
expect(frontmatter).toContain('user-invocable: true');
|
|
expect(frontmatter).not.toContain('allowed-tools:');
|
|
expect(frontmatter).not.toContain('preamble-tier:');
|
|
expect(frontmatter).not.toContain('sensitive:');
|
|
}
|
|
});
|
|
|
|
test('sensitive skills have disable-model-invocation', () => {
|
|
const SENSITIVE = ['gstack-ship', 'gstack-land-and-deploy', 'gstack-guard', 'gstack-careful', 'gstack-freeze', 'gstack-unfreeze'];
|
|
for (const name of SENSITIVE) {
|
|
const content = fs.readFileSync(path.join(FACTORY_DIR, name, 'SKILL.md'), 'utf-8');
|
|
const fmEnd = content.indexOf('\n---', 4);
|
|
const frontmatter = content.slice(4, fmEnd);
|
|
expect(frontmatter).toContain('disable-model-invocation: true');
|
|
}
|
|
});
|
|
|
|
test('non-sensitive skills lack disable-model-invocation', () => {
|
|
const NON_SENSITIVE = ['gstack-qa', 'gstack-review', 'gstack-investigate', 'gstack-browse'];
|
|
for (const name of NON_SENSITIVE) {
|
|
const content = fs.readFileSync(path.join(FACTORY_DIR, name, 'SKILL.md'), 'utf-8');
|
|
const fmEnd = content.indexOf('\n---', 4);
|
|
const frontmatter = content.slice(4, fmEnd);
|
|
expect(frontmatter).not.toContain('disable-model-invocation');
|
|
}
|
|
});
|
|
|
|
test('no .claude/skills/ in Factory output', () => {
|
|
for (const skill of FACTORY_SKILLS) {
|
|
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
|
expect(content).not.toContain('.claude/skills');
|
|
}
|
|
});
|
|
|
|
test('no ~/.claude/skills/ paths in Factory output', () => {
|
|
for (const skill of FACTORY_SKILLS) {
|
|
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
|
// ~/.claude/skills should be rewritten, but ~/.claude/plans is legitimate
|
|
// (plan directory lookup) and ~/.claude/ in codex prompts is intentional
|
|
expect(content).not.toContain('~/.claude/skills');
|
|
}
|
|
});
|
|
|
|
test('/codex skill excluded from Factory output', () => {
|
|
expect(fs.existsSync(path.join(FACTORY_DIR, 'gstack-codex', 'SKILL.md'))).toBe(false);
|
|
expect(fs.existsSync(path.join(FACTORY_DIR, 'gstack-codex'))).toBe(false);
|
|
});
|
|
|
|
test('Factory keeps Codex integration blocks', () => {
|
|
// Factory users CAN use Codex second opinions (codex exec is a standalone binary)
|
|
const shipContent = fs.readFileSync(path.join(FACTORY_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
|
expect(shipContent).toContain('codex');
|
|
});
|
|
|
|
test('no agents/openai.yaml in Factory output', () => {
|
|
for (const skill of FACTORY_SKILLS) {
|
|
const yamlPath = path.join(FACTORY_DIR, skill.factoryName, 'agents', 'openai.yaml');
|
|
expect(fs.existsSync(yamlPath)).toBe(false);
|
|
}
|
|
});
|
|
|
|
test('--host droid alias works', () => {
|
|
const factoryResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
|
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
|
|
});
|
|
const droidResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'droid', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
|
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
|
|
});
|
|
expect(factoryResult.exitCode).toBe(0);
|
|
expect(droidResult.exitCode).toBe(0);
|
|
expect(factoryResult.stdout.toString()).toBe(droidResult.stdout.toString());
|
|
});
|
|
|
|
test('--host factory --dry-run freshness', () => {
|
|
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
|
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
|
|
});
|
|
expect(result.exitCode).toBe(0);
|
|
const output = result.stdout.toString();
|
|
for (const skill of FACTORY_SKILLS) {
|
|
expect(output).toContain(`FRESH: .factory/skills/${skill.factoryName}/SKILL.md`);
|
|
}
|
|
expect(output).not.toContain('STALE');
|
|
});
|
|
|
|
test('Factory preamble uses .factory paths', () => {
|
|
const content = fs.readFileSync(path.join(FACTORY_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('GSTACK_ROOT');
|
|
expect(content).toContain('$_ROOT/.factory/skills/gstack');
|
|
expect(content).toContain('$GSTACK_BIN/gstack-config');
|
|
});
|
|
});
|
|
|
|
// ─── Parameterized host smoke tests (config-driven) ─────────
|
|
|
|
import { ALL_HOST_CONFIGS, getExternalHosts } from '../hosts/index';
|
|
|
|
describe('Parameterized host smoke tests', () => {
|
|
// Every external host was rendered up front by the module-level
|
|
// `--host all --out-dir EXTERNAL_OUT` render, so the per-host `--dry-run`
|
|
// freshness checks are deterministic: they compare a regeneration against
|
|
// that render — an idempotency/determinism check that catches
|
|
// non-deterministic gen without ever writing (or depending on) the live
|
|
// gitignored host dirs. The tracked-claude freshness test
|
|
// (`generated files are fresh`) runs earlier and is unaffected.
|
|
for (const hostConfig of getExternalHosts()) {
|
|
describe(`${hostConfig.displayName} (--host ${hostConfig.name})`, () => {
|
|
const hostDir = path.join(EXTERNAL_OUT, hostConfig.hostSubdir, 'skills');
|
|
|
|
test('generates output that exists on disk', () => {
|
|
// The module-level --host all render must have produced this host's tree.
|
|
expect(fs.existsSync(hostDir)).toBe(true);
|
|
const skills = fs.readdirSync(hostDir).filter(d =>
|
|
fs.existsSync(path.join(hostDir, d, 'SKILL.md'))
|
|
);
|
|
expect(skills.length).toBeGreaterThan(0);
|
|
});
|
|
|
|
test('no .claude/skills path leakage outside repo-root sidecar symlinks', () => {
|
|
if (!fs.existsSync(hostDir)) return; // skip if not generated
|
|
const skills = fs.readdirSync(hostDir);
|
|
for (const skill of skills) {
|
|
// Dev installs may mount the repo root at host/skills/gstack as a runtime
|
|
// sidecar. The generator skips that symlink loop, so leakage checks should too.
|
|
if (isRepoRootSymlink(path.join(hostDir, skill))) continue;
|
|
const skillMd = path.join(hostDir, skill, 'SKILL.md');
|
|
if (!fs.existsSync(skillMd)) continue;
|
|
const content = fs.readFileSync(skillMd, 'utf-8');
|
|
// Strip bash blocks (which have legitimate fallback paths)
|
|
const noBash = content.replace(/```bash\n[\s\S]*?```/g, '');
|
|
const leaks = noBash.split('\n').filter(l => l.includes('.claude/skills'));
|
|
if (leaks.length > 0) {
|
|
throw new Error(`${skill}: .claude/skills leakage:\n${leaks.slice(0, 3).join('\n')}`);
|
|
}
|
|
}
|
|
});
|
|
|
|
test('frontmatter has name and description', () => {
|
|
if (!fs.existsSync(hostDir)) return;
|
|
const skills = fs.readdirSync(hostDir);
|
|
for (const skill of skills) {
|
|
const skillMd = path.join(hostDir, skill, 'SKILL.md');
|
|
if (!fs.existsSync(skillMd)) continue;
|
|
const content = fs.readFileSync(skillMd, 'utf-8');
|
|
expect(content).toMatch(/^---\n/);
|
|
expect(content).toMatch(/^name:\s/m);
|
|
expect(content).toMatch(/^description:\s/m);
|
|
}
|
|
});
|
|
|
|
test('--dry-run freshness check passes', () => {
|
|
const result = Bun.spawnSync(
|
|
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', hostConfig.name, '--dry-run', '--out-dir', EXTERNAL_OUT],
|
|
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }
|
|
);
|
|
expect(result.exitCode).toBe(0);
|
|
const output = result.stdout.toString();
|
|
expect(output).not.toContain('STALE');
|
|
});
|
|
|
|
if (hostConfig.generation.skipSkills?.includes('codex')) {
|
|
test('/codex skill excluded', () => {
|
|
expect(fs.existsSync(path.join(hostDir, 'gstack-codex', 'SKILL.md'))).toBe(false);
|
|
});
|
|
}
|
|
});
|
|
}
|
|
});
|
|
|
|
// ─── --host all tests ────────────────────────────────────────
|
|
|
|
describe('--host all', () => {
|
|
// Same determinism guard as the parameterized block: the module-level
|
|
// `--host all --out-dir EXTERNAL_OUT` render is the comparison baseline, so
|
|
// this dry-run reports FRESH regardless of live-tree state — and proves the
|
|
// claude host plus every external host regenerate deterministically.
|
|
test('--host all generates for all registered hosts', () => {
|
|
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
|
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
|
|
});
|
|
expect(result.exitCode).toBe(0);
|
|
const output = result.stdout.toString();
|
|
// All hosts should appear in output
|
|
expect(output).toContain('FRESH: SKILL.md'); // claude
|
|
for (const hostConfig of getExternalHosts()) {
|
|
expect(output).toContain(`FRESH: ${hostConfig.hostSubdir}/skills/`);
|
|
}
|
|
});
|
|
});
|
|
|
|
// ─── Setup script validation ─────────────────────────────────
|
|
// These tests verify the setup script's install layout matches
|
|
// what the generator produces — catching the bug where setup
|
|
// installed Claude-format source dirs for Codex users.
|
|
|
|
describe('setup script validation', () => {
|
|
const setupContent = fs.readFileSync(path.join(ROOT, 'setup'), 'utf-8');
|
|
|
|
test('setup has separate link functions for Claude and Codex', () => {
|
|
expect(setupContent).toContain('link_claude_skill_dirs');
|
|
expect(setupContent).toContain('link_codex_skill_dirs');
|
|
// Old unified function must not exist
|
|
expect(setupContent).not.toMatch(/^link_skill_dirs\(\)/m);
|
|
});
|
|
|
|
test('Claude install uses link_claude_skill_dirs', () => {
|
|
// The Claude install section (section 4) should use the Claude function
|
|
const claudeSection = setupContent.slice(
|
|
setupContent.indexOf('# 4. Install for Claude'),
|
|
setupContent.indexOf('# 5. Install for Codex')
|
|
);
|
|
expect(claudeSection).toContain('link_claude_skill_dirs');
|
|
expect(claudeSection).not.toContain('link_codex_skill_dirs');
|
|
});
|
|
|
|
test('Codex install uses link_codex_skill_dirs', () => {
|
|
// The Codex install section (section 5) should use the Codex function
|
|
// End marker: the next numbered section header (a marker that doesn't
|
|
// exist slices to EOF and the assertion reads unrelated sections).
|
|
const codexSection = setupContent.slice(
|
|
setupContent.indexOf('# 5. Install for Codex'),
|
|
setupContent.indexOf('# 6. Install for Kiro')
|
|
);
|
|
expect(setupContent.indexOf('# 6. Install for Kiro')).toBeGreaterThan(-1);
|
|
expect(codexSection).toContain('create_codex_runtime_root');
|
|
expect(codexSection).toContain('link_codex_skill_dirs');
|
|
expect(codexSection).not.toContain('link_claude_skill_dirs');
|
|
expect(codexSection).not.toContain('_link_or_copy "$GSTACK_DIR" "$CODEX_GSTACK"');
|
|
});
|
|
|
|
test('Codex install prefers repo-local .agents/skills when setup runs from there', () => {
|
|
expect(setupContent).toContain('SKILLS_PARENT_BASENAME');
|
|
expect(setupContent).toContain('CODEX_REPO_LOCAL=0');
|
|
expect(setupContent).toContain('[ "$SKILLS_PARENT_BASENAME" = ".agents" ]');
|
|
expect(setupContent).toContain('CODEX_REPO_LOCAL=1');
|
|
expect(setupContent).toContain('CODEX_SKILLS="$INSTALL_SKILLS_DIR"');
|
|
});
|
|
|
|
test('setup separates install path from source path for symlinked repo-local installs', () => {
|
|
expect(setupContent).toContain('INSTALL_GSTACK_DIR=');
|
|
expect(setupContent).toContain('SOURCE_GSTACK_DIR=');
|
|
expect(setupContent).toContain('INSTALL_SKILLS_DIR=');
|
|
expect(setupContent).toContain('CODEX_GSTACK="$INSTALL_GSTACK_DIR"');
|
|
expect(setupContent).toContain('link_codex_skill_dirs "$SOURCE_GSTACK_DIR" "$CODEX_SKILLS"');
|
|
});
|
|
|
|
test('Codex installs always create sidecar runtime assets for the real skill target', () => {
|
|
expect(setupContent).toContain('if [ "$INSTALL_CODEX" -eq 1 ]; then');
|
|
expect(setupContent).toContain('create_agents_sidecar "$SOURCE_GSTACK_DIR"');
|
|
});
|
|
|
|
test('link_codex_skill_dirs reads from .agents/skills/', () => {
|
|
// The Codex link function must reference .agents/skills for generated Codex skills
|
|
const fnStart = setupContent.indexOf('link_codex_skill_dirs()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('.agents/skills');
|
|
expect(fnBody).toContain('gstack*');
|
|
});
|
|
|
|
test('link_claude_skill_dirs creates real directories with absolute SKILL.md symlinks', () => {
|
|
// Claude links should be real directories with absolute SKILL.md symlinks
|
|
// to ensure Claude Code discovers them as top-level skills (not nested under gstack/)
|
|
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('mkdir -p "$target"');
|
|
// v1.36.0.0: routes through _link_or_copy helper for Windows fallback (cp on MSYS2/Git Bash).
|
|
// v1.67 (#2569): the source is render-aware — canonical SKILL.md, or the
|
|
// rendered :user variant from ${GSTACK_HOME}/render/claude when present.
|
|
expect(fnBody).toContain('_skill_md_src="$gstack_dir/$dir_name/SKILL.md"');
|
|
expect(fnBody).toContain('_link_or_copy "$_skill_md_src" "$target/SKILL.md"');
|
|
});
|
|
|
|
// REGRESSION: cleanup functions must handle both old symlinks AND new real-directory pattern
|
|
test('cleanup functions handle real directories with symlinked SKILL.md', () => {
|
|
// cleanup_old_claude_symlinks must detect and remove real dirs with SKILL.md symlinks
|
|
const cleanupOldStart = setupContent.indexOf('cleanup_old_claude_symlinks()');
|
|
const cleanupOldEnd = setupContent.indexOf('}', setupContent.indexOf('cleaned up old', cleanupOldStart));
|
|
const cleanupOldBody = setupContent.slice(cleanupOldStart, cleanupOldEnd);
|
|
expect(cleanupOldBody).toContain('-d "$old_target"');
|
|
expect(cleanupOldBody).toContain('-L "$old_target/SKILL.md"');
|
|
expect(cleanupOldBody).toContain('rm -rf "$old_target"');
|
|
|
|
// cleanup_prefixed_claude_symlinks must also handle the new pattern
|
|
const cleanupPrefixedStart = setupContent.indexOf('cleanup_prefixed_claude_symlinks()');
|
|
const cleanupPrefixedEnd = setupContent.indexOf('}', setupContent.indexOf('cleaned up prefixed', cleanupPrefixedStart));
|
|
const cleanupPrefixedBody = setupContent.slice(cleanupPrefixedStart, cleanupPrefixedEnd);
|
|
expect(cleanupPrefixedBody).toContain('-d "$prefixed_target"');
|
|
expect(cleanupPrefixedBody).toContain('-L "$prefixed_target/SKILL.md"');
|
|
expect(cleanupPrefixedBody).toContain('rm -rf "$prefixed_target"');
|
|
});
|
|
|
|
// REGRESSION: link function must upgrade old directory symlinks
|
|
test('link_claude_skill_dirs removes old directory symlinks before creating real dirs', () => {
|
|
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
// Must check for and remove old symlinks before mkdir
|
|
expect(fnBody).toContain('if [ -L "$target" ]');
|
|
expect(fnBody).toContain('rm -f "$target"');
|
|
});
|
|
|
|
test('setup links root gstack skill through a thin Claude wrapper alias', () => {
|
|
const fnStart = setupContent.indexOf('link_claude_root_skill_alias()');
|
|
const fnEnd = setupContent.indexOf('# ─── Helper: remove old unprefixed Claude skill entries', fnStart);
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('_gstack-command');
|
|
// #2511: the alias must be a rewritten COPY (unique frontmatter name),
|
|
// never a verbatim symlink of the canonical SKILL.md.
|
|
expect(fnBody).toContain('_install_alias_skill_md "$gstack_dir/SKILL.md" "$target" "_gstack-command"');
|
|
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir/SKILL.md"');
|
|
|
|
const claudeSection = setupContent.slice(
|
|
setupContent.indexOf('# 4. Install for Claude'),
|
|
setupContent.indexOf('# 5. Install for Codex')
|
|
);
|
|
expect(claudeSection).toContain('link_claude_root_skill_alias "$SOURCE_GSTACK_DIR" "$INSTALL_SKILLS_DIR"');
|
|
});
|
|
|
|
test('setup supports --host auto|claude|codex|kiro|opencode|cursor; slate is informational', () => {
|
|
expect(setupContent).toContain('--host');
|
|
// #2361: slate moved OUT of the install accept-list (it was accepted but
|
|
// never dispatched — a silent exit-0 no-op) into an informational arm.
|
|
expect(setupContent).toContain('claude|codex|kiro|factory|opencode|cursor|auto');
|
|
expect(setupContent).toMatch(/^ {2}slate\)/m);
|
|
});
|
|
|
|
test('auto mode detects claude, codex, kiro, and opencode binaries', () => {
|
|
expect(setupContent).toContain('command -v claude');
|
|
expect(setupContent).toContain('command -v codex');
|
|
expect(setupContent).toContain('command -v kiro-cli');
|
|
expect(setupContent).toContain('command -v opencode');
|
|
});
|
|
|
|
// T1: Sidecar skip guard — prevents .agents/skills/gstack from being linked as a skill
|
|
test('link_codex_skill_dirs skips the gstack sidecar directory', () => {
|
|
const fnStart = setupContent.indexOf('link_codex_skill_dirs()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('[ "$skill_name" = "gstack" ] && continue');
|
|
});
|
|
|
|
// T2: Dynamic $GSTACK_ROOT paths in generated Codex preambles
|
|
test('generated Codex preambles use dynamic GSTACK_ROOT paths', () => {
|
|
// Read the module-level out-dir render (always present).
|
|
const codexSkillDir = path.join(EXTERNAL_OUT, '.agents', 'skills', 'gstack-ship');
|
|
const content = fs.readFileSync(path.join(codexSkillDir, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('GSTACK_ROOT=');
|
|
expect(content).toContain('$GSTACK_BIN/');
|
|
});
|
|
|
|
test('setup supports --host kiro with install section and sed rewrites', () => {
|
|
expect(setupContent).toContain('INSTALL_KIRO=');
|
|
expect(setupContent).toContain('kiro-cli');
|
|
expect(setupContent).toContain('KIRO_SKILLS=');
|
|
expect(setupContent).toContain('~/.kiro/skills/gstack');
|
|
expect(setupContent).toContain('$KIRO_GSTACK/lib');
|
|
});
|
|
|
|
test('setup supports --host opencode with install section and OpenCode skill path vars', () => {
|
|
expect(setupContent).toContain('INSTALL_OPENCODE=');
|
|
expect(setupContent).toContain('OPENCODE_SKILLS="$HOME/.config/opencode/skills"');
|
|
expect(setupContent).toContain('OPENCODE_GSTACK="$OPENCODE_SKILLS/gstack"');
|
|
});
|
|
|
|
// --host cursor full install slice (#1358, PR #2547 by @szsunyuan re-derived)
|
|
test('auto mode detects Cursor via binary or ~/.cursor directory', () => {
|
|
expect(setupContent).toContain('command -v cursor');
|
|
expect(setupContent).toContain('[ -d "$HOME/.cursor" ] && INSTALL_CURSOR=1');
|
|
});
|
|
|
|
test('setup supports --host cursor with install section and Cursor skill path vars', () => {
|
|
expect(setupContent).toContain('INSTALL_CURSOR=');
|
|
expect(setupContent).toContain('CURSOR_SKILLS="$HOME/.cursor/skills"');
|
|
expect(setupContent).toContain('CURSOR_GSTACK="$CURSOR_SKILLS/gstack"');
|
|
expect(setupContent).toContain('create_cursor_runtime_root');
|
|
expect(setupContent).toContain('create_cursor_sidecar');
|
|
expect(setupContent).toContain('link_cursor_skill_dirs');
|
|
expect(setupContent).toContain('gstack ready (cursor).');
|
|
});
|
|
|
|
test('create_cursor_runtime_root exposes only Cursor runtime assets', () => {
|
|
const fnStart = setupContent.indexOf('create_cursor_runtime_root()');
|
|
const fnEnd = setupContent.indexOf('create_cursor_sidecar()', fnStart);
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('gstack/SKILL.md');
|
|
expect(fnBody).toContain('browse/dist');
|
|
expect(fnBody).toContain('browse/bin');
|
|
expect(fnBody).toContain('gstack-upgrade/SKILL.md');
|
|
expect(fnBody).toContain('checklist.md');
|
|
expect(fnBody).toContain('TODOS-format.md');
|
|
// bin scripts import ../lib — the two must travel together.
|
|
expect(fnBody).toContain('$cursor_gstack/lib');
|
|
expect(fnBody).not.toContain('design-checklist.md');
|
|
expect(fnBody).not.toContain('greptile-triage.md');
|
|
expect(fnBody).not.toContain('review/specialists');
|
|
expect(fnBody).not.toContain('qa/templates');
|
|
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir" "$cursor_gstack"');
|
|
});
|
|
|
|
test('create_cursor_sidecar plants runtime assets without wiping generated SKILL.md', () => {
|
|
const fnStart = setupContent.indexOf('create_cursor_sidecar()');
|
|
const fnEnd = setupContent.indexOf('link_cursor_skill_dirs()', fnStart);
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('.cursor/skills/gstack');
|
|
expect(fnBody).toContain('bin');
|
|
expect(fnBody).toContain('browse/dist');
|
|
expect(fnBody).toContain('browse/bin');
|
|
expect(fnBody).toContain('ETHOS.md');
|
|
expect(fnBody).not.toContain('rm -rf');
|
|
});
|
|
|
|
test('link_cursor_skill_dirs skips the gstack runtime root directory', () => {
|
|
const fnStart = setupContent.indexOf('link_cursor_skill_dirs()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('[ "$skill_name" = "gstack" ] && continue');
|
|
// #2444-aware guard: Windows bypass, else only replace symlink-or-missing.
|
|
expect(fnBody).toContain('[ "$IS_WINDOWS" -eq 1 ] || [ -L "$target" ] || [ ! -e "$target" ]');
|
|
});
|
|
|
|
// #2142 deleted existing ~/.cursor/skills/<name> dirs with `rm -rf "$target"`
|
|
// before relinking. That can wipe unowned Cursor skills. Only replace a
|
|
// symlink or a missing path; never the whole skills directory.
|
|
test('link_cursor_skill_dirs does not delete unowned Cursor skill directories', () => {
|
|
const fnStart = setupContent.indexOf('link_cursor_skill_dirs()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).not.toContain('rm -rf "$target"');
|
|
expect(fnBody).not.toContain('rm -rf "$skills_dir"');
|
|
expect(setupContent).not.toContain('rm -rf "$CURSOR_SKILLS"');
|
|
});
|
|
|
|
test('Cursor install links generated skills before planting the sidecar', () => {
|
|
const cursorInstall = setupContent.slice(
|
|
setupContent.indexOf('# 6d. Install for Cursor'),
|
|
setupContent.indexOf('# 7. Create .agents/ sidecar'),
|
|
);
|
|
const linkCall = cursorInstall.indexOf('link_cursor_skill_dirs "$SOURCE_GSTACK_DIR"');
|
|
const sidecarCall = cursorInstall.indexOf('create_cursor_sidecar "$SOURCE_GSTACK_DIR"');
|
|
expect(linkCall).toBeGreaterThan(-1);
|
|
expect(sidecarCall).toBeGreaterThan(-1);
|
|
expect(linkCall).toBeLessThan(sidecarCall);
|
|
});
|
|
|
|
test('setup installs OpenCode skills into a nested gstack runtime root', () => {
|
|
expect(setupContent).toContain('create_opencode_runtime_root');
|
|
expect(setupContent).toContain('.opencode/skills');
|
|
expect(setupContent).toContain('review/specialists');
|
|
expect(setupContent).toContain('qa/templates');
|
|
expect(setupContent).toContain('qa/references');
|
|
expect(setupContent).toContain('dx-hall-of-fame.md');
|
|
expect(setupContent).toContain('$opencode_gstack/lib');
|
|
});
|
|
|
|
test('create_agents_sidecar links runtime assets', () => {
|
|
// Sidecar must link bin with its shared lib modules, plus browse, review, qa
|
|
const fnStart = setupContent.indexOf('create_agents_sidecar()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('bin');
|
|
expect(fnBody).toContain('lib');
|
|
expect(fnBody).toContain('browse');
|
|
expect(fnBody).toContain('review');
|
|
expect(fnBody).toContain('qa');
|
|
});
|
|
|
|
test('create_codex_runtime_root exposes only runtime assets', () => {
|
|
const fnStart = setupContent.indexOf('create_codex_runtime_root()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', setupContent.indexOf('review/', fnStart)));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('gstack/SKILL.md');
|
|
expect(fnBody).toContain('$codex_gstack/lib');
|
|
expect(fnBody).toContain('browse/dist');
|
|
expect(fnBody).toContain('browse/bin');
|
|
expect(fnBody).toContain('gstack-upgrade/SKILL.md');
|
|
// Review runtime assets (individual files, not the whole dir)
|
|
expect(fnBody).toContain('checklist.md');
|
|
expect(fnBody).toContain('design-checklist.md');
|
|
expect(fnBody).toContain('greptile-triage.md');
|
|
expect(fnBody).toContain('TODOS-format.md');
|
|
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir" "$codex_gstack"');
|
|
});
|
|
|
|
test('create_factory_runtime_root links shared lib modules beside bin', () => {
|
|
const fnStart = setupContent.indexOf('create_factory_runtime_root()');
|
|
const fnEnd = setupContent.indexOf('create_opencode_runtime_root()', fnStart);
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('$factory_gstack/bin');
|
|
expect(fnBody).toContain('$factory_gstack/lib');
|
|
});
|
|
|
|
test('direct Codex installs are migrated out of ~/.codex/skills/gstack', () => {
|
|
expect(setupContent).toContain('migrate_direct_codex_install');
|
|
expect(setupContent).toContain('$HOME/.gstack/repos/gstack');
|
|
expect(setupContent).toContain('avoid duplicate skill discovery');
|
|
});
|
|
|
|
// --- Symlink prefix tests (PR #503) ---
|
|
|
|
test('link_claude_skill_dirs applies gstack- prefix by default', () => {
|
|
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('SKILL_PREFIX');
|
|
expect(fnBody).toContain('link_name="gstack-$skill_name"');
|
|
});
|
|
|
|
test('link_claude_skill_dirs preserves already-prefixed dirs', () => {
|
|
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
// gstack-* dirs should keep their name (e.g., gstack-upgrade stays gstack-upgrade)
|
|
expect(fnBody).toContain('gstack-*) link_name="$skill_name"');
|
|
});
|
|
|
|
test('setup supports --no-prefix flag', () => {
|
|
expect(setupContent).toContain('--no-prefix');
|
|
expect(setupContent).toContain('SKILL_PREFIX=0');
|
|
});
|
|
|
|
test('cleanup_old_claude_symlinks removes only gstack-pointing symlinks', () => {
|
|
expect(setupContent).toContain('cleanup_old_claude_symlinks');
|
|
const fnStart = setupContent.indexOf('cleanup_old_claude_symlinks()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('removed[@]}', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
// Should check readlink before removing
|
|
expect(fnBody).toContain('readlink');
|
|
expect(fnBody).toContain('gstack/*');
|
|
// Should skip already-prefixed dirs
|
|
expect(fnBody).toContain('gstack-*) continue');
|
|
});
|
|
|
|
test('cleanup runs before link when prefix is enabled', () => {
|
|
// In the Claude install section, cleanup should happen before linking
|
|
const claudeInstallSection = setupContent.slice(
|
|
setupContent.indexOf('INSTALL_CLAUDE'),
|
|
setupContent.lastIndexOf('link_claude_skill_dirs')
|
|
);
|
|
expect(claudeInstallSection).toContain('cleanup_old_claude_symlinks');
|
|
});
|
|
|
|
// --- Persistent config + interactive prompt tests ---
|
|
|
|
test('setup reads skill_prefix from config', () => {
|
|
expect(setupContent).toContain('get skill_prefix');
|
|
expect(setupContent).toContain('GSTACK_CONFIG');
|
|
});
|
|
|
|
test('setup supports --prefix flag', () => {
|
|
expect(setupContent).toContain('--prefix)');
|
|
expect(setupContent).toContain('SKILL_PREFIX=1; SKILL_PREFIX_FLAG=1');
|
|
});
|
|
|
|
test('--prefix and --no-prefix persist to config', () => {
|
|
expect(setupContent).toContain('set skill_prefix');
|
|
});
|
|
|
|
test('interactive prompt shows when no config', () => {
|
|
expect(setupContent).toContain('Short names');
|
|
expect(setupContent).toContain('Namespaced');
|
|
expect(setupContent).toContain('Choice [1/2]');
|
|
});
|
|
|
|
test('non-TTY defaults to flat names', () => {
|
|
// Should check if stdin is a TTY before prompting
|
|
expect(setupContent).toContain('-t 0');
|
|
});
|
|
|
|
test('cleanup_prefixed_claude_symlinks exists and uses readlink', () => {
|
|
expect(setupContent).toContain('cleanup_prefixed_claude_symlinks');
|
|
const fnStart = setupContent.indexOf('cleanup_prefixed_claude_symlinks()');
|
|
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('removed[@]}', fnStart));
|
|
const fnBody = setupContent.slice(fnStart, fnEnd);
|
|
expect(fnBody).toContain('readlink');
|
|
expect(fnBody).toContain('gstack-$skill_name');
|
|
});
|
|
|
|
test('reverse cleanup runs before link when prefix is disabled', () => {
|
|
const claudeInstallSection = setupContent.slice(
|
|
setupContent.indexOf('INSTALL_CLAUDE'),
|
|
setupContent.lastIndexOf('link_claude_skill_dirs')
|
|
);
|
|
expect(claudeInstallSection).toContain('cleanup_prefixed_claude_symlinks');
|
|
});
|
|
|
|
test('welcome message references SKILL_PREFIX', () => {
|
|
// gstack-upgrade is always called gstack-upgrade (it's the actual dir name)
|
|
// but the welcome section should exist near the prefix logic
|
|
expect(setupContent).toContain('Run /gstack-upgrade anytime');
|
|
});
|
|
});
|
|
|
|
describe('discover-skills hidden directory filtering', () => {
|
|
test('discoverTemplates skips dot-prefixed directories', () => {
|
|
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-discover-'));
|
|
try {
|
|
// Create a hidden dir with a template (should be excluded)
|
|
fs.mkdirSync(path.join(tmpDir, '.hidden'), { recursive: true });
|
|
fs.writeFileSync(path.join(tmpDir, '.hidden', 'SKILL.md.tmpl'), '---\nname: evil\n---\ntest');
|
|
// Create a visible dir with a template (should be included)
|
|
fs.mkdirSync(path.join(tmpDir, 'visible'), { recursive: true });
|
|
fs.writeFileSync(path.join(tmpDir, 'visible', 'SKILL.md.tmpl'), '---\nname: good\n---\ntest');
|
|
|
|
const { discoverTemplates } = require('../scripts/discover-skills');
|
|
const results = discoverTemplates(tmpDir);
|
|
const dirs = results.map((r: { tmpl: string }) => r.tmpl);
|
|
|
|
expect(dirs).toContain('visible/SKILL.md.tmpl');
|
|
expect(dirs).not.toContain('.hidden/SKILL.md.tmpl');
|
|
} finally {
|
|
fs.rmSync(tmpDir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('telemetry', () => {
|
|
test('telemetry start block lives in gstack-skill-start; render notes the handoff keys', () => {
|
|
// The start-block bash moved into the script (Phase 1): it reads the
|
|
// config, mints the session identity, and echoes the STATUS keys.
|
|
expect(SKILL_START_SCRIPT).toContain('_TEL_START=$(date +%s)');
|
|
expect(SKILL_START_SCRIPT).toContain('_SESSION_ID=');
|
|
expect(SKILL_START_SCRIPT).toContain('echo "TELEMETRY:');
|
|
expect(SKILL_START_SCRIPT).toContain('echo "TEL_PROMPTED:');
|
|
expect(SKILL_START_SCRIPT).toMatch(/gstack-config" get telemetry/);
|
|
// The render must tell the model to carry SESSION_ID/TEL_START to skill end.
|
|
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('SESSION_ID');
|
|
expect(content).toContain('TEL_START');
|
|
});
|
|
|
|
test('telemetry opt-in prompt lives in gstack-skill-start (marker-gated emit)', () => {
|
|
// Token-reduction Phase 2: the one-time consent prompt left the renders
|
|
// (absence pinned by test/onboarding-moved-literals.test.ts); the script
|
|
// gates it on the marker files and emits it as a GSTACK_INSTRUCTION block
|
|
// with all three config-set outcomes and the ack INSIDE the block.
|
|
expect(SKILL_START_SCRIPT).toContain(
|
|
'if [ "$_TEL_PROMPTED" = "no" ] && [ "$_LAKE_SEEN" = "yes" ]; then',
|
|
);
|
|
expect(SKILL_START_SCRIPT).toContain('_emit_block telemetry-prompt');
|
|
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry community');
|
|
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry anonymous');
|
|
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry off');
|
|
expect(SKILL_START_SCRIPT).toContain('touch "$_GH/.telemetry-prompted"');
|
|
});
|
|
|
|
test('generated SKILL.md contains telemetry epilogue (one gstack-skill-end call)', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Telemetry (run last)');
|
|
expect(content).toContain('gstack-skill-end --skill "gstack" --outcome OUTCOME');
|
|
expect(content).toContain('--tel-start "TEL_START"');
|
|
expect(content).toContain('PLAN MODE EXCEPTION');
|
|
// The duration math + remote-log dispatch moved into gstack-skill-end.
|
|
expect(SKILL_END_SCRIPT).toContain('_TEL_END');
|
|
expect(SKILL_END_SCRIPT).toContain('_TEL_DUR');
|
|
expect(SKILL_END_SCRIPT).toContain('SKILL_NAME');
|
|
expect(SKILL_END_SCRIPT).toContain('OUTCOME');
|
|
expect(SKILL_END_SCRIPT).toContain('gstack-telemetry-log');
|
|
});
|
|
|
|
test('pending marker handling lives in the scripts', () => {
|
|
// gstack-skill-start finalizes stale markers; gstack-skill-end clears the
|
|
// session's own marker.
|
|
expect(SKILL_START_SCRIPT).toContain("-name '.pending-*'");
|
|
expect(SKILL_START_SCRIPT).toContain('_pending_finalize');
|
|
expect(SKILL_END_SCRIPT).toContain('.pending-$SESSION_ID');
|
|
});
|
|
|
|
test('telemetry blocks appear in all skill files that use PREAMBLE', () => {
|
|
const skills = ['qa', 'ship', 'review', 'plan-ceo-review', 'plan-eng-review', 'retro'];
|
|
for (const skill of skills) {
|
|
const skillPath = path.join(ROOT, skill, 'SKILL.md');
|
|
if (fs.existsSync(skillPath)) {
|
|
const content = fs.readFileSync(skillPath, 'utf-8');
|
|
expect(content).toContain('Telemetry (run last)');
|
|
expect(content).toContain(`gstack-skill-end --skill "${skill}"`);
|
|
expect(content).toContain('--tel-start "TEL_START"');
|
|
}
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('community fixes wave', () => {
|
|
// Helper to get all generated SKILL.md files
|
|
function getAllSkillMds(): Array<{ name: string; content: string }> {
|
|
const results: Array<{ name: string; content: string }> = [];
|
|
const rootPath = path.join(ROOT, 'SKILL.md');
|
|
if (fs.existsSync(rootPath)) {
|
|
results.push({ name: 'root', content: fs.readFileSync(rootPath, 'utf-8') });
|
|
}
|
|
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
|
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
|
const skillPath = path.join(ROOT, entry.name, 'SKILL.md');
|
|
if (fs.existsSync(skillPath)) {
|
|
results.push({ name: entry.name, content: fs.readFileSync(skillPath, 'utf-8') });
|
|
}
|
|
}
|
|
return results;
|
|
}
|
|
|
|
// #594 — Discoverability: every SKILL.md.tmpl description contains "gstack"
|
|
test('every SKILL.md.tmpl description contains "gstack"', () => {
|
|
for (const skill of ALL_SKILLS) {
|
|
const tmplPath = skill.dir === '.' ? path.join(ROOT, 'SKILL.md.tmpl') : path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
|
const content = fs.readFileSync(tmplPath, 'utf-8');
|
|
const desc = extractDescription(content);
|
|
expect(desc.toLowerCase()).toContain('gstack');
|
|
}
|
|
});
|
|
|
|
// #594 — Discoverability: first line of each description is under 120 chars
|
|
test('every SKILL.md.tmpl description first line is under 120 chars', () => {
|
|
for (const skill of ALL_SKILLS) {
|
|
const tmplPath = skill.dir === '.' ? path.join(ROOT, 'SKILL.md.tmpl') : path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
|
const content = fs.readFileSync(tmplPath, 'utf-8');
|
|
const desc = extractDescription(content);
|
|
const firstLine = desc.split('\n')[0];
|
|
expect(firstLine.length).toBeLessThanOrEqual(120);
|
|
}
|
|
});
|
|
|
|
// #573 — Feature signals: ship/SKILL.md contains feature signal detection
|
|
test('ship/SKILL.md contains feature signal detection in Step 4', () => {
|
|
const content = readShipUnion();
|
|
expect(content.toLowerCase()).toContain('feature signal');
|
|
});
|
|
|
|
// #510 — Context warnings: no SKILL.md contains "running low on context"
|
|
test('no generated SKILL.md contains "running low on context"', () => {
|
|
const skills = getAllSkillMds();
|
|
for (const { name, content } of skills) {
|
|
expect(content).not.toContain('running low on context');
|
|
}
|
|
});
|
|
|
|
// #510 — Context warnings: plan-eng-review has explicit anti-warning
|
|
test('plan-eng-review/SKILL.md contains "Do not preemptively warn"', () => {
|
|
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
|
expect(content).toContain('Do not preemptively warn');
|
|
});
|
|
|
|
// #474 — Safety Net: no SKILL.md uses find with -delete
|
|
test('no generated SKILL.md contains find with -delete flag', () => {
|
|
const skills = getAllSkillMds();
|
|
for (const { name, content } of skills) {
|
|
// Match find commands that use -delete (but not prose mentioning the word "delete")
|
|
const lines = content.split('\n');
|
|
for (const line of lines) {
|
|
if (line.includes('find ') && line.includes('-delete')) {
|
|
throw new Error(`${name}/SKILL.md contains find with -delete: ${line.trim()}`);
|
|
}
|
|
}
|
|
}
|
|
});
|
|
|
|
// #467 — Telemetry: preamble JSONL writes are gated by telemetry setting
|
|
test('preamble JSONL writes are inside telemetry conditional', () => {
|
|
const preamble = fs.readFileSync(path.join(ROOT, 'scripts/resolvers/preamble.ts'), 'utf-8');
|
|
// Find all skill-usage.jsonl write lines
|
|
const lines = preamble.split('\n');
|
|
for (let i = 0; i < lines.length; i++) {
|
|
if (lines[i].includes('skill-usage.jsonl') && lines[i].includes('>>')) {
|
|
// Look backwards for a telemetry conditional within 5 lines
|
|
let foundConditional = false;
|
|
for (let j = i - 1; j >= Math.max(0, i - 5); j--) {
|
|
if (lines[j].includes('_TEL') && lines[j].includes('off')) {
|
|
foundConditional = true;
|
|
break;
|
|
}
|
|
}
|
|
expect(foundConditional).toBe(true);
|
|
}
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('codex commands must not use inline $(git rev-parse --show-toplevel) for cwd', () => {
|
|
// Regression test: inline $(git rev-parse --show-toplevel) in codex exec -C
|
|
// or codex review without cd evaluates in whatever cwd the background shell
|
|
// inherits, which may be a different project in Conductor workspaces.
|
|
// The fix is to resolve _REPO_ROOT eagerly at the top of each bash block.
|
|
|
|
// Scan all source files that could contain codex commands
|
|
// Use Bun.Glob to avoid ELOOP from .claude/skills/gstack symlink back to ROOT
|
|
const tmplGlob = new Bun.Glob('**/*.tmpl');
|
|
const sourceFiles = [
|
|
...Array.from(tmplGlob.scanSync({ cwd: ROOT, followSymlinks: false })),
|
|
...fs.readdirSync(path.join(ROOT, 'scripts/resolvers'))
|
|
.filter(f => f.endsWith('.ts'))
|
|
.map(f => `scripts/resolvers/${f}`),
|
|
'scripts/gen-skill-docs.ts',
|
|
];
|
|
|
|
test('no codex exec command uses inline $(git rev-parse --show-toplevel) in -C flag', () => {
|
|
const violations: string[] = [];
|
|
for (const rel of sourceFiles) {
|
|
const abs = path.join(ROOT, rel);
|
|
if (!fs.existsSync(abs)) continue;
|
|
const content = fs.readFileSync(abs, 'utf-8');
|
|
const lines = content.split('\n');
|
|
for (let i = 0; i < lines.length; i++) {
|
|
const line = lines[i];
|
|
if (line.includes('codex exec') && line.includes('-C') && line.includes('$(git rev-parse --show-toplevel)')) {
|
|
violations.push(`${rel}:${i + 1}`);
|
|
}
|
|
}
|
|
}
|
|
expect(violations).toEqual([]);
|
|
});
|
|
|
|
test('no generated SKILL.md has codex exec with inline $(git rev-parse --show-toplevel) in -C flag', () => {
|
|
const violations: string[] = [];
|
|
const skillMdGlob = new Bun.Glob('**/SKILL.md');
|
|
const skillMdFiles = Array.from(skillMdGlob.scanSync({ cwd: ROOT, followSymlinks: false }));
|
|
for (const rel of skillMdFiles) {
|
|
const abs = path.join(ROOT, rel);
|
|
if (!fs.existsSync(abs)) continue;
|
|
const content = fs.readFileSync(abs, 'utf-8');
|
|
const lines = content.split('\n');
|
|
for (let i = 0; i < lines.length; i++) {
|
|
const line = lines[i];
|
|
if (line.includes('codex exec') && line.includes('-C') && line.includes('$(git rev-parse --show-toplevel)')) {
|
|
violations.push(`${rel}:${i + 1}`);
|
|
}
|
|
}
|
|
}
|
|
expect(violations).toEqual([]);
|
|
});
|
|
|
|
test('codex review commands must be preceded by cd "$_REPO_ROOT" (no -C support)', () => {
|
|
// codex review does not support -C, so the pattern must be:
|
|
// _REPO_ROOT=$(git rev-parse --show-toplevel) || { ... }
|
|
// cd "$_REPO_ROOT"
|
|
// codex review ...
|
|
// NOT: codex review ... with inline $(git rev-parse --show-toplevel)
|
|
const allFiles = [
|
|
...Array.from(tmplGlob.scanSync({ cwd: ROOT, followSymlinks: false })),
|
|
...Array.from(new Bun.Glob('**/SKILL.md').scanSync({ cwd: ROOT, followSymlinks: false })),
|
|
...fs.readdirSync(path.join(ROOT, 'scripts/resolvers'))
|
|
.filter(f => f.endsWith('.ts'))
|
|
.map(f => `scripts/resolvers/${f}`),
|
|
'scripts/gen-skill-docs.ts',
|
|
];
|
|
const violations: string[] = [];
|
|
for (const rel of allFiles) {
|
|
const abs = path.join(ROOT, rel);
|
|
if (!fs.existsSync(abs)) continue;
|
|
const content = fs.readFileSync(abs, 'utf-8');
|
|
const lines = content.split('\n');
|
|
for (let i = 0; i < lines.length; i++) {
|
|
const line = lines[i];
|
|
// Skip non-executable lines (markdown table cells, prose references)
|
|
if (line.includes('|') && line.includes('`/codex review`')) continue;
|
|
if (line.includes('`codex review`')) continue;
|
|
// Check for codex review with inline $(git rev-parse)
|
|
if (line.includes('codex review') && line.includes('$(git rev-parse --show-toplevel)')) {
|
|
violations.push(`${rel}:${i + 1} — inline git rev-parse in codex review`);
|
|
}
|
|
}
|
|
}
|
|
expect(violations).toEqual([]);
|
|
});
|
|
|
|
test('codex review commands take their scope from a flag, never from prompt text', () => {
|
|
// `codex review` scope comes ONLY from --base/--commit/--uncommitted. The
|
|
// positional [PROMPT] is mutually exclusive with all three (#1428, #1479),
|
|
// and a prompt-only `codex review` silently falls back to the *uncommitted
|
|
// working-tree* scope (`git status --short; git diff`) — so describing the
|
|
// diff range in prompt text produces a confident review of the wrong
|
|
// changes, with no error. Both halves are pinned here:
|
|
// (a) every `codex review` invocation carries a scope flag, and
|
|
// (b) no invocation puts a positional prompt in front of that flag.
|
|
//
|
|
// This does NOT apply to `codex exec`, which is agentic and really does run
|
|
// the git command it's told to — the adversarial pass legitimately scopes
|
|
// itself in prompt text.
|
|
const checkedFiles = [
|
|
'scripts/resolvers/review.ts',
|
|
'review/SKILL.md',
|
|
'ship/SKILL.md',
|
|
'codex/SKILL.md.tmpl',
|
|
'codex/SKILL.md',
|
|
// codex's scoped invocations moved into the carved review-mode section
|
|
// (T9) — keep sweeping both the .tmpl source and the generated section.
|
|
'codex/sections/review-mode.md.tmpl',
|
|
'codex/sections/review-mode.md',
|
|
];
|
|
|
|
const violations: string[] = [];
|
|
for (const rel of checkedFiles) {
|
|
// ship's codex/adversarial command moved into sections/adversarial.md (T9 carve).
|
|
const content = rel === 'ship/SKILL.md' ? readShipUnion() : fs.readFileSync(path.join(ROOT, rel), 'utf-8');
|
|
const lines = content.split('\n');
|
|
for (let i = 0; i < lines.length; i++) {
|
|
const line = lines[i];
|
|
// Only inspect real shell invocations, not prose mentioning the command.
|
|
if (line.includes('`codex review`')) continue;
|
|
const match = line.match(/(?:^|[;&|]\s*|\s)codex\s+review\b(.*)$/);
|
|
if (!match) continue;
|
|
const rest = match[1];
|
|
const scopeFlag = /--base\b|--commit\b|--uncommitted\b/;
|
|
if (!scopeFlag.test(rest)) {
|
|
// A quoted prompt with no scope flag is the silent-wrong-scope bug.
|
|
if (/^\s*["'$]/.test(rest)) {
|
|
violations.push(`${rel}:${i + 1} — prompt-only codex review (falls back to working-tree scope)`);
|
|
}
|
|
continue;
|
|
}
|
|
const beforeFlag = rest.split(scopeFlag)[0].trim();
|
|
if (/^["'$]|^--\s*["']/.test(beforeFlag)) {
|
|
violations.push(`${rel}:${i + 1} — positional prompt passed alongside a scope flag`);
|
|
}
|
|
}
|
|
}
|
|
expect(violations).toEqual([]);
|
|
});
|
|
});
|
|
|
|
// ─── Learnings + Confidence Resolver Tests ─────────────────────
|
|
|
|
describe('LEARNINGS_SEARCH resolver', () => {
|
|
const SEARCH_SKILLS = ['review', 'ship', 'plan-eng-review', 'investigate', 'office-hours', 'plan-ceo-review'];
|
|
|
|
for (const skill of SEARCH_SKILLS) {
|
|
test(`${skill} generated SKILL.md contains learnings search`, () => {
|
|
const content = readSkillUnion(skill); // ship: moved to sections/plan-completion.md
|
|
expect(content).toContain('Prior Learnings');
|
|
expect(content).toContain('gstack-learnings-search');
|
|
});
|
|
}
|
|
|
|
test('learnings search includes cross-project config check', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('cross_project_learnings');
|
|
expect(content).toContain('--cross-project');
|
|
});
|
|
|
|
test('learnings search includes AskUserQuestion for first-time cross-project opt-in', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Enable cross-project learnings');
|
|
expect(content).toContain('project-scoped only');
|
|
});
|
|
|
|
test('learnings search mentions prior learning applied display format', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Prior learning applied');
|
|
});
|
|
});
|
|
|
|
describe('LEARNINGS_LOG resolver', () => {
|
|
const LOG_SKILLS = ['review', 'retro', 'investigate'];
|
|
|
|
for (const skill of LOG_SKILLS) {
|
|
test(`${skill} generated SKILL.md contains learnings log`, () => {
|
|
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Capture Learnings');
|
|
expect(content).toContain('gstack-learnings-log');
|
|
});
|
|
}
|
|
|
|
test('learnings log documents all type values', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
for (const type of ['pattern', 'pitfall', 'preference', 'architecture', 'tool']) {
|
|
expect(content).toContain(type);
|
|
}
|
|
});
|
|
|
|
test('learnings log documents all source values', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
for (const source of ['observed', 'user-stated', 'inferred', 'cross-model']) {
|
|
expect(content).toContain(source);
|
|
}
|
|
});
|
|
|
|
test('learnings log includes files field for staleness detection', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('"files"');
|
|
expect(content).toContain('staleness detection');
|
|
});
|
|
});
|
|
|
|
describe('CONFIDENCE_CALIBRATION resolver', () => {
|
|
const CONFIDENCE_SKILLS = ['review', 'ship', 'plan-eng-review', 'cso'];
|
|
|
|
for (const skill of CONFIDENCE_SKILLS) {
|
|
test(`${skill} generated SKILL.md contains confidence calibration`, () => {
|
|
const content = readSkillUnion(skill); // ship: moved to sections/review-army.md
|
|
expect(content).toContain('Confidence Calibration');
|
|
expect(content).toContain('confidence score');
|
|
});
|
|
}
|
|
|
|
test('confidence calibration includes scoring rubric with all tiers', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('9-10');
|
|
expect(content).toContain('7-8');
|
|
expect(content).toContain('5-6');
|
|
expect(content).toContain('3-4');
|
|
expect(content).toContain('1-2');
|
|
});
|
|
|
|
test('confidence calibration includes display rules', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('Show normally');
|
|
expect(content).toContain('Suppress from main report');
|
|
});
|
|
|
|
test('confidence calibration includes finding format example', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('[P1] (confidence:');
|
|
expect(content).toContain('SQL injection');
|
|
});
|
|
|
|
test('confidence calibration includes calibration learning feedback loop', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('calibration event');
|
|
expect(content).toContain('Log the corrected pattern');
|
|
});
|
|
|
|
test('skills without confidence calibration do NOT contain it', () => {
|
|
// office-hours and retro do NOT use confidence calibration
|
|
for (const skill of ['office-hours', 'retro']) {
|
|
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
expect(content).not.toContain('## Confidence Calibration');
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('gen-skill-docs prefix warning (#620/#578)', () => {
|
|
const { execSync } = require('child_process');
|
|
|
|
test('warns about skill_prefix when config has prefix=true', () => {
|
|
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-prefix-warn-'));
|
|
try {
|
|
// Create a fake ~/.gstack/config.yaml with skill_prefix: true
|
|
const fakeHome = tmpDir;
|
|
const fakeGstack = path.join(fakeHome, '.gstack');
|
|
fs.mkdirSync(fakeGstack, { recursive: true });
|
|
fs.writeFileSync(path.join(fakeGstack, 'config.yaml'), 'skill_prefix: true\n');
|
|
|
|
// Render into an out-dir under the fixture (the warning fires on any
|
|
// non-dry-run generation) so the live tree is never rewritten.
|
|
const outDir = path.join(tmpDir, 'out');
|
|
const output = execSync(`bun run scripts/gen-skill-docs.ts --out-dir "${outDir}"`, {
|
|
cwd: ROOT,
|
|
env: { ...process.env, HOME: fakeHome },
|
|
encoding: 'utf-8',
|
|
timeout: 30000,
|
|
});
|
|
expect(output).toContain('skill_prefix is true');
|
|
expect(output).toContain('gstack-relink');
|
|
} finally {
|
|
fs.rmSync(tmpDir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
test('no warning when skill_prefix is false or absent', () => {
|
|
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-prefix-warn-'));
|
|
try {
|
|
const fakeHome = tmpDir;
|
|
const fakeGstack = path.join(fakeHome, '.gstack');
|
|
fs.mkdirSync(fakeGstack, { recursive: true });
|
|
fs.writeFileSync(path.join(fakeGstack, 'config.yaml'), 'skill_prefix: false\n');
|
|
|
|
const outDir = path.join(tmpDir, 'out');
|
|
const output = execSync(`bun run scripts/gen-skill-docs.ts --out-dir "${outDir}"`, {
|
|
cwd: ROOT,
|
|
env: { ...process.env, HOME: fakeHome },
|
|
encoding: 'utf-8',
|
|
timeout: 30000,
|
|
});
|
|
expect(output).not.toContain('skill_prefix is true');
|
|
} finally {
|
|
fs.rmSync(tmpDir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('voice-triggers processing', () => {
|
|
const { extractVoiceTriggers, processVoiceTriggers } = require('../scripts/gen-skill-docs') as {
|
|
extractVoiceTriggers: (content: string) => string[];
|
|
processVoiceTriggers: (content: string) => string;
|
|
};
|
|
|
|
test('extractVoiceTriggers parses valid YAML list', () => {
|
|
const content = `---\nname: cso\ndescription: |\n Security audit.\nvoice-triggers:\n - "see-so"\n - "security review"\n---\nBody`;
|
|
const triggers = extractVoiceTriggers(content);
|
|
expect(triggers).toEqual(['see-so', 'security review']);
|
|
});
|
|
|
|
test('extractVoiceTriggers returns [] when no field present', () => {
|
|
const content = `---\nname: qa\ndescription: |\n QA testing.\n---\nBody`;
|
|
expect(extractVoiceTriggers(content)).toEqual([]);
|
|
});
|
|
|
|
test('processVoiceTriggers appends voice triggers to description', () => {
|
|
const content = `---\nname: cso\ndescription: |\n Security audit. (gstack)\nvoice-triggers:\n - "see-so"\n - "security review"\n---\nBody`;
|
|
const result = processVoiceTriggers(content);
|
|
expect(result).toContain('Voice triggers (speech-to-text aliases): "see-so", "security review".');
|
|
});
|
|
|
|
test('processVoiceTriggers strips voice-triggers field from output', () => {
|
|
const content = `---\nname: cso\ndescription: |\n Security audit. (gstack)\nvoice-triggers:\n - "see-so"\n---\nBody`;
|
|
const result = processVoiceTriggers(content);
|
|
expect(result).not.toContain('voice-triggers:');
|
|
});
|
|
|
|
test('processVoiceTriggers returns content unchanged when no voice-triggers', () => {
|
|
const content = `---\nname: qa\ndescription: |\n QA testing.\n---\nBody`;
|
|
expect(processVoiceTriggers(content)).toBe(content);
|
|
});
|
|
|
|
test('generated CSO SKILL.md contains voice triggers in description', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
|
expect(content).toContain('"see-so"');
|
|
expect(content).toContain('Voice triggers (speech-to-text aliases):');
|
|
});
|
|
|
|
test('generated CSO SKILL.md does NOT contain raw voice-triggers field', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
|
const fmEnd = content.indexOf('\n---', 4);
|
|
const frontmatter = content.slice(0, fmEnd);
|
|
expect(frontmatter).not.toContain('voice-triggers:');
|
|
});
|
|
|
|
// Gen-time-only keys: interactive + benefits-from are read from the .tmpl by
|
|
// buildContext; the generated copy has no reader (the host reads name/
|
|
// description/allowed-tools/hooks; gbrain: is runtime-read and NOT stripped).
|
|
// Pin the strip so a stripFields refactor can't silently re-add the always-on
|
|
// frontmatter weight — mirrors the voice-triggers pins above.
|
|
test('generated SKILL.md strips gen-time-only keys the .tmpl still declares', () => {
|
|
const tmpl = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf-8');
|
|
const tmplFm = tmpl.slice(0, tmpl.indexOf('\n---', 4));
|
|
expect(tmplFm).toContain('interactive:');
|
|
expect(tmplFm).toContain('benefits-from:');
|
|
|
|
const generated = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
|
const genFm = generated.slice(0, generated.indexOf('\n---', 4));
|
|
expect(genFm).not.toContain('interactive:');
|
|
expect(genFm).not.toContain('benefits-from:');
|
|
|
|
// The runtime-read and host-read keys survive the strip.
|
|
const investigate = fs.readFileSync(path.join(ROOT, 'investigate', 'SKILL.md'), 'utf-8');
|
|
const invFm = investigate.slice(0, investigate.indexOf('\n---', 4));
|
|
expect(invFm).toContain('hooks:');
|
|
expect(invFm).toContain('gbrain:');
|
|
});
|
|
});
|
|
|
|
describe('plan-mode-info resolver (handshake-replacement)', () => {
|
|
const REVIEW_SKILLS = [
|
|
'plan-ceo-review',
|
|
'plan-eng-review',
|
|
'plan-design-review',
|
|
'plan-devex-review',
|
|
];
|
|
|
|
// Header for the vestigial handshake that was removed. If it ever reappears,
|
|
// someone accidentally re-introduced the resolver.
|
|
const HANDSHAKE_MARKER = '## Plan Mode Handshake';
|
|
// Header for the new plan-mode-info section (previously lived at the tail
|
|
// of completion-status.ts; now hoisted to position 1 of the preamble).
|
|
const PLAN_MODE_INFO_MARKER = '## Skill Invocation During Plan Mode';
|
|
|
|
test('vestigial handshake is absent from all generated Claude SKILL.md files', () => {
|
|
// Scan every generated SKILL.md under ROOT (top-level directory per skill).
|
|
// Using fs.readdirSync + filter instead of a glob so we catch any skill
|
|
// that gets added later without updating this list.
|
|
const entries = fs.readdirSync(ROOT, { withFileTypes: true });
|
|
let checked = 0;
|
|
for (const entry of entries) {
|
|
if (!entry.isDirectory()) continue;
|
|
const skillMd = path.join(ROOT, entry.name, 'SKILL.md');
|
|
if (!fs.existsSync(skillMd)) continue;
|
|
const content = fs.readFileSync(skillMd, 'utf-8');
|
|
expect(content, `handshake marker in ${entry.name}/SKILL.md`).not.toContain(HANDSHAKE_MARKER);
|
|
checked++;
|
|
}
|
|
expect(checked).toBeGreaterThan(0);
|
|
});
|
|
|
|
test('vestigial handshake is absent from non-Claude host outputs', () => {
|
|
// Non-Claude hosts render to hostSubdirs (.agents/, .openclaw/, etc). The
|
|
// plan-mode-info resolver has no host-scoping — all hosts get the new
|
|
// section, none get the old handshake. Scan every candidate host tree in
|
|
// the module-level out-dir render (--host all), which is always present —
|
|
// so the check can no longer silently degrade to a console warning.
|
|
const hostDirs = ['.agents', '.openclaw', '.opencode', '.factory', '.hermes', '.kiro', '.cursor', '.slate'];
|
|
let checked = 0;
|
|
for (const host of hostDirs) {
|
|
const skillsRoot = path.join(EXTERNAL_OUT, host, 'skills');
|
|
if (!fs.existsSync(skillsRoot)) continue;
|
|
const entries = fs.readdirSync(skillsRoot, { withFileTypes: true });
|
|
for (const entry of entries) {
|
|
if (!entry.isDirectory()) continue;
|
|
const skillMd = path.join(skillsRoot, entry.name, 'SKILL.md');
|
|
if (!fs.existsSync(skillMd)) continue;
|
|
const content = fs.readFileSync(skillMd, 'utf-8');
|
|
expect(content, `handshake marker in ${host}/skills/${entry.name}/SKILL.md`).not.toContain(HANDSHAKE_MARKER);
|
|
checked++;
|
|
}
|
|
}
|
|
expect(checked).toBeGreaterThan(0);
|
|
});
|
|
|
|
test.each(REVIEW_SKILLS)(
|
|
'%s/SKILL.md contains the new plan-mode-info section near the top',
|
|
(skill) => {
|
|
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
const idx = content.indexOf(PLAN_MODE_INFO_MARKER);
|
|
expect(idx).toBeGreaterThan(0);
|
|
// Position 1 in preamble composition = within the first ~300 lines.
|
|
// Roughly translates to first ~15KB of text.
|
|
expect(idx).toBeLessThan(15_000);
|
|
},
|
|
);
|
|
|
|
test('plan-mode-info is wired BEFORE generateUpgradeCheck in preamble', () => {
|
|
// Token-reduction Phase 2: generateUpgradeCheck's render output is now
|
|
// ONLY the steady-state PROACTIVE-false + SKILL_PREFIX rules (the
|
|
// UPGRADE_AVAILABLE prose emits from bin/gstack-skill-start at runtime),
|
|
// so those rules are the resolver's order marker.
|
|
const content = fs.readFileSync(
|
|
path.join(ROOT, 'plan-ceo-review', 'SKILL.md'),
|
|
'utf-8',
|
|
);
|
|
const planModeIdx = content.indexOf(PLAN_MODE_INFO_MARKER);
|
|
const upgradeIdx = content.indexOf('If `PROACTIVE` is `"false"`');
|
|
expect(planModeIdx).toBeGreaterThan(0);
|
|
expect(upgradeIdx).toBeGreaterThan(0);
|
|
expect(planModeIdx).toBeLessThan(upgradeIdx);
|
|
});
|
|
|
|
test('0C-bis STOP block present in plan-ceo-review/SKILL.md', () => {
|
|
const content = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
|
const presentIdx = content.indexOf('Present these approach options via AskUserQuestion');
|
|
const preludeIdx = content.indexOf('### 0D-prelude');
|
|
expect(presentIdx).toBeGreaterThan(0);
|
|
expect(preludeIdx).toBeGreaterThan(presentIdx);
|
|
const between = content.slice(presentIdx, preludeIdx);
|
|
expect(between).toContain('**STOP.**');
|
|
expect(between).toContain('Do NOT proceed to Step 0D or 0F until the user responds to 0C-bis');
|
|
});
|
|
});
|
|
|
|
// GSTACK REVIEW REPORT report-at-bottom contract — verifies the prompt-text
|
|
// fix in scripts/resolvers/review.ts (the load-bearing change for the
|
|
// "report not at bottom of plan in plan mode" bug). The bug is in the
|
|
// prompt's contradictory write-flow instructions, not in observable
|
|
// runtime behavior we can cheaply gate in CI. Verifying the prompt text
|
|
// directly is the deterministic equivalent of the regression test the
|
|
// PTY harness can't reliably drive (autoplan needs auto-progression of
|
|
// AskUserQuestions to reach the report-write step, which the harness
|
|
// doesn't support today).
|
|
describe('GSTACK REVIEW REPORT delete-then-append flow', () => {
|
|
const PLAN_REVIEW_SKILLS = [
|
|
'plan-ceo-review',
|
|
'plan-design-review',
|
|
'plan-devex-review',
|
|
'plan-eng-review',
|
|
];
|
|
|
|
for (const skill of PLAN_REVIEW_SKILLS) {
|
|
test(`${skill}/SKILL.md prescribes delete-then-append, not in-place replace`, () => {
|
|
// Carved skills (v2 plan Phase B) relocate the review-report prose into
|
|
// sections/*.md; readSkillUnion follows the content wherever the carve put it.
|
|
const content = readSkillUnion(skill);
|
|
|
|
// The new (correct) instruction must be present.
|
|
expect(content).toContain('delete-then-append flow');
|
|
expect(content).toContain('never mid-file');
|
|
expect(content).toContain('Do NOT replace the section in place');
|
|
|
|
// The old contradictory bullets must be gone. The signature phrase
|
|
// from the buggy prompt was 'replace it entirely using the Edit tool'
|
|
// which is what allowed mid-file reports to stay mid-file.
|
|
expect(content).not.toContain('replace it** entirely using the Edit tool');
|
|
expect(content).not.toContain('If it was found mid-file, move it');
|
|
});
|
|
}
|
|
|
|
test('scripts/resolvers/review.ts source has the rewritten flow', () => {
|
|
const src = fs.readFileSync(path.join(ROOT, 'scripts', 'resolvers', 'review.ts'), 'utf-8');
|
|
expect(src).toContain('delete-then-append flow');
|
|
expect(src).toContain('never mid-file');
|
|
expect(src).toContain('Do NOT replace the section in place');
|
|
// Old contradictory bullets are gone from the source resolver.
|
|
expect(src).not.toContain('replace it** entirely using the Edit tool');
|
|
expect(src).not.toContain('If it was found mid-file, move it');
|
|
});
|
|
});
|
|
|
|
describe('LEARNINGS_SEARCH resolver: query parameter', () => {
|
|
// Lazy-load resolver and types after describe block to keep test file self-contained.
|
|
const { generateLearningsSearch } = require('../scripts/resolvers/learnings');
|
|
const { HOST_PATHS } = require('../scripts/resolvers/types');
|
|
|
|
const claudeCtx = {
|
|
skillName: 'test',
|
|
tmplPath: 'test/SKILL.md.tmpl',
|
|
host: 'claude',
|
|
paths: HOST_PATHS.claude,
|
|
};
|
|
const codexCtx = { ...claudeCtx, host: 'codex', paths: HOST_PATHS.codex };
|
|
|
|
test('no args → bash does not contain --query (backwards-compat)', () => {
|
|
const out = generateLearningsSearch(claudeCtx);
|
|
expect(out).not.toContain('--query');
|
|
});
|
|
|
|
test('claude host + query=foo bar → both cross-project and project-scoped branches contain --query', () => {
|
|
const out = generateLearningsSearch(claudeCtx, ['query=foo bar']);
|
|
// Both branches of the if/else must carry the flag.
|
|
const lines = out.split('\n').filter(l => l.includes('gstack-learnings-search'));
|
|
expect(lines.length).toBeGreaterThanOrEqual(2);
|
|
for (const line of lines) {
|
|
expect(line).toContain('--query "foo bar"');
|
|
}
|
|
});
|
|
|
|
test('codex host + query=foo bar → codex bash variant contains --query', () => {
|
|
const out = generateLearningsSearch(codexCtx, ['query=foo bar']);
|
|
expect(out).toContain('--query "foo bar"');
|
|
expect(out).toContain('$GSTACK_BIN/gstack-learnings-search');
|
|
});
|
|
|
|
test('empty value query= → bash does not contain --query (locked semantics: falls through)', () => {
|
|
const claudeOut = generateLearningsSearch(claudeCtx, ['query=']);
|
|
expect(claudeOut).not.toContain('--query');
|
|
const codexOut = generateLearningsSearch(codexCtx, ['query=']);
|
|
expect(codexOut).not.toContain('--query');
|
|
});
|
|
|
|
test('shell-injection chars in query= → throws at gen-time (defense in depth)', () => {
|
|
for (const bad of ['$(whoami)', '`cmd`', 'a;b', 'a&b', 'a"b', 'a\\b', 'foo$x']) {
|
|
expect(() => generateLearningsSearch(claudeCtx, [`query=${bad}`])).toThrow(/alphanumeric/);
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('EXIT PLAN MODE GATE placement', () => {
|
|
// Fresh skill list — do NOT reuse REVIEW_SKILLS upstream (3 entries, missing plan-devex).
|
|
const planSkills = ['plan-eng-review', 'plan-ceo-review', 'plan-design-review', 'plan-devex-review'];
|
|
|
|
// Strip fenced code blocks before matching headings — PLAN_FILE_REVIEW_REPORT
|
|
// already contains `## GSTACK REVIEW REPORT` inside a markdown example fence,
|
|
// and the gate text itself shows `## GSTACK REVIEW REPORT` inside a fence too.
|
|
const stripFences = (md: string) => md.replace(/```[\s\S]*?```/g, '');
|
|
|
|
test('gate is the terminal ## heading in every plan-* review SKILL.md', () => {
|
|
for (const skill of planSkills) {
|
|
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
const stripped = stripFences(md);
|
|
const headings = [...stripped.matchAll(/^## .+$/gm)].map(m => m[0]);
|
|
const lastH2 = headings.at(-1);
|
|
expect(lastH2, `${skill}/SKILL.md last ## heading (fences stripped)`).toBe('## EXIT PLAN MODE GATE (BLOCKING)');
|
|
expect(md, `${skill}/SKILL.md gate body`).toContain('Failing this gate and calling ExitPlanMode anyway is a contract violation');
|
|
}
|
|
});
|
|
|
|
test('codex/SKILL.md contains gate (mid-file per D5; Step 2B/2C follow)', () => {
|
|
const codex = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
|
expect(codex).toContain('## EXIT PLAN MODE GATE (BLOCKING)');
|
|
expect(codex).toContain('Failing this gate and calling ExitPlanMode anyway is a contract violation');
|
|
});
|
|
});
|
|
|
|
describe('scope-gate exceptions drift-guard', () => {
|
|
// The plan-mode auto-select-B exceptions block is hand-duplicated in the
|
|
// plan-eng-review and plan-design-review templates (matching the gate
|
|
// around it, which predates this block). The two copies must stay
|
|
// byte-identical modulo exactly two known variant slots:
|
|
// 1. the plan-mode bullet's action tail (Design Doc Check vs pre-review
|
|
// audit + mockups),
|
|
// 2. the named-target vocabulary ("a path, a doc" vs "a path, a page, a doc").
|
|
// A future edit to one copy that silently misses the other fails here
|
|
// instead of drifting. The real fix (shared {{SCOPE_GATE}} resolver) is a
|
|
// filed TODO — this guard is the stopgap that makes the duplication safe.
|
|
const START_MARKER = '**Exceptions — check in this order, BEFORE asking:**';
|
|
const END_MARKER = 'in any mode — it is a hard STOP.';
|
|
|
|
function extractExceptionsBlock(skill: string): string {
|
|
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
const start = md.indexOf(START_MARKER);
|
|
expect(start, `${skill}/SKILL.md: exceptions block start marker present`).toBeGreaterThan(-1);
|
|
const end = md.indexOf(END_MARKER, start);
|
|
expect(end, `${skill}/SKILL.md: exceptions block end marker present`).toBeGreaterThan(start);
|
|
return md.slice(start, end + END_MARKER.length);
|
|
}
|
|
|
|
const normalizeVariantSlots = (block: string) =>
|
|
block
|
|
.replace('Then run the Design Doc Check and Step 0 against that plan.', '<ACTION_TAIL>')
|
|
.replace('Then run the pre-review audit, mockups, and Step 0 against that plan.', '<ACTION_TAIL>')
|
|
.replace('a path, a page, a doc they pasted,', 'a path, a doc they pasted,');
|
|
|
|
test('eng and design exceptions blocks are identical modulo the two variant slots', () => {
|
|
const eng = normalizeVariantSlots(extractExceptionsBlock('plan-eng-review'));
|
|
const design = normalizeVariantSlots(extractExceptionsBlock('plan-design-review'));
|
|
expect(eng).toBe(design);
|
|
// The action tail must actually have been normalized in both (guards
|
|
// against a rewording that bypasses the normalizer and vacuously passes).
|
|
expect(eng).toContain('<ACTION_TAIL>');
|
|
});
|
|
|
|
test('exceptions block carries the announcement string the PTY detectors pin', () => {
|
|
for (const skill of ['plan-eng-review', 'plan-design-review']) {
|
|
const block = extractExceptionsBlock(skill);
|
|
expect(block, `${skill}: verbatim announcement`).toContain(
|
|
'Scope gate: plan mode — auto-selected B (reviewing <target>).',
|
|
);
|
|
}
|
|
});
|
|
|
|
test('gate menu carries the question strings the PTY question detector pins', () => {
|
|
// isScopeGateQuestionVisible (claude-pty-runner.ts) anchors on the
|
|
// question text + option A's body. If the menu is reworded without
|
|
// updating the detector, the paid smokes' must-stay-false assertions go
|
|
// vacuous — this free pin fails first.
|
|
for (const skill of ['plan-eng-review', 'plan-design-review']) {
|
|
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
expect(md, `${skill}: gate question text`).toContain('What should I review?');
|
|
expect(md, `${skill}: option A body text`).toContain('The current branch diff');
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('GSTACK REVIEW REPORT mandatory unresolved-decisions status', () => {
|
|
// Report text rides in PLAN_FILE_REVIEW_REPORT → every report consumer gets it.
|
|
// devex-review is a report consumer but NOT a gate consumer, so the two target
|
|
// sets differ (CP5/CX5). Regression guard: a future token-cut that drops the
|
|
// unresolved-status line again fails here. See plan-flag-unresolved-issues.
|
|
const REPORT_CONSUMERS = [
|
|
'plan-ceo-review',
|
|
'plan-eng-review',
|
|
'plan-design-review',
|
|
'plan-devex-review',
|
|
'codex',
|
|
'devex-review',
|
|
];
|
|
// Gate text rides in EXIT_PLAN_MODE_GATE (lives in SKILL.md, not sections).
|
|
const GATE_SKILLS = [
|
|
'plan-ceo-review',
|
|
'plan-eng-review',
|
|
'plan-design-review',
|
|
'plan-devex-review',
|
|
'codex',
|
|
];
|
|
|
|
for (const skill of REPORT_CONSUMERS) {
|
|
test(`${skill}: report mandates the unresolved-decisions status as final content`, () => {
|
|
const content = readSkillUnion(skill);
|
|
expect(content).toContain('NO UNRESOLVED DECISIONS');
|
|
// The "never omit / always final" contract must be present, not just the phrase.
|
|
expect(content).toContain('Unresolved-decisions status (MANDATORY');
|
|
expect(content).toMatch(/never omitted/);
|
|
// \s+ tolerates prose line-wraps within "final non-whitespace line".
|
|
expect(content).toMatch(/final\s+non-whitespace\s+line/);
|
|
});
|
|
}
|
|
|
|
for (const skill of GATE_SKILLS) {
|
|
test(`${skill}: exit gate blocks unless the unresolved status is the final line`, () => {
|
|
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
|
// Gate check #4 — present, sentinel named, and explicitly blocking (no escape).
|
|
expect(md).toContain('NO UNRESOLVED DECISIONS');
|
|
expect(md).toContain('FINAL non-whitespace line is the unresolved-decisions');
|
|
expect(md).toContain('FAILS the gate');
|
|
});
|
|
}
|
|
|
|
test('scripts/resolvers/review.ts source carries the mandatory block + blocking gate', () => {
|
|
const src = fs.readFileSync(path.join(ROOT, 'scripts', 'resolvers', 'review.ts'), 'utf-8');
|
|
// Report resolver: mandatory, never-omitted, exact sentinel, anti-double-count algorithm.
|
|
expect(src).toContain('Unresolved-decisions status (MANDATORY');
|
|
expect(src).toContain('NO UNRESOLVED DECISIONS');
|
|
expect(src).toContain('avoids double-counting');
|
|
expect(src).toContain('DROP the current skill');
|
|
// Gate resolver: the blocking final-line check with no "if applicable" escape.
|
|
expect(src).toContain('FINAL non-whitespace line is the unresolved-decisions');
|
|
expect(src).toContain('FAILS the gate');
|
|
// The old soft wording must be gone from the gate.
|
|
expect(src).not.toContain('absorbs CODEX / CROSS-MODEL / UNRESOLVED lines if applicable');
|
|
});
|
|
});
|
|
|
|
// ─── {{PREAMBLE}} requires an explicit preamble-tier ────────
|
|
|
|
describe('PREAMBLE resolution requires declared preamble-tier', () => {
|
|
test('resolving {{PREAMBLE}} without preamble-tier throws with the template path', async () => {
|
|
const { generatePreamble } = await import('../scripts/resolvers/preamble');
|
|
const { HOST_PATHS } = await import('../scripts/resolvers/types');
|
|
const ctx = {
|
|
skillName: 'tierless-skill',
|
|
tmplPath: 'tierless-skill/SKILL.md.tmpl',
|
|
host: 'claude' as const,
|
|
paths: HOST_PATHS.claude,
|
|
// preambleTier deliberately absent — the generator must refuse to default it.
|
|
};
|
|
expect(() => generatePreamble(ctx)).toThrow(/tierless-skill\/SKILL\.md\.tmpl/);
|
|
expect(() => generatePreamble(ctx)).toThrow(/preamble-tier/);
|
|
});
|
|
|
|
test('every template that resolves {{PREAMBLE}} declares preamble-tier in frontmatter', () => {
|
|
const entries = fs.readdirSync(ROOT, { withFileTypes: true });
|
|
const offenders: string[] = [];
|
|
const checkTmpl = (tmplPath: string) => {
|
|
const tmpl = fs.readFileSync(tmplPath, 'utf-8');
|
|
if (tmpl.includes('{{PREAMBLE}}') && !/^preamble-tier:\s*\d+$/m.test(tmpl)) {
|
|
offenders.push(path.relative(ROOT, tmplPath));
|
|
}
|
|
};
|
|
checkTmpl(path.join(ROOT, 'SKILL.md.tmpl'));
|
|
for (const e of entries) {
|
|
if (!e.isDirectory() || e.name.startsWith('.') || e.name === 'node_modules') continue;
|
|
const tmplPath = path.join(ROOT, e.name, 'SKILL.md.tmpl');
|
|
if (fs.existsSync(tmplPath)) checkTmpl(tmplPath);
|
|
}
|
|
expect(offenders).toEqual([]);
|
|
});
|
|
});
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// #2499: gbrain MCP detection must read BOTH ~/.claude.json scopes.
|
|
// Claude Code registers MCP servers at user scope (.mcpServers) and project
|
|
// scope (.projects["/abs/path"].mcpServers — what `claude mcp add` without
|
|
// --scope user writes). The rendered brain-sync block previously read only
|
|
// user scope, so a correctly configured project-scoped brain was invisible.
|
|
// ---------------------------------------------------------------------------
|
|
describe('brain-sync block reads project-scoped MCP registrations (#2499)', () => {
|
|
// Phase 1: the artifacts-sync bash (including the MCP-scope jq probe) moved
|
|
// from the rendered SKILL.md into bin/gstack-skill-start. Pin the LIVE
|
|
// script bytes — same assertions, new home. The render carries only the
|
|
// ARTIFACTS_SYNC interpretation prose.
|
|
const rendered = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
|
|
|
|
test('rendered _GBRAIN_MCP_ENTRY jq resolves project scope with nearest-ancestor cwd match', () => {
|
|
const line = rendered.split('\n').find((l) => l.includes('_GBRAIN_MCP_ENTRY=$('));
|
|
expect(line).toBeDefined();
|
|
// Project-scope read present, driven by $PWD.
|
|
expect(line!).toContain('--arg cwd "$PWD"');
|
|
expect(line!).toContain('.projects');
|
|
// User scope still resolved first.
|
|
expect(line!).toContain('.mcpServers.gbrain');
|
|
// The old user-scope-only filter is gone from the rendered output.
|
|
expect(rendered).not.toContain('.mcpServers.gbrain.type // .mcpServers.gbrain.transport');
|
|
expect(rendered).not.toContain(".mcpServers.gbrain.url // empty");
|
|
});
|
|
|
|
test('rendered _GBRAIN_MCP_TYPE and _GBRAIN_HOST extract from the resolved entry', () => {
|
|
const typeLine = rendered.split('\n').find((l) => l.includes('_GBRAIN_MCP_TYPE=$('));
|
|
const hostLine = rendered.split('\n').find((l) => l.includes('_GBRAIN_HOST=$('));
|
|
expect(typeLine).toBeDefined();
|
|
expect(hostLine).toBeDefined();
|
|
expect(typeLine!).toContain('_GBRAIN_MCP_ENTRY');
|
|
expect(hostLine!).toContain('_GBRAIN_MCP_ENTRY');
|
|
});
|
|
|
|
test('rendered jq lines FUNCTION: project-scoped registration resolves for a cwd inside the project', () => {
|
|
// Execute the exact rendered bytes, not a re-derivation: extract the
|
|
// _GBRAIN_MCP_ENTRY + _GBRAIN_MCP_TYPE lines from the generated SKILL.md
|
|
// and run them in bash against a fixture ~/.claude.json that carries ONLY
|
|
// a project-scoped gbrain registration.
|
|
const lines = rendered.split('\n');
|
|
const entryLine = lines.find((l) => l.includes('_GBRAIN_MCP_ENTRY=$('));
|
|
const typeLine = lines.find((l) => l.includes('_GBRAIN_MCP_TYPE=$('));
|
|
expect(entryLine).toBeDefined();
|
|
expect(typeLine).toBeDefined();
|
|
|
|
const tmpHome = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-2499-home-'));
|
|
const projectDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-2499-proj-'));
|
|
const nestedCwd = path.join(projectDir, 'src', 'deep');
|
|
fs.mkdirSync(nestedCwd, { recursive: true });
|
|
try {
|
|
fs.writeFileSync(
|
|
path.join(tmpHome, '.claude.json'),
|
|
JSON.stringify({
|
|
projects: {
|
|
[projectDir]: {
|
|
mcpServers: { gbrain: { type: 'http', url: 'https://brain.example.com/mcp' } },
|
|
},
|
|
},
|
|
}),
|
|
);
|
|
const script = `cd "$1" || exit 1\n${entryLine!.trim()}\n${typeLine!.trim()}\necho "RESOLVED:$_GBRAIN_MCP_TYPE"`;
|
|
const r = spawnSync('bash', ['-c', script, 'bash', nestedCwd], {
|
|
encoding: 'utf-8',
|
|
env: { ...process.env, HOME: tmpHome },
|
|
timeout: 10_000,
|
|
});
|
|
expect(r.stdout).toContain('RESOLVED:http');
|
|
|
|
// Discriminator: a cwd OUTSIDE the project must NOT resolve it.
|
|
const outside = spawnSync('bash', ['-c', script, 'bash', os.tmpdir()], {
|
|
encoding: 'utf-8',
|
|
env: { ...process.env, HOME: tmpHome },
|
|
timeout: 10_000,
|
|
});
|
|
expect(outside.stdout).toContain('RESOLVED:\n');
|
|
} finally {
|
|
fs.rmSync(tmpHome, { recursive: true, force: true });
|
|
fs.rmSync(projectDir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|