mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-13 03:48:06 +01:00
* chore(scripts): retire hollow and legacy paid package scripts behind one-release stubs (W8a, DX-4, CEO-31) test:codex*, test:evals*, test:e2e*, test:gate, test:periodic, eval:bg, eval:bg:all, eval:flake-rank, eval:watch and test:audit now print their replacement and exit 1 via scripts/retired-command.ts. CONTRIBUTING.md carries the same table; test/retired-commands.test.ts pins both. Eval CLI empty-state hints point at eval:bg:pr. * chore: delete dead update-readme-throughput, eval-watch, orphan fixtures and QA_FUNCTIONAL_INPUTS (W8b, CEO-24) Retired with named behavior: - update-readme-throughput.ts + readme-throughput test: replaced a README anchor that no longer exists from a JSON file that was never produced; the script was a no-op. Guarantee withdrawn (nothing to protect). - eval-watch.ts + its dashboard tests: rendered ~/.gstack-dev/e2e-live.json and _partial-e2e.json, written only by unsharded runs, so it showed nothing for every supported (sharded) entry point. Guarantee withdrawn; live progress is the gstack-detach log or gh run watch. Manual-review rendering stays covered by eval-list and eval-summary tests. - Orphan fixtures (autoplan-pending-artifact-ae.json, autoplan-existing-app/, autoplan-sign-in-ui/, parity baselines v1.47.0.0/v1.57.7.0/v1.69.1.0): no test, touchfile or script reads them; superseded baselines stay in git history. - QA_FUNCTIONAL_INPUTS: exported, never imported. * refactor(test-free): delete dead WORKER_HOSTILE list and buildShardArgs parallel mode Neither had a production caller: full-suite mode runs shard processes, so the worker quarantine was inert and --parallel was never passed (W8b). * docs(test-value): narrow prompt-byte contract to machine-read tokens (D2) Define prompt-byte contract as machine-read tokens (markers, field names, command lines, enum values, file paths, security-boundary strings) and add a sentence-pin glossary entry: convert to structural checks or delete when a behavioral eval covers the behavior. * test(ratchet): count free tests that slice a paid test's source A free test that binds readFileSync(<paid test file>) and cuts it with .slice() tests the paid harness's code out of context even when it imports a lib/ symbol. Record the 15 existing files with reasons; new ones fail with fix text. Self-test covers positive and negative shapes. * test(browse): un-quarantine compare-board; all 16 board tests pass headless (W8d, CEO-24) The GSTACK_COMPARE_BOARD_TESTS opt-in hid a headless mismatch that no longer reproduces: all 16 tests pass on Bun 1.4.0 with the bundled headless Chromium, solo, repeated, and beside other in-process browser files. Protected behavior (the comparison-board DOM contract the plan-design-review/office-hours/design-consultation feedback loop polls) is checked on every free run again. * test: replace silent existsSync early returns with explicit skips or assertions (W8d) - browse/test/build.test.ts: server-node.mjs checks skip only when the bundle is absent and GSTACK_EXPECT_BINARIES is unset; the free CI lane and test:ubicloud build it and set the flag, so a dropped build step fails with the build command instead of passing with no assertions. - gstack-update-check: VERSION is tracked; the guard could only hide a bug. - gen-skill-docs host leakage/frontmatter checks assert the host tree exists (the module-level render always produces it). - cso-docker staged-runtime test uses test.skipIf so reports show it as skipped, not passed. * test(ship): convert ship-workflow-clarity sentence pins to structural checks Add test/helpers/prompt-structure.ts (between, expectTokens, expectAbsent, expectOrdered, expectMentions) with fix text on failure. ship-workflow-clarity now pins headings, step order and machine-read tokens (commands, fields, enum and status strings); safety rules are meaning-level keyword checks. Wording is judged by the 'ship/SKILL.md workflow' judge in skill-llm-eval. Sentence pins: 90 -> 0. * ci: add platform-qualification lane for opt-in ML, gitleaks and Swift tests (W8d, CEO-24) security-bench + security-live-playwright L4 (SECURITY_BENCH), the real gitleaks 8.30.1 memory-ingest cases (GSTACK_TEST_GITLEAKS) and the DebugBridge Swift build invariants (GSTACK_TEST_SWIFT) self-skip in every lane today. No scheduled lane has their prerequisites, so per CEO-24 they move to an owned dispatch + quarterly workflow instead of being deleted. Each job was run locally (ML 10 pass, gitleaks 130 pass; Swift needs macOS). test/platform-qualification-workflow.test.ts pins gate variable, job and file together. * fix(scripts): capture-baseline and preflight-agent-sdk no longer act on --help or typos (W8e) capture-baseline parses its flags strictly: --help prints usage, an unknown or valueless flag exits 2, and only a valid invocation writes a fixture (--help used to write test/fixtures/parity-baseline-current.json). preflight-agent-sdk gains --help, rejects unknown flags, and runs the live SDK query (API spend) only with --live. * test(evals): register the three keyless periodic paid files with case ids (W1b) skill-e2e-auq-consistency, skill-e2e-auq-verbose-vs-carved-ab and codex-e2e-recommendation-substance now carry touchfiles, a periodic tier and kind rule (one trial, unchanged), so editing one selects its own case instead of restoring the full PR gate. The verbose A/B case also registers the vendored pre-cut fixture it reads, and review-army-migration-safety registers the SQL fixture it reads. * feat(evals): narrow PR fallback for free-only inputs (W1a, CEO-13/20, ENG-13) The PR profile now exempts the duration seeds, every free-lane workflow (explicit list; inventory test fails on an unclassified workflow, .yaml and workflow_call included), scripts/ubicloud/** and fixtures declared in the explicit FREE_FIXTURES map. Each remaining unknown file carries a label and fix (needs touchfile entry / add to FREE_ONLY_PR_FILES / real unknown dependency). evals.yml itself still restores the full gate. * ci: delete ci-image.yml; the eval workflows already build and push the same content-hash tag (W8f) evals.yml, evals-periodic.yml and evals-marathon.yml each compute the identical hashFiles() tag and build/push it (with :buildcache and :latest) when missing, so ci-image's push trigger and weekly cron were no-ops (30-day median 0.5 min, 36/36 success). ci-image-tag-binding now pins the three eval workflows. * ci(evals): key periodic/marathon concurrency on ref+event; branch periodic dispatch skips the gate census (W2a, ENG-9, CEO-04) * refactor(paid): drop ~160 unused imports; leaf paid-types/paid-select modules break the paid-* import cycle (W8c) * ci(quality-gate): per-PR concurrency, free runner, drop unscoped bun audit (W2a, W8f) - Concurrency keys on pull_request.number || run_id, so back-to-back main pushes no longer cancel each other's credential scan. - ubuntu-24.04 instead of a billed 8-vCPU Ubicloud runner for a ~1 min job. - bun audit --audit-level=critical ran on every PR regardless of diff, so a newly published advisory turned unrelated PRs red, with its own ignore policy. dependency-review (lockfile PRs, fail >= high) and the weekly OSV sweep remain the dependency gates. * chore(test-free): refresh the free duration seed from the audit's Ubicloud recording Adds the 19 files merged after the last refresh, including the 115 s claude-overlay-setup that packed as ~1 s and doubled a shard (W6a). The seed is re-recorded on the final tree at the end of this wave. * ci: move tiny ubicloud-standard-2 jobs to free ubuntu-24.04 runners (W8f) actionlint, skill-docs, dependency-review, version-gate and pr-title-sync are sub-minute jobs; GitHub-hosted runners are free for this public repo. * feat(test-free): pessimistic unseeded packing, seed growth ratchet and CI health summary - Unseeded files pack at the seed's p99 instead of p75 (W6a). - --ci-plan warns in the job summary past 5 unseeded files; --ci-verify lists shards that outran their prediction by 1.5x (and 30 s) plus every flaky pass absorbed by the serial retry (W6a, W7a job summary). - test/free-seed-ratchet.test.ts fails when a seeded file over 60 s is not on scripts/free-test-seed-allowlist.json, when an entry is stale, and when the allowlist grows versus its merge-base copy unless the new entry is an annotated exception with a todo (CEO-10, DX-7). free-plan runs it with full history so the shrink check is real in the required lane. - Every message names the fix command and docs anchor (DX-6). * feat(evals): state PR coverage mode, counts, reuse and fallback files in the summary and comment (CEO-13) * test: make the tracked-tree skill-doc freshness check local-only; tidy skill-docs header (W8f) Every free CI shard regenerates the tracked SKILL.md files before running tests, so the --dry-run freshness assertion compared the regenerated tree with itself and could not fail there. It now skips under GitHub Actions; skill-docs.yml's git diff --exit-code is CI's freshness gate. Removes a stale concurrency comment in skill-docs.yml. * ci(free-tests): install zsh so the bash+zsh portability arms run (W6f) ~42 zsh-arm tests (aside-probe-shell, qa-browser-preservation, generated-docs-fences) skipped in CI because zsh is not in the runner image. All 141 tests in those files pass with zsh present. * feat(evals): executors write negative receipts at the execution boundary (CEO-25, ENG-6) A completed single-trial rule/judge shard that did not pass now writes its negative receipt beside the pass receipts in the slice artifact, so a cancelled run's FAIL still blocks an older PASS when its report never runs. Receipts merge negatives first and order by (run attempt, completedAt); a same-second FAIL/PASS tie resolves to FAIL. The runner call site is a one-line hunk in scripts/test-paid-shards.ts. * docs(cso): state that contained runtime/scanner execution is unavailable today The scanner catalog is empty while its build inputs await review and the runtime catalog holds no qualified profile until the private evaluator exists, so /cso runs static assessment only. AGENTS.md, README and docs/cso-release-qualification.md now say so; docs/cso-scanners.md already did. * test(helpers): expectMentions splits on sentence ends only Clauses joined by ';' or ':' are one rule; splitting there made a reworded rule fail for punctuation. * test(plan-ceo): convert skill-ceo-section-ordering sentence pins Safety and authority rules (never/do not/stop/only/without/wait/approval) are meaning-level expectMentions checks; other sentence pins are dropped. Headings, labels, option rows, placeholders and machine-read lines (auto-decide annotation, Mode:/approved decisions format, STATUS/SOURCE fields, CROSS-MODEL marker, gstack-ceo-mode-handoff) stay as tokens; ordering checks unchanged. Behavior is covered by the periodic PTY case in skill-e2e-plan-ceo-review-section-loading.test.ts. Sentence pins: 210 -> 23. * test(helpers): expectMentions treats wrapped lines as one sentence Templates hard-wrap prose; only sentence ends and blank lines separate rules. * test(gen-skill-docs): convert sentence pins to meaning checks or drop them Safety/authority sentences become expectMentions keyword checks; other English sentences are dropped (wording is judged by the per-skill workflow judges in skill-llm-eval and the E2E cases in touchfiles-data). Commands, config keys, markers (PLAN MODE EXCEPTION, NO UNRESOLVED DECISIONS, Gate outcome), field names and alternation-based meaning regexes stay. Removed the now-empty 'contains opt-in AskUserQuestion text' test (only pinned a sentence; the resolver's AskUserQuestion wiring is covered by the codex second-opinion tests in the same describe). Lane E's existsSync hunks are untouched. Sentence pins: 216 -> 65 (remaining lines carry machine tokens). * ci(windows): shard the Windows lane via --ci-plan/--ci-run/--ci-verify; move native campaigns to native-qualification.yml - windows-free-tests.yml: a Linux plan job curates and packs the Windows-safe files into 6 shards, one strict serial shard per windows-latest job, and a Linux aggregate verifies exact coverage and every result (8.1 min on one runner today). --windows-only now combines with the CI modes; the plan binds the curated file set (W6e). - Every Windows plan and --list run prints the curated count with its top exclusion reasons in the job summary; a free ratchet fails when the curated subset drops below its floor, with the fix (W6e). - The four Dia/cookie diagnostic dispatch campaigns move to the dispatch-only native-qualification.yml with one exclusive `mode` choice input, so their red dispatch runs no longer read as a red PR lane (W8f). Their workflow pins move to test/windows-native-workflows.test.ts; the boolean-exclusivity guard is replaced by the choice input. * feat(paid): per-slice CI job ceilings and an in-process slice deadline with not_run/hung outcomes and checkpoints (W2c, ENG-2) * fix(pass-rates): segment trial history strictly by EVAL_POLICY version Reader-compatibility floor for EVAL_POLICY v2 (ENG-14/CEO-35/DX-12). Only trials under the reader's own policy version feed labels and alarms; older policy trials stay visible but are never scored or used as a drift baseline; trials from a newer policy are ignored with a printed count and fix text. Reverting the later v2 writer commits leaves a reader that cannot pool v2 records into v1 series (the pre-PR reader at2db0b3adoes: it reports a mixed fixture as FAILING with drift alarms). * docs: one contributor command table; retired commands, seed sources and current workflows (W8g, DX-5, DX-13, CEO-12) - CONTRIBUTING.md gains 'Which command do I run?' (task -> command -> cost/time -> needs, including branch validation via gh workflow run); CLAUDE.md, AGENTS.md and TESTING_INTERNALS.md link to it instead of repeating command lists. Tier tables drop retired commands and stale per-run dollar estimates. - Retired names (test:evals, test:e2e, eval:bg, eval:flake-rank, eval:watch, ...) are gone from CLAUDE.md, CONTRIBUTING.md, TESTING_INTERNALS.md, ARCHITECTURE.md and USING_GBRAIN_WITH_GSTACK.md. - The flake-ledger claim now says what happens: CI uploads flake-ledger-* artifacts; eval:pass-rates reads only the local ledger. - TESTING_INTERNALS states that only test:ubicloud --record-durations produces an acceptable duration seed (DX-13). - CI section: quality gate no longer runs bun audit; dependency-review's real trigger; the platform-qualification lane. - docs/test-audit-2026-09.md gets a corrections section for its stale claims. - TODOS.md: audit follow-ups (post-merge metrics, stub removal, name reclaim, diff-aware planner); eval-watch shard-awareness and the image freshness tripwire closed as won't-do. * feat(evals): recover receipts of cancelled PR runs (W1c, CEO-14/34, ENG-4/5/7) A new recover-receipts job runs base-ref code with actions: read, restores the PR's store, merges the receipts of completed cancelled evals.yml runs for this PR (provenance-checked, at most 5, oldest first, negatives first), reconstructs completed panel negatives with panelVerdict() (partial panels get no credit) and keeps a recovery.json checkpoint that never advances past an unresolved run. plan-slices keeps contents: read, takes the store as an artifact, turns reuse off when recovery degraded or a run is unresolved, and carries the whole store to slices-report so nothing is lost. Any gh problem degrades to no recovered receipts with a named reason. * feat(evals): classify this wave's free-only tools and workflows for PR selection platform-qualification.yml, retired-command.ts, test-health-report.ts, bump-harness-version.ts and eval-bg.ts never run inside a paid case. * feat(evals): debounce PR push bursts before planning paid slices (W1d, CEO-15, ENG-11, DX-7) A tiny ubuntu-24.04 job waits 90 s only on synchronize when another run for the PR started in the last 15 minutes; cancel-in-progress supersedes, and a run that outlives the wait reports "superseded by <sha>" and plans nothing. Labels: evals-no-debounce skips the wait, evals-fresh disables receipt reuse. * perf(test): move the real-process browser settlement cases out of test-free-shards.test.ts (W6b) The 19 owned-browser settlement modes were 85 s of the file's 95 s and could not be packed apart from the fast runner unit cases. They move to a shared fixture with two files (lifecycle 47 s, ownership 39 s); the unit file drops to 10 s. Same modes, same assertions. * chore(evals): delete the finished validation_phase dispatch and its inline planner (W8f) Retirement (CEO-24): the protected behavior was a one-off cookie-repair validation subset (eight dependent cases) and a quality/behavior split of the census; that validation is complete, so the guarantee is withdrawn. Branch validation dispatches evals.yml with evals_all. * perf(test): inject the office-hours record grace in non-cooperative deadline tests * perf(test): shrink aside-render process slack via GSTACK_RENDER_SLACK_MS in timeout tests * perf(test): make the memorable lock give-up configurable for lock-contention tests * perf(test): check install-timeout knob normalization through its own setup lines * perf(test): inject a 2 s storage-reset deadline in the late isolated dispatch test * perf(test): shorten the slow gbrain fake to 1 s, still past the 300 ms probe timeout * ci(free-tests): run only cso-docker-integration in the free CSO Docker job (F16) cso-node-lifecycle-integration and cso-stack-cold-integration need GSTACK_CSO_TEST_STACK images and always skipped there; they stay in the CSO runtime workflows through test:cso:docker. * test(gen-skill-docs,plan-ceo): split mixed sentence pins into tokens and rules Sentences that wrap a backticked or bold token keep the token (expectTokens) and, when they state a safety rule, a keyword check; short safety sentences (25-39 chars) get the same treatment. * test(plan-ceo): convert ceo-workflow-clarity.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: periodic PTY case skill-e2e-plan-ceo-review-section-loading and the plan-ceo workflow judges. Sentence pins: 13 -> 1. * test(plan-eng): convert eng-review-routing.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: plan-eng E2E cases (skill-e2e-plan-eng-*) and the plan-eng workflow judge. Sentence pins: 28 -> 6. * test(plan-eng): convert eng-scope-entry-ap.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: plan-eng E2E cases (skill-e2e-plan-eng-*) and the plan-eng workflow judge. Sentence pins: 16 -> 5. * test(plan): convert plan-scope-selection.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: plan-review E2E cases and workflow judges in skill-llm-eval. Sentence pins: 19 -> 2. * test(qa): convert qa-lazy-sections.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 71 -> 11. * test(qa): convert qa-browser-preservation.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 25 -> 11. * test(qa): convert qa-probe-gates.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 20 -> 5. * test(qa): convert qa-checkpoint-evidence.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 7 -> 5. * test(qa): convert qa-evidence.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: qa/qa-only E2E cases (skill-e2e-qa-*) and the qa workflow judge. Sentence pins: 4 -> 3. * test(design): convert design-consultation-contract.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: design-consultation E2E/judge cases in touchfiles-data. Sentence pins: 44 -> 11. * test(outside-voice): convert run-in-background-guidance.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: outside-voice E2E cases and plan-review workflow judges. Sentence pins: 35 -> 5. * test(spec): convert spec-template-invariants.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: spec E2E/judge cases in touchfiles-data. Sentence pins: 36 -> 21. * perf(evals): 420 s census slice budget; case-shard skill-e2e-qa-workflow; seed its case walls (W5c) * feat(evals): evals.yml dispatch contract for eval:bg (CEO-17/29, DX-1, ENG-8/12) workflow_dispatch gains base_ref, base_sha, expected_sha, nonce and pr_receipts. The nonce appears in run-name so the dispatcher resolves its own run; evals_all=false runs the PR profile against base_sha/base_ref; a checkout that differs from expected_sha fails before any paid work with a named fix; pr_receipts reads that PR's receipts read-only through the recover-receipts artifact path (never the cache) and never saves. * ci(windows): pack the Windows lane with Windows-measured durations (W6e) The first sharded run (37221451711) finished in 6.0 min: the Linux seed has browse/test/cookie-import-native-job.test.ts at ~0 s (its cases skip on Linux) but it takes 122 s on Windows, so one shard ran 242 s against an 84 s prediction. The plan now reads scripts/free-test-durations-windows.json; a record_durations dispatch times every Windows-safe file alone and uploads it. * test(free): settlement case files import the runner rule they exercise Keeps the split files out of the test-of-test ratchet: the retry-eligibility rule is a direct runner import passed to the shared fixture. * fix(marathon): no hollow plan-decision-classification shard; office-hours marathon scenario scoped to the pilot (W2f) * test(free): satisfy the typecheck:test ratchet in the seed ratchet test * refactor(ci-history): one shared GitHub history reader with the CI flake-ledger reader Extract pass-rates' gh fetch layer into scripts/lib/ci-history.ts (CEO-09) and add the W7a reader: paged run listings that fail whole on any page error, per-run artifact downloads (cached, size-capped, data only), flake-ledger parsing, the pre-registered flaky-file rule (> 5% of main free-tests runs over a window of at least 20 runs, widened to the newest 20 runs when the window is thinner), a job-summary table, and named unavailable reasons with next steps. * feat(test-health): test:health report of the audit success metrics scripts/test-health-report.ts (CEO-09/06/22/32, DX-8/9/12/13) prints every success metric with its counting rule from CI history: PR full-fallback rate, reused PR trial records, PR-lane spend per merged PR and per validated revision, cancelled slice-minutes per week, push-to-verdict p50/p95, census expected failed trials per run with raw failure rate per kind/class and P(lane red) under the recorded panels vs single-trial aggregation, INFRA re-dispatches, quarantine evidence (insufficient evidence distinct from healthy), free flaky-pass rate, free/Windows wall, free serial time and the net-complexity line. An unreadable input prints 'unavailable: <reason>; next: <step>', never 0. --enforce fails on a flaky file (W7a) or > 5 unseeded free files (CEO-10); unavailable checks never fail and keep the issue open. * test(review): convert review-workflow-clarity.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval. Sentence pins: 85 -> 17. * test(review): convert review-entry-and-design-clarity-au.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval. Sentence pins: 15 -> 4. * test(review): convert review-enum-lifecycle.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: review E2E cases (skill-e2e-review*) and the review workflow judge in skill-llm-eval. Sentence pins: 8 -> 0. * test(ship): convert ship-control-flow.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 74 -> 9. * test(ship): convert ship-skip-requeue.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 28 -> 3. * test(ship): convert ship-plan-completion-invariants.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 28 -> 12. * test(ship): convert ship-reentry-gates.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 16 -> 4. * test(ship): convert ship-document-release-dispatch.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 11 -> 4. * test(ship): convert ship-apple-gate.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 7 -> 0. * test(ship): convert ship-skip-actor.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 6 -> 1. * test(ship): convert ship-template-redaction.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 5 -> 4. * test(ship): convert ship-publication-gates.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 5 -> 2. * test(ship): convert ship-review-loop.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 3 -> 1. * test(ship): convert ship-version-sync.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 5 -> 2. * test(ship): convert ship-pr-liveness-policy.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: ship E2E cases (skill-e2e-ship*, skill-e2e-workflow) and the 'ship/SKILL.md workflow' judge. Sentence pins: 4 -> 0. * test(plan-review): convert plan-review-cases.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: plan-review E2E cases and workflow judges in skill-llm-eval. Sentence pins: 58 -> 15. * test(autoplan): convert autoplan-phase-order.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge. Sentence pins: 42 -> 11. * test(autoplan): convert autoplan-phase-handoff.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge. Sentence pins: 20 -> 4. * test(autoplan): convert autoplan-snapshot.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: autoplan E2E cases (skill-e2e-autoplan-*) and the autoplan workflow judge. Sentence pins: 21 -> 5. * test(outside-voice): convert outside-voice-preflight.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: outside-voice E2E cases and plan-review workflow judges. Sentence pins: 26 -> 9. * test(ask-format): convert resolver-ask-user-format.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: AUQ E2E cases (skill-e2e-auq-*); prompt-injection rule kept as a meaning check. Sentence pins: 32 -> 15. * test(skills): convert binding-template-drift.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 19 -> 13. * test(skills): convert carve-plan-fixture.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 20 -> 6. * test(land-and-deploy): convert land-and-deploy-postfail.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge. Sentence pins: 21 -> 14. * test(land-and-deploy): convert land-and-deploy-ci-gate.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge. Sentence pins: 5 -> 4. * test(land-and-deploy): convert land-and-deploy-flow.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: land-and-deploy E2E cases and its workflow judge. Sentence pins: 5 -> 3. * test(office-hours): convert office-hours-completion.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: office-hours E2E cases (skill-e2e-office-hours*). Sentence pins: 16 -> 1. * test(plan-ceo): restore the stale closing-route guard The negative check for 'Closing hooks' / 'return to section' guards a retired route token, not a sentence; the first conversion pass dropped it. * ci(test-health): weekly test-health workflow with an honest tracking issue New test-health.yml (CEO-18/DX-9): Monday 10:00 UTC after the periodic census plus dispatch, free ubuntu-24.04 runner, permissions contents:read + actions:read + issues:write only. Runs bun run test:health --since-days 7 --enforce; an enforced failure upserts one tracking issue, a run with unavailable inputs keeps it open and lists them, and it closes only when every enforced check has evidence and passes. Only main runs touch the issue. * feat(evals): an edited paid test file runs its own gate cases in the PR lane (DX-11) A gate case outside the audited fast profile is selected when its single owning paid test file changed and the runner can select it by name (its Bun test is named by the case id). The manifest records directCases and validation accepts only recorded, name-addressable ones; the runner and planner read the file->cases map from test-pr-profile. Cases the runner cannot address stay deferred with the fix. An inventory test covers every owned gate case. Touches one hunk each in test-paid-shards.ts and lib/paid-plan.ts, and pr-shared-input-selection.test.ts's ship-skip pin. * ci(osv): tracking issue for scheduled reds and a 14-day suppression expiry test F11/W8f: a failed main OSV scan now upserts one tracking issue and a clean scan closes it (scheduled reds notified no one during the August backlog). A free test fails 14 days before any .osv-scanner.toml ignoreUntil (all four expire 2026-11-30), with the fix: upgrade and delete, or re-justify and move the date. * test(ship,review): keep guards against retired fail-open instructions 'skip silently' on reviewer failure and 'do not block /ship on subagent failure' were harmful instructions, not prose style; keep them absent with the shortest distinctive phrase. * feat(eval-select): default to the pr profile DX-10: eval:select previews the PR lane's selection by default, matching test:pr and eval:bg:pr; --profile full (or EVALS_PROFILE=full) keeps the plain touchfile selection. * test(evals): type-clean the recovery gh fake (typecheck:test ratchet) * chore(windows): record the Windows duration seed (run 37221963236) 573 Windows-safe files timed alone on windows-latest; cookie-import-native-job is 102 s there. Six shards now predict 154 s each. * feat(evals): eval:bg dispatches the pushed ref to CI or runs locally under a 1.5x-planned cap (W5d, CEO-17/29, DX-1/2/3/7, ENG-12/19) * feat(runners): --help on both shard runners via the shared flag loop; paid runner header describes today's planner/executor/report (W8e) * test(windows): drop the stale Playwright exclusion and force-include probe-proven files (W6e) A temporary windows-latest probe (run 37221995077) ran each of the 97 files excluded for 'launches Playwright browser' or 'raw /tmp/ path' alone: all 17 Playwright files pass (Chromium is installed and Bun drives it there), and 41 of the /tmp files pass with executed cases. The Playwright pattern is removed and the 41 files are force-included with the probe as their reason; the 36 real failures stay excluded. The curated subset grows from 573 to 631 files and the ratchet floor follows. Windows curation moves to scripts/lib/windows-curation.ts (the runner drops to 2000 lines). * test(free): one install fixture cloned from a read-only seed (W6c, ENG-1) Merge test/helpers/setup-codex-scope-fixture.ts into install-fixture.ts. The checkout is built once per test file into a read-only seed and cloned per fixture with copy-on-write where the filesystem has it (full copy otherwise), restoring each file's own mode; clones never hard-link the seed because put() and tests write in place. Callers register afterAll(cleanupSeed); Bun's runner fires no exit event, so seeds of killed runs are swept by pid. test/install-fixture.test.ts is the ENG-1 tripwire: put/setVersion/chmod in one clone leave the seed and a sibling unchanged, and clones share no inodes with the read-only seed. * test(free): split claude-overlay-setup by describe (W6b) The 115 s serial file becomes default/lifecycle/installs files (5/4/3 tests, unchanged bodies) sharing test/helpers/claude-overlay-fixture.ts, so the shard packer can balance them. * test(cso): convert cso-contracts.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval). Sentence pins: 16 -> 8. * test(cso): convert cso-preserved.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval). Sentence pins: 10 -> 3. * test(cso): convert cso-eval.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: cso E2E/judge cases (skill-e2e-cso*, cso-eval). Sentence pins: 7 -> 4. * test(browser): convert aside-driver.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: Aside/browse E2E cases; the installer ban stays a meaning check. Sentence pins: 27 -> 9. * test(browser): convert third-party-actions.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: Aside/browse E2E cases; the installer ban stays a meaning check. Sentence pins: 17 -> 2. * test(skills): convert skill-validation.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 19 -> 6. * test(skills): convert workflow-excerpt.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 37 -> 10. * test(skills): convert question-tuning-registry-path.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 4 -> 0. * test(skills): convert host-conformance.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: workflow judges in skill-llm-eval. Sentence pins: 4 -> 4. * test(memory): convert memory-ingest-landing.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: memory-ingest free tests that run the binary. Sentence pins: 11 -> 9. * test(windows): exclude the skip-only claude-overlay setup files from the Windows subset Their real-setup cases are describe.skipIf(win32); after the split their bin/ spawns live in the shared helper, so the source scan stopped seeing them. * test(skills): convert agents-digest.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 1 -> 0. * test(skills): convert brain-preflight.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 2 -> 1. * test(skills): convert codex-skill-isolation.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 1 -> 0. * test(skills): convert office-hours-review.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 2 -> 1. * test(skills): convert onboarding-moved-literals.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 2 -> 2. * test(skills): convert open-gstack-browser-liveness.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 3 -> 2. * test(skills): convert plan-floor-review.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 3 -> 2. * test(skills): convert plan-review-report-recording.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 4 -> 2. * test(skills): convert regression-1539-review-self-verify.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 4 -> 0. * test(skills): convert skill-fixture.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 6 -> 3. * test(skills): convert template-context-parity.test.ts sentence pins Safety and authority sentences become expectMentions keyword checks (reword freely, keep the rule); other English sentence pins are dropped. Headings, labels, placeholders, commands, field names, enum/status tokens and ordering checks stay. Behavior and wording: the owning skills' E2E cases and workflow judges; CLI/hook output and fixture-data pins in the tail were left alone. Sentence pins: 1 -> 0. * test: restore pins on renderer output and paid-fixture inputs cso-contracts (the report renderer's required empty phrase), memory-ingest (binary output), carve-plan-fixture, ship-skip-actor and cso-eval (inputs the paid harness builds) assert product or fixture output, not skill prose, so the value-bar narrowing does not apply to them. * feat(eval-policy): v2 series identity, HARNESS_VERSION guard and identity backtest W4a (CEO-27/ENG-3/DX-6), not yet wired to the writer: caseSeriesIdentitiesV2 keys a case by the bytes it owns (paid test file, fixtures, its skills' prompt files including the generated SKILL.md) plus HARNESS_VERSION, and records the full consumed-input fingerprint as provenance. scripts/harness-version.json pins the blob of every shared-harness file (GLOBAL_TOUCHFILES plus the session runners, PTY harness, e2e gate and LLM judge), one line per file; test/harness-version.test.ts fails on drift with both fixes, and scripts/bump-harness-version.ts records the decision: --bump (new series for every case) or --non-behavioral (compatibility entry, no reset). eval-trial-series --backtest replays identities over weekly main revisions and recorded trials (ENG-3/DX-12). * feat(pass-rates): weekly history is main's scheduled runs plus main dispatches W4b: pass-rates reads evals-periodic history from main only, keeping scheduled runs and main dispatches and never a branch dispatch, so back-to-back branch validation runs neither feed pass rates nor count as weeks for quarantine expiry or drift. --branch <name> still reads one branch's runs for inspection. * docs(test-audit): EVAL_POLICY v2 decision memo with the identity backtest D1 memo (CEO-33, ENG-3, DX-12, CEO-27): P(green) arithmetic, false-blocking targets, the v2 identity rule and harness guard, weekly history, W4c evidence (the named 'stochastic' cases pass 26/27 to 34/34, so they stay rule), the backtest over 29 weekly main revisions and 14,898 recorded trials, what it means for quarantine reachability, alternatives (gating vs trended, adaptive panels not adopted, buy vs build), and the rollback proof on the pre-PR reader. * feat(eval-policy): EVAL_POLICY v2 writer: case-owned series identity The v2 writer, last commit of the W4 block (DX-12/ENG-14/ENG-21): the report job's stamp now writes caseSeriesIdentitiesV2 (case-owned bytes plus HARNESS_VERSION) as series_identity, the full consumed-input fingerprint as series_fingerprint and harness_version as provenance, and EVAL_POLICY.version is 2 (approved 2026-10-04, D1). The v1 touchfile identity is removed. Rollback = revert this commit (and the W4 commits above the reader-compat floor); readers then ignore the v2 records with a printed count. * test: restore pins on paid-harness prompt inputs review-enum-lifecycle and plan-review-report-recording assert the prompt a paid case's harness builds (opts.prompt / call.prompt), a fixture contract, not skill prose. * test(security): check injected safety lines on meaning, not capitals content-security, the canary instruction and the arm judge prompt still must carry their never-follow / never-output / exact-"none" rules; the checks are case-insensitive keyword checks so emphasis can change. The UNTRUSTED envelope markers and "none" value stay exact tokens. (Hidden element stripping fails locally only because Playwright Chromium is not installed; same at base.) * test(plan-ceo): keep the outside-voice scope-summary rule as a meaning check The first codemod pass left an empty if body (TS1313). * chore(windows): re-record the Windows duration seed for the 631-file subset (run 37222668939) * ci(free-tests): summarize every shard's flake ledger in the aggregate job (W7a) The aggregate downloads flake-ledger-* (one subdirectory per shard) and appends lane C's `scripts/test-health-report.ts flake-summary` output to the job summary before strict verification. The runner's own flaky table is dropped so the ledger reader stays the single source; --ci-verify keeps the shard-overrun warning. * feat(evals): free test files anywhere and the stored-zip helper never restore the full PR gate (W1a) The CEO-23 replay found design/test free tests forcing fallback (36485436963). * feat(evals): replay recorded PR pushes through today's selector (CEO-23, ENG-18) scripts/replay-pr-selection.ts reports per push the recorded vs replayed mode and case count, dropped gate cases, retired failed cases and every miss (a failed case that depends on the diff but is no longer selected). * refactor(evals): one case-sharded carve-section-loading file with a case id and touchfiles per skill (W5a) * test(free): hold qa-deadline's full-pipe case on a FIFO, not a paused Bun stream Bun gives a child 'pipe' stdio a unix socketpair, so whether the 2 MB fill actually blocks depends on the parent's reader and the host's socket buffer sizes. CI (run 36761671811, both attempts) saw exit 0 after ~130 ms; with net.core.{w,r}mem_default=4 MiB locally the old case fails 3/3 the same way. A FIFO the test never reads is bounded by the kernel pipe buffer, so the receipt write genuinely blocks and the 5 s settlement deadline must fire. The 7 s bound now starts once the preload has queued the write. * test(free): event-driven waits in pty-output-wake close-releases-observers: the 500 ms wall race becomes an ordering check (the waiter is released while only SIGINT has been sent, before close's escalation finishes), and the Bun.spawn spy fakes only the PTY launch so a shard neighbor's in-process spawn cannot add a stray SIGTERM (CI runs 36015568981, 36036582721, 36074636848). continuous-redraws: the fixture spinner stops once the observer has made five coalesced reads, so the run reaches its work deadline with no redraw in flight. Streaming right up to the deadline raced the helper's screen drain and failed as 'PTY screen parse failed; viewport is incomplete' (5 CI runs 09-30..10-03; 4/40 locally under 8 busy loops, 0/40 after). * test(free): session-runner long-capture handshake waits for the collected Read event The fake CLI's init line arms the runner's work-phase timer, so the worker's handshake (work-ready + a >200 s timer) could pass before the runner's read loop had collected the Read event printed just before work-ready. Advancing the clock then killed the run with the Read lost: readSection false (CI run 37194511124). The worker now also waits for the runner's own progress line for that Read. Forcing the observed ordering (work-ready 200 ms before the Read line) fails the old test deterministically and passes the new one. * test(free): budget-override restores the CI variables it rewrites Its provenance cases set and then deleted CI, GITHUB_ACTIONS, GITHUB_REF_NAME, GITHUB_SHA and the GitLab CI_* names without restoring them, so in a CI shard every later file ran with CI unset. qa-only-cleanup's real browse daemons then launched Chromium without --no-sandbox, the sandbox failed on the runner, and both real-Chromium cases timed out with 'QA cleanup fixture did not become ready' (shard 12 attempt 1 of six runs on 10-04, retry alone green). Retained startup log: 'Chromium sandboxing failed!', child env with no CI. Reproduced 3/3 with CI=true, userns denied and budget-override ahead of qa-only-cleanup in one process; 0/3 after. * test(free): pr-shared-input-selection names its base and package history Every selection now passes EVALS_BASE and packageVersionOnly explicitly, so no case reads the checkout's git state. Before, 12 calls ran detectBaseBranch and the cumulative case diffed package.json against merge-base(origin/main, HEAD) — the ambient input that turned main red in runs 36572856858 and 36594314027. A PATH git spy counts 14 git calls in the old file and 0 now. * feat(evals): derive PR-lane dependents for unmapped files instead of restoring the full gate scripts/pr-dependencies.ts derives, from the checked-out tree, which paid cases consume each repository file: test import closures, repo paths and path.join segments in code/shell/skill text (whole-line comments and selection data excluded), bin script names, compiled binaries by outfile path (scripts/build.sh), package scripts, template placeholders through RESOLVERS, and host configs by host id. The global touchfiles' import closure is global. PR selection: an unmapped file with derived consumers puts the run in the new 'dependents' mode (every dependent gate case and judge, no profile cut); a tracked file under a derivable directory that nothing reaches is exempt; periodic/marathon workflows, .gitignore and tsconfig.test.json never run the PR gate. PR_FULL_GATE_FILES is the fail-closed list (deps, CI image, setup, build, evals.yml and its actions, test preload, tsconfig.json, .gitattributes, the skill generator and preamble resolvers, the eval policy). Untracked or underivable files restore the full gate with fix text. The summary and PR comment list derived dependents. * feat(evals): replay misses count derived dependents; report wave vs ordinary pushes (CEO-23) * test: drop FREE_FIXTURES entries for fixtures deleted in W8b * test: re-shape the census attribution fixture to qa-workflow and carve case shards * test: census fixture case order * ci: 420 s PR-lane slices with per-slice ceilings; periodic tracking issue only on main evals.yml joins the census lanes at the 420 s slice budget (W5c/CEO-15) and indexes each eval slice's job timeout from .plan.sliceCiTimeoutMinutes with the slice-deadline clock as its first step (W2c/ENG-2). evals-periodic.yml's tracking-issue upsert and close run only on refs/heads/main, so a branch dispatch whose gate census is skipped (CEO-04) never opens or closes #2755. * docs: integrate lane docs for the audit wave (eval-bg backends, PR fallback, policy v2, ceilings, evidence) - CLAUDE.md: 'Running evals as an agent' describes scripts/eval-bg.ts's dispatch and local backends, forcing flags, the sentinel and status; eval:select's PR-profile default; prompt-structure helper and value-bar terms. instruction-facts now requires scripts/eval-bg.ts in that section. - CONTRIBUTING.md: decision table gains test:health, eval:select default, eval:bg backends and the include_gate_census / evals_all dispatch lines; 420 s runners and per-runner ceilings; flake ledger, OSV tracking issue and expiry test, weekly test health; prompt-structure guidance. - AGENTS.md: build commands name the eval:bg backend choice and test:health. - docs/TESTING_INTERNALS.md: anchors the fix texts point to (#pr-paid-lane-fallback, #pass-rate-policy-versions, #harness-version, #flake-ledger, #duration-seed), receipts/recovery/debounce, per-slice ceilings, the slice deadline with not_run/hung and checkpoints, the eval-bg cap formula in place of the detach floors, EVAL_POLICY v2 history, carve case shards via --case. - docs/test-audit-2026-10.md: CEO-23 replay (165/168 -> 164/168 full-fallback, 0 misses), ENG-17 slice table (re-measured on this tree), D2 prose-pin metric. - TODOS.md: progress note on 'Let pass-rate history decide the rest'. - test/testing-internals-anchors.test.ts: every TESTING_INTERNALS.md#anchor named in code must match a heading. * test(free): hold qa-deadline's blocked-output-pipe receipts on a FIFO The five 'guard %s receipts survive a blocked output pipe' cases filled a paused Bun child pipe with 2 MiB. That pipe is a unix socketpair, so the fill only blocks while the host's socket buffers are smaller than the write: with net.core.{w,r}mem_default=4 MiB the old cases fail 13 of 15 case runs (the guard finishes before the test resumes the stream). The held stream is now a FIFO the test drains only after resume(), bounded by the kernel pipe buffer; Windows keeps the paused pipe. Bounds and assertions are unchanged. * test(free): qa-deadline waits for a written pid, not just the pid file DRIVER and LEAF write their pid with writeFileSync, so under load ready() can see the file before its digits and fail 'expected > 0, received 0' (seen once in 150 loaded runs). After any early failure the finally SIGKILLs the CLI, which orphans the detached command group; a leaf whose pid file is not yet readable then keeps the output stream open and the case hangs to its 15 s timeout (2/80 old-file runs and 3/310 FIFO-only runs under 8 busy loops; the instrumented hang showed the CLI already SIGKILLed by the finally). ready() now polls within the same 3 s bound until the file holds a positive pid. * fix(evals): a deleted file selects the dependents of the live files that still reference it PR selection treated every path absent from the head tree as an unknown dependency, so any PR that deleted a file forced the full gate. A deleted path is now placed like a live one, against the head tree: a matching touchfile selects its cases; live code, skill text or a full-gate input that still references it (path, extensionless import target, relative import, bin name, ancestor directory, compiled-binary outfile; whole-line comments and data files excluded) contributes its dependents, a full-gate referencer restores the full gate with a 'deleted but still referenced' fix, and an unreferenced deleted path has no dependents. Deleted PR_FULL_GATE_FILES entries still force the full gate. Tests that used absent paths as stand-ins for new unmapped files now use tracked root files (or an explicit tracked set); the CEO-23 replay counts misses through those references too. * docs: integrate free-suite lane docs (seed anchor, Windows lane, native qualification, install fixture, flake rules) - docs/TESTING_INTERNALS.md: '### Free suite duration seed' (the anchor the seed messages link) with DX-13, the 99th-percentile packing, --ci-plan warning, --ci-verify overruns and the CEO-10 seed ratchet; the Windows 6-job lane, its Windows-measured seed and refresh dispatch, curation and WINDOWS_CURATED_FLOOR; native-qualification.yml modes; the merged install fixture; test-only timing knobs; the flake-ledger job summary, Windows ledger names and three free-suite flake rules; zsh in the Ubicloud mirror. - scripts/test-health-report.ts links #free-suite-duration-seed, so the two seed messages share one heading (the anchor test caught the split). - CONTRIBUTING.md: Windows seed refresh and native qualification rows; seed-source pointer. AGENTS.md: Windows six-job lane. - docs/cso-release-qualification.md: the free CSO Docker job runs only cso-docker-integration.test.ts. - docs/test-audit-2026-10.md: free-suite evidence lines. - TODOS.md: CEO-11b progress note on the pre-push hook hang. * test: final duration seeds on the integrated tree; retire the emptied cookie-validation-phases file Linux seed re-recorded on Ubicloud standard-16 and the Windows seed from dispatch run 37227140871, both atc903310. The seed allowlist drops the three entries that are no longer over the ceiling. cookie-validation-phases lost its validation-phase pins with W8f and its Windows/Dia pins moved to windows-native-workflows with native-qualification.yml, so the file is deleted and the curated Windows floor rises to 638. The Windows recorder writes the seed at its default path so GSTACK_FREE_TEST_DURATIONS never leaks into timed tests. * test: re-express stale PR-lane pins against the new evals.yml contract - ci-eval-cache: the profile/fresh expressions now include evals_all=false dispatches and pr_receipts/evals-fresh; the PR store is restored only by recover-receipts (still scoped to repository_id + PR, PR events only), plan-slices takes it as an artifact and never touches the cache, saves stay PR-only, EVALS_CACHE_PR binds pr_receipts dispatches, and the report verdict artifact carries report receipts. - ci-paid-coordination: plan-slices needs only the two ubuntu coordination jobs (never the image); the fork/Dependabot clause is unchanged on plan-slices and also gates debounce and recover-receipts; only recovery holds actions: read. - qa-supervision-selection: auq-consistency and auq-verbose-vs-carved-ab (W1b) reach claude-pty-runner and pty-screen through auq-sdk-capture, exactly as auq-format-gate does. - paid-retry-supervision: the full-validation plan uses a tracked full-gate input (setup); an absent path is now a deletion placed by its references. * ci(windows): record the Windows seed at the default path * chore: HARNESS_VERSION non-behavioral decision for the audit-wave integration * v1.91.20.0 release metadata: test, eval and CI audit wave * test(compare-board): wait for the post-submit state the feedback POST settles The two post-submit assertions read the board once, but the board disables the submit button and shows the message only after its feedback POST resolves. On a loaded Ubicloud shard the read beat the round trip (FLAKY-PASS in the final free-suite run). Delaying the POST by 400 ms reproduces both failures at HEAD; polling for the settled state passes. * test(plan-eng plan-mode): run the seeded STOP-gate smoke in its own fixture project The seeded plan proposes a custom test runner, so the review's Step 0 reads the working tree's test setup. In the gstack checkout that is gstack's own sharded runner, and the case timed out at 300 s in census 37228573062 after this wave reshaped the runner. It now runs in a four-file Bun project with git history, with the product-notice suppression auto-decide-preserved's standalone fixture already uses. Local pass on Bun 1.4.0 + Claude Code 2.1.284: asked in 121 s, seed tokens observed. * test(plan-mode evidence): pin the seeded eng smoke's isolated fixture cwd and env * ci(image): Git >= 2.44 so paid shared-libs cases run the supported safe-git path bin/gstack-safe-git prefixes every /deslop-shared-libs Git read with --no-lazy-fetch, which noble's git 2.43 rejects (exit 129). The eval image therefore ran every paid shared-libs case on the unsupported-Git fallback: local history unavailable, branch HEAD unknown. shared-libs-read-only failed on exactly that in PR run 37230639146 ("Current branch HEAD: unknown. The safe-git probe failed (unknown option: --no-lazy-fetch, git 2.43.0)") and in 2 of 74 earlier PR runs. The image installs current Git from the Ubuntu Git Maintainers PPA and fails its build if --no-lazy-fetch is missing; the deliberate unsupported-Git case keeps its fake Git. * eval: pool matching branch census trials into main's pass-rate series (D1 option b) Approved 2026-10-05. eval:pass-rates also reads completed branch census runs in the weekly window. A branch trial counts only toward a series a main run also has (same case-owned bytes, HARNESS_VERSION, model and CLI), so a branch extends main's history but never starts or becomes a case's current series; weeks for quarantine expiry still count main runs only. The D1 backtest put this at 67 of 210 cases with enough trials for the entry rule, against 0 of 195 on main alone. * C2: /qa defines its outcome filename fields; QA report templates name the measured timing fields Low judge samples in failing panels named both: qa workflow (37186854666) 'an undefined {user} in the outcome filename', and qa-only workflow (37198445662) 'template asks for Duration / durationMs totals while reporting.md mandates Probe budget and Guarded command time'. The qa union stays under its 1.11 size ratio (93292 of 93294 bytes). (cherry picked from commite14c0c1de1) * C2: Browser Quick gets a feasible budget; QA loop names its directory and stop path Judge rationales on the qa-only panels (2 of the last 3) cited the 30 s Browser Quick budget: homepage + 5 targets, each a checkpoint write and a guarded command, left ~5 s per probe. Quick is now 180 s (method text: 3 minutes); the qa-only-no-fix paid check expects the same native deadline. PROBE_DIR points at its source line, a stopped loop goes to the final report steps (annotations, materialize, Markdown), and /review and /ship say that after smoke expiry only plan checks rerun in revalidation. (cherry picked from commit78d4a2a343) * qa: fit the C2 judge fixes under the qa and review size caps Brings GSTA-21's qa.ts trims (3bd625d,ce73af0) for the cherry-picked C2 wording: PROBE_DIR keeps 'owned', the stop line points at §4, and the post-expiry rule is one sentence, so /review fits its cap. qa measures 93,303 / 84,049 = 1.11010 against the same 1.1102 cap GSTA-21 set. The qa-only workflow judge passed 10 of 10 local trials on this wording (7 of 10 before the C2 fixes). * test: qa-only fixture keeps its caller-set 30 s deadline The qa-only-no-fix fixture tells the model to start the guard with 30 s (a caller limit), so its deadline assertion's default stays 30 000 ms; the product's Browser Quick default (180 s) does not apply to it. Reverts the helper and its test to their base. (cherry picked from commite36e622add) * test: qa-quick sets a caller probe limit and drains past its session budget C2 raised Browser Quick's native budget to 180 s. qa-quick is the only gate case on the native budget, and its wall went 194 s (main periodic, pre-C2) -> 254 s and 274 s (this PR's runs) -> 300 s session timeout (run 37242691913): the agent now probes every navigation target with a checkpoint, then needs ~80 s to report. Like qa-only-no-fix, the eval now sets a caller probe limit (30 s), scopes the fixture, and gives bun's outer timeout the session drain margin instead of racing the 300 s session. The product's 180 s budget and the session budget are unchanged. Diagnostic rerun passed in 195 s. (cherry picked from commitf8dc64031e) * chore: restore the audit wave's package scripts after the main merge (version 1.91.23) * qa: keep main's one-sentence post-expiry rule for /review and /ship The C2 variants ("step 4c reruns plan checks only" and "even in 4c") read to the review workflow judge as conflicting with Step 5.8's repeat-pass reruns: 6 of 10 local trials passed with the longer sentence and PR run 37248650879 failed on the shorter one. Main's "Post-expiry smoke rechecks are not-run." passed 10 of 10. qa (9/10), qa-only (10/10) and ship-managed-hook-refresh (10/10) were measured on the same tree. * v1.91.24.0: move off the slot #3036 claims
4881 lines
239 KiB
TypeScript
4881 lines
239 KiB
TypeScript
import { describe, test, expect, afterAll } from 'bun:test';
|
||
import { assertSinglePreamble } from '../scripts/gen-skill-docs';
|
||
import { validateSkillFrontmatter } from '../scripts/skill-check';
|
||
import { externalHostPathLeaks } from './helpers/skill-parser';
|
||
import { COMMAND_DESCRIPTIONS } from '../browse/src/commands';
|
||
import { SNAPSHOT_FLAGS } from '../browse/src/snapshot';
|
||
import * as fs from 'fs';
|
||
import * as path from 'path';
|
||
import * as os from 'os';
|
||
import { spawnSync } from 'child_process';
|
||
import { runCapturedCommand } from './helpers/sync-command-capture';
|
||
import { expectMentions, expectTokens } from './helpers/prompt-structure';
|
||
|
||
const ROOT = path.resolve(import.meta.dir, '..');
|
||
const MAX_SKILL_DESCRIPTION_LENGTH = 1024;
|
||
|
||
// Carved-skill aware (v2 plan T9): ship is now a skeleton SKILL.md + sections/*.md.
|
||
// Read the union so assertions about content that MOVED into a section still pass.
|
||
// The skeleton is a subset of the union, so skeleton-only assertions also hold,
|
||
// and negative assertions stay safe (the absent phrases live in neither file).
|
||
function readSkillUnion(skill: string): string {
|
||
let t = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
const secDir = path.join(ROOT, skill, 'sections');
|
||
if (fs.existsSync(secDir)) {
|
||
for (const f of fs.readdirSync(secDir).sort()) {
|
||
if (f.endsWith('.md')) t += '\n' + fs.readFileSync(path.join(secDir, f), 'utf-8');
|
||
}
|
||
}
|
||
return t;
|
||
}
|
||
function readShipUnion(): string {
|
||
return readSkillUnion('ship');
|
||
}
|
||
|
||
function ordered(text: string, anchors: string[]): void {
|
||
const positions = anchors.map(anchor => text.indexOf(anchor));
|
||
expect(positions.every(position => position >= 0), `missing one of ${JSON.stringify(anchors)}`).toBe(true);
|
||
expect(positions).toEqual([...positions].sort((a, b) => a - b));
|
||
}
|
||
|
||
function readExternalSkillUnion(output: string, hostSubdir: string, skill: string): string {
|
||
const dir = path.join(output, hostSubdir, 'skills', `gstack-${skill}`);
|
||
const sections = path.join(dir, 'sections');
|
||
return fs.readFileSync(path.join(dir, 'SKILL.md'), 'utf-8')
|
||
+ (fs.existsSync(sections) ? fs.readdirSync(sections).sort()
|
||
.filter(file => file.endsWith('.md'))
|
||
.map(file => '\n' + fs.readFileSync(path.join(sections, file), 'utf-8')).join('') : '');
|
||
}
|
||
|
||
|
||
// Token-reduction Phase 1: the preamble's inline bash (session bookkeeping,
|
||
// config echoes, telemetry producers, artifacts sync) moved into
|
||
// bin/gstack-skill-start / bin/gstack-skill-end. The render carries a one-line
|
||
// invocation fence + interpretation prose. Assertions that pinned inline-bash
|
||
// internals now pin the scripts (the new home); render-side assertions pin the
|
||
// fence + prose. Script behavior is pinned by test/gstack-skill-start.test.ts.
|
||
const SKILL_START_SCRIPT = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
|
||
const SKILL_END_SCRIPT = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-end'), 'utf-8');
|
||
|
||
function extractDescription(content: string): string {
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
expect(fmEnd).toBeGreaterThan(0);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
const lines = frontmatter.split('\n');
|
||
let description = '';
|
||
let inDescription = false;
|
||
const descLines: string[] = [];
|
||
|
||
for (const line of lines) {
|
||
if (line.match(/^description:\s*\|?\s*$/)) {
|
||
inDescription = true;
|
||
continue;
|
||
}
|
||
if (line.match(/^description:\s*\S/)) {
|
||
return line.replace(/^description:\s*/, '').trim();
|
||
}
|
||
if (inDescription) {
|
||
if (line === '' || line.match(/^\s/)) {
|
||
descLines.push(line.replace(/^ /, ''));
|
||
} else {
|
||
break;
|
||
}
|
||
}
|
||
}
|
||
|
||
if (descLines.length > 0) {
|
||
description = descLines.join('\n').trim();
|
||
}
|
||
return description;
|
||
}
|
||
|
||
function extractMarkdownSection(content: string, heading: string): string {
|
||
const escaped = heading.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||
const startMatch = content.match(new RegExp(`^${escaped}.*$`, 'm'));
|
||
expect(startMatch?.index).toBeDefined();
|
||
const start = startMatch!.index!;
|
||
const afterHeading = start + startMatch![0].length;
|
||
const nextSection = content.slice(afterHeading).match(/\n## /);
|
||
const end = nextSection?.index === undefined
|
||
? content.length
|
||
: afterHeading + nextSection.index;
|
||
return content.slice(start, end).trim();
|
||
}
|
||
|
||
function extractPreambleBeforeWorkflow(content: string, workflowMarkers: string[]): string {
|
||
const markerIndexes = workflowMarkers
|
||
.map(marker => content.indexOf(marker))
|
||
.filter(index => index >= 0);
|
||
expect(markerIndexes.length).toBeGreaterThan(0);
|
||
return content.slice(0, Math.min(...markerIndexes));
|
||
}
|
||
|
||
function isRepoRootSymlink(candidateDir: string): boolean {
|
||
try {
|
||
return fs.realpathSync(candidateDir) === fs.realpathSync(ROOT);
|
||
} catch {
|
||
return false;
|
||
}
|
||
}
|
||
|
||
// Dynamic template discovery — matches the generator's findTemplates() behavior.
|
||
// New skills automatically get test coverage without updating a static list.
|
||
const ALL_SKILLS = (() => {
|
||
const skills: Array<{ dir: string; name: string }> = [];
|
||
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
||
skills.push({ dir: '.', name: 'root gstack' });
|
||
}
|
||
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
||
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||
if (fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) {
|
||
skills.push({ dir: entry.name, name: entry.name });
|
||
}
|
||
}
|
||
return skills;
|
||
})();
|
||
|
||
// The claude host deliberately skips some skills (skipSkills — e.g. the
|
||
// /claude-code outside-voice skill exists only for non-Claude hosts), so those
|
||
// dirs have a SKILL.md.tmpl but no generated claude-host SKILL.md on a fresh
|
||
// checkout. Every generated-file assertion must exclude them or it is red on
|
||
// every clean clone (it was, invisibly, until the free suite ran in CI).
|
||
import { getHostConfig as __getHostConfig } from '../hosts/index';
|
||
const CLAUDE_SKIPPED = new Set(__getHostConfig('claude').generation.skipSkills ?? []);
|
||
const CLAUDE_GENERATED_SKILLS = ALL_SKILLS.filter(s => !CLAUDE_SKIPPED.has(s.dir));
|
||
|
||
// ─── Out-dir render isolation ────────────────────────────────
|
||
// Every generator invocation in this file that used to regenerate the live
|
||
// tree (the gitignored .agents/.factory/... host dirs included) now renders
|
||
// into this module-level out-dir: ONE `--host all` render covers the claude
|
||
// host plus every external host, and all golden-artifact reads plus the
|
||
// per-host `--dry-run` determinism checks point here. The tracked tree is
|
||
// only ever READ (the `generated files are fresh` dry-run deliberately
|
||
// compares against the committed files — that is a read, not a write).
|
||
// Out-dir renders of external hosts are byte-identical to in-place renders
|
||
// (pinned by test/gen-skill-docs-out-dir.test.ts).
|
||
const EXTERNAL_OUT = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-gen-docs-out-'));
|
||
{
|
||
const render = Bun.spawnSync(
|
||
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--out-dir', EXTERNAL_OUT],
|
||
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 },
|
||
);
|
||
if (render.exitCode !== 0) {
|
||
throw new Error(
|
||
`gen-skill-docs --host all --out-dir failed (exit ${render.exitCode}):\n${render.stderr.toString()}`,
|
||
);
|
||
}
|
||
}
|
||
afterAll(() => {
|
||
fs.rmSync(EXTERNAL_OUT, { recursive: true, force: true });
|
||
});
|
||
|
||
describe('gen-skill-docs', () => {
|
||
// Browse carve (token-reduction Phase 4): the command reference + snapshot
|
||
// flags render into browse/sections/command-list.md now — read the
|
||
// skeleton+sections union so these pins hold across the carve.
|
||
test('generated SKILL.md contains all command categories', () => {
|
||
const content = readSkillUnion('browse');
|
||
const categories = new Set(Object.values(COMMAND_DESCRIPTIONS).map(d => d.category));
|
||
for (const cat of categories) {
|
||
expect(content).toContain(`### ${cat}`);
|
||
}
|
||
});
|
||
|
||
test('generated SKILL.md contains all commands', () => {
|
||
const content = readSkillUnion('browse');
|
||
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
||
const display = meta.usage || cmd;
|
||
expect(content).toContain(display);
|
||
}
|
||
});
|
||
|
||
test('command table is sorted alphabetically within categories', () => {
|
||
const content = readSkillUnion('browse');
|
||
// Extract command names from the Navigation section as a test
|
||
const navSection = content.match(/### Navigation\n\|.*\n\|.*\n([\s\S]*?)(?=\n###|\n## )/);
|
||
expect(navSection).not.toBeNull();
|
||
const rows = navSection![1].trim().split('\n');
|
||
const commands = rows.map(r => {
|
||
const match = r.match(/\| `(\w+)/);
|
||
return match ? match[1] : '';
|
||
}).filter(Boolean);
|
||
const sorted = [...commands].sort();
|
||
expect(commands).toEqual(sorted);
|
||
});
|
||
|
||
test('snapshot flags section contains all flags', () => {
|
||
const content = readSkillUnion('browse');
|
||
for (const flag of SNAPSHOT_FLAGS) {
|
||
expect(content).toContain(flag.short);
|
||
expect(content).toContain(flag.description);
|
||
}
|
||
});
|
||
|
||
test('every skill has a SKILL.md.tmpl template', () => {
|
||
for (const skill of ALL_SKILLS) {
|
||
const tmplPath = path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
||
expect(fs.existsSync(tmplPath)).toBe(true);
|
||
}
|
||
});
|
||
|
||
test('every skill has a generated SKILL.md with auto-generated header', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const mdPath = path.join(ROOT, skill.dir, 'SKILL.md');
|
||
expect(fs.existsSync(mdPath)).toBe(true);
|
||
const content = fs.readFileSync(mdPath, 'utf-8');
|
||
expect(content).toContain('AUTO-GENERATED from SKILL.md.tmpl');
|
||
expect(content).toContain('Regenerate: bun run gen:skill-docs');
|
||
}
|
||
});
|
||
|
||
// #1778: strict YAML parsers (Codex/OpenAI skill loading) reject frontmatter
|
||
// whose plain `description:` scalar contains an interior ": " (read as a nested
|
||
// mapping). Parse EVERY generated frontmatter block with a strict YAML parser,
|
||
// not just string-check that name:/description: exist.
|
||
test('every generated SKILL.md frontmatter parses as strict YAML', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
expect(validateSkillFrontmatter(content),
|
||
`frontmatter for ${skill.dir} must be valid YAML with string name/description`).toEqual([]);
|
||
}
|
||
});
|
||
|
||
test('every generated Codex (.agents/skills) frontmatter parses as strict YAML', () => {
|
||
// Reads the module-level out-dir render (guaranteed present — the render
|
||
// throws at module load if it fails), never the live gitignored tree.
|
||
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
||
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
||
if (!entry.isDirectory()) continue;
|
||
const mdPath = path.join(agentsDir, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(mdPath)) continue;
|
||
expect(validateSkillFrontmatter(fs.readFileSync(mdPath, 'utf-8')),
|
||
`Codex frontmatter for ${entry.name} must be valid YAML with string name/description`).toEqual([]);
|
||
}
|
||
});
|
||
|
||
test(`every generated SKILL.md description stays within ${MAX_SKILL_DESCRIPTION_LENGTH} chars`, () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
const description = extractDescription(content);
|
||
expect(description.length).toBeLessThanOrEqual(MAX_SKILL_DESCRIPTION_LENGTH);
|
||
}
|
||
});
|
||
|
||
test(`every Codex SKILL.md description stays within ${MAX_SKILL_DESCRIPTION_LENGTH} chars`, () => {
|
||
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
||
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
||
if (!entry.isDirectory()) continue;
|
||
const skillMd = path.join(agentsDir, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
const description = extractDescription(content);
|
||
expect(description.length).toBeLessThanOrEqual(MAX_SKILL_DESCRIPTION_LENGTH);
|
||
}
|
||
});
|
||
|
||
test('every Codex SKILL.md description stays under 900-char warning threshold', () => {
|
||
const WARN_THRESHOLD = 900;
|
||
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
||
const violations: string[] = [];
|
||
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
||
if (!entry.isDirectory()) continue;
|
||
const skillMd = path.join(agentsDir, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
const description = extractDescription(content);
|
||
if (description.length > WARN_THRESHOLD) {
|
||
violations.push(`${entry.name}: ${description.length} chars (limit ${MAX_SKILL_DESCRIPTION_LENGTH}, ${MAX_SKILL_DESCRIPTION_LENGTH - description.length} remaining)`);
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
|
||
test('package.json version matches VERSION file (npm-valid translation)', () => {
|
||
// Decision 11 (v1.67 wave): VERSION stays the 4-digit source of truth;
|
||
// package.json carries the npm-valid 3-digit translation (npm rejects a
|
||
// fourth component). The pre-v1.67 1:1 four-digit mirror is also accepted
|
||
// (grandfathered until the next write), matching gstack-version-bump's
|
||
// own drift contract.
|
||
const pkg = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf-8'));
|
||
const version = fs.readFileSync(path.join(ROOT, 'VERSION'), 'utf-8').trim();
|
||
const npmTranslation = version.split('.').slice(0, 3).join('.');
|
||
expect([npmTranslation, version]).toContain(pkg.version);
|
||
});
|
||
|
||
test('generated files are fresh (match --dry-run)', () => {
|
||
// Deliberately compares against the LIVE TRACKED SKILL.md files (no
|
||
// --out-dir): this is the freshness gate for the committed tree. Dry-run
|
||
// writes nothing — it is a read.
|
||
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--dry-run'], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
// Every skill should be FRESH
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const file = skill.dir === '.' ? 'SKILL.md' : `${skill.dir}/SKILL.md`;
|
||
expect(output).toContain(`FRESH: ${file}`);
|
||
}
|
||
expect(output).not.toContain('STALE');
|
||
});
|
||
|
||
test('no generated SKILL.md contains unresolved placeholders', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
const unresolved = content.match(/\{\{\w+\}\}/g);
|
||
expect(unresolved).toBeNull();
|
||
}
|
||
});
|
||
|
||
test('templates contain placeholders', () => {
|
||
// P2 (v1.2.0): the root template is a pure router — only {{PREAMBLE}}.
|
||
// The browse command/snapshot placeholders live in browse/SKILL.md.tmpl now.
|
||
const rootTmpl = fs.readFileSync(path.join(ROOT, 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(rootTmpl).toContain('{{PREAMBLE}}');
|
||
expect(rootTmpl).not.toContain('{{COMMAND_REFERENCE}}');
|
||
expect(rootTmpl).not.toContain('{{SNAPSHOT_FLAGS}}');
|
||
|
||
// Browse carve: the reference resolvers moved into the on-demand section
|
||
// template (so gen-skill-docs keeps them fresh from browse/src); the
|
||
// skeleton points at the section instead of inlining the reference.
|
||
const browseTmpl = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(browseTmpl).not.toContain('{{COMMAND_REFERENCE}}');
|
||
expect(browseTmpl).not.toContain('{{SNAPSHOT_FLAGS}}');
|
||
expect(browseTmpl).toContain('{{SECTION:command-list}}');
|
||
expect(browseTmpl).toContain('{{PREAMBLE}}');
|
||
|
||
const browseSectionTmpl = fs.readFileSync(
|
||
path.join(ROOT, 'browse', 'sections', 'command-list.md.tmpl'), 'utf-8');
|
||
expect(browseSectionTmpl).toContain('{{COMMAND_REFERENCE}}');
|
||
expect(browseSectionTmpl).toContain('{{SNAPSHOT_FLAGS}}');
|
||
|
||
// Aside is the primary browser: every browsing skill renders the Aside
|
||
// contract ({{ASIDE_SETUP}}); the browse binary is its fallback.
|
||
const qaTmpl = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(qaTmpl).not.toContain('{{ASIDE_SETUP}}');
|
||
expect(qaTmpl).toContain('{{SECTION:browser-setup}}');
|
||
expect(fs.readFileSync(path.join(ROOT, 'qa/sections/browser-setup.md.tmpl'), 'utf8'))
|
||
.toContain('{{ASIDE_SETUP}}');
|
||
expect(browseTmpl).toContain('{{ASIDE_SETUP}}');
|
||
});
|
||
|
||
test('generated SKILL.md contains operational self-improvement (replaced contributor mode)', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('Contributor Mode');
|
||
expect(content).not.toContain('gstack_contributor');
|
||
expect(content).not.toContain('contributor-logs');
|
||
expect(content).toContain('Operational Self-Improvement');
|
||
expect(content).toContain('gstack-learnings-log');
|
||
// The learnings-resurface call moved from the inline preamble bash into
|
||
// the skill-start script (Phase 1) — same command, new home.
|
||
expect(SKILL_START_SCRIPT).toContain('gstack-learnings-search" --limit 3');
|
||
});
|
||
|
||
test('generated SKILL.md with LEARNINGS_LOG contains operational type', () => {
|
||
// Check a skill that has LEARNINGS_LOG (e.g., review)
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('operational');
|
||
});
|
||
|
||
test('session awareness lives in gstack-skill-start (registry touch + stale cleanup)', () => {
|
||
// The sessions registry moved from inline preamble bash into the script:
|
||
// it records the harness pid (--parent-pid identity) and expires entries
|
||
// older than 120 minutes.
|
||
expect(SKILL_START_SCRIPT).toContain('sessions/$PARENT_PID');
|
||
expect(SKILL_START_SCRIPT).toContain('-mmin +120');
|
||
// The render keeps the completion-status protocol the sessions feed into.
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('RECOMMENDATION');
|
||
});
|
||
|
||
test('branch detection lives in gstack-skill-start and is echoed as BRANCH', () => {
|
||
expect(SKILL_START_SCRIPT).toContain('_BRANCH=$(git branch --show-current');
|
||
expect(SKILL_START_SCRIPT).toContain('echo "BRANCH: $_BRANCH"');
|
||
});
|
||
|
||
// #2001: update_check: false silences the binary but the upgrade-handling
|
||
// instruction prose used to ship unconditionally. Token-reduction Phase 2
|
||
// made the gate STRUCTURAL: the prose left the renders entirely (absence is
|
||
// pinned by test/onboarding-moved-literals.test.ts) and now emits from
|
||
// gstack-skill-start's instruction layer ONLY when the update-check binary
|
||
// produced output — and that binary silences itself on update_check=false.
|
||
// Opted-out installs can never see the prose, by construction.
|
||
test('update_check opt-out gates the update binary and upgrade-flow emission (issue #2001)', () => {
|
||
// The config-echo cluster lives in gstack-skill-start: the flag is still
|
||
// read and echoed as a STATUS line for the model.
|
||
expect(SKILL_START_SCRIPT, 'script must read update_check config').toContain('_UPDATE_CHECK=$(');
|
||
expect(SKILL_START_SCRIPT, 'script must echo UPDATE_CHECK').toContain('echo "UPDATE_CHECK: $_UPDATE_CHECK"');
|
||
// Gate half 1: the update-check binary exits silently when opted out.
|
||
const updateCheck = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-update-check'), 'utf-8');
|
||
expect(updateCheck, 'binary must read update_check config').toContain('get update_check');
|
||
expect(updateCheck, 'binary must exit silently on update_check=false')
|
||
.toMatch(/if \[ "\$_UC" = "false" \]; then\n\s*exit 0/);
|
||
// Gate half 2: the upgrade-flow instruction block emits only when the
|
||
// binary emitted something (empty when opted out, cached, or up to date).
|
||
expect(SKILL_START_SCRIPT, 'upgrade-flow must be gated on update-check output')
|
||
.toMatch(/if \[ -n "\$_UPD" \]; then\n\s*_emit_block upgrade-flow/);
|
||
});
|
||
|
||
test('tier 2+ skills contain ELI10 simplification rules (AskUserQuestion format)', () => {
|
||
// Root SKILL.md is tier 1 and CSO intentionally uses a private startup with
|
||
// no shared PREAMBLE. Check a regular tier 2+ PREAMBLE consumer instead.
|
||
// v1.7.0.0 Pros/Cons format uses "ELI10 (ALWAYS)" rather than "Simplify (ELI10".
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('ELI10');
|
||
expect(content).toContain('plain English');
|
||
expect(content).toContain('not function names');
|
||
});
|
||
|
||
test('tier 1 skills do NOT contain AskUserQuestion format', () => {
|
||
// Use benchmark (tier 1) instead of root — root SKILL.md gets overwritten by Codex test setup
|
||
const content = fs.readFileSync(path.join(ROOT, 'benchmark', 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('## AskUserQuestion Format');
|
||
expect(content).not.toContain('## Completeness Principle');
|
||
});
|
||
|
||
test('telemetry producer lives in the scripts; render documents the analytics sink', () => {
|
||
// The skill-usage.jsonl producers moved into the scripts (Phase 1).
|
||
expect(SKILL_START_SCRIPT).toContain('analytics/skill-usage.jsonl');
|
||
expect(SKILL_END_SCRIPT).toContain('analytics/skill-usage.jsonl');
|
||
// The render still tells the model where telemetry lands.
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('$GSTACK_STATE_ROOT/analytics/');
|
||
});
|
||
|
||
test('plan-review generated preambles stay under the Option A budget', () => {
|
||
const reviewSkills = [
|
||
{
|
||
path: path.join(ROOT, 'plan-ceo-review', 'SKILL.md'),
|
||
markers: ['# Mega Plan Review Mode', '## Step 0: Detect platform and base branch'],
|
||
},
|
||
{
|
||
path: path.join(ROOT, 'plan-eng-review', 'SKILL.md'),
|
||
markers: ['# Plan Review Mode'],
|
||
},
|
||
];
|
||
|
||
// Plan skills carry the same preamble surface as other tier-≥2 skills
|
||
// (Artifacts Sync, Context Recovery, Routing Injection are load-bearing
|
||
// functionality, not optional). Budget is set to current size + small
|
||
// headroom; ratchet down if a future slim trims real bytes.
|
||
// Ratcheted from 33000 → 35000 when the gbrain context-load block was
|
||
// added (per /sync-gbrain plan §4). Ratcheted 35000 → 36500 in v1.27.0.0
|
||
// when generate-brain-sync-block.ts gained the gbrain_mcp_mode probe +
|
||
// remote-mode ARTIFACTS_SYNC status line (Path 4 of /setup-gbrain).
|
||
// Ratcheted 36500 → 39000 in the contributor wave when #1205 added the
|
||
// \\u-escape CJK rule (rule 12 + self-check item) to the AskUserQuestion
|
||
// preamble.
|
||
// Ratcheted 39000 → 40000 in plan-tune cathedral T14: question-tuning
|
||
// resolver gained the <gstack-qid:...> marker convention + the
|
||
// (recommended) label requirement (D2 + D18 — both load-bearing for
|
||
// hook enforcement). Adds ~700 bytes.
|
||
// Ratcheted 40000 → 60000 in v1.52.0.0 cap audit: ~20K headroom so
|
||
// future preamble adds don't trip the gate on each PR. Real runaway
|
||
// (preamble doubling) still trips; normal scope growth doesn't.
|
||
for (const skill of reviewSkills) {
|
||
const content = fs.readFileSync(skill.path, 'utf-8');
|
||
const preamble = extractPreambleBeforeWorkflow(content, skill.markers);
|
||
expect(Buffer.byteLength(preamble, 'utf-8')).toBeLessThan(60_000);
|
||
}
|
||
});
|
||
|
||
test('voice and writing-style preamble sections stay compact', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
const voice = extractMarkdownSection(content, '## Voice');
|
||
const writingStyle = extractMarkdownSection(content, '## Writing Style');
|
||
|
||
expect(Buffer.byteLength(voice, 'utf-8')).toBeLessThan(3_000);
|
||
expect(Buffer.byteLength(writingStyle, 'utf-8')).toBeLessThan(2_000);
|
||
});
|
||
|
||
test('slim voice section preserves the gstack voice contract', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
const voice = extractMarkdownSection(content, '## Voice');
|
||
|
||
expect(voice).toMatch(/lead with the point|direct/i);
|
||
expect(voice).toMatch(/file|function|line|command|real numbers/i);
|
||
expect(voice).toMatch(/user.*outcome|user.*experience|real user/i);
|
||
expect(voice).toMatch(/corporate|academic|PR|hype/i);
|
||
expect(voice).toMatch(/AI vocabulary|delve|crucial|robust/i);
|
||
expect(voice).toMatch(/user decides|user.*context|sovereignty|recommendation, not a decision/i);
|
||
});
|
||
|
||
test('preamble .pending-* glob is zsh-safe (uses find, not shell glob)', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
if (!content.includes('.pending-')) continue;
|
||
// Must NOT have a bare shell glob ".pending-*" outside of find's -name argument
|
||
expect(content).not.toMatch(/for _PF in [^\n]*\/\.pending-\*/);
|
||
// Must use find to avoid zsh NOMATCH error on glob expansion
|
||
expect(content).toContain("find ~/.gstack/analytics -maxdepth 1 -name '.pending-*'");
|
||
}
|
||
});
|
||
|
||
test('bash blocks with shell globs are zsh-safe (setopt guard or find)', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
const bashBlocks = [...content.matchAll(/```bash\n([\s\S]*?)```/g)].map(m => m[1]);
|
||
|
||
for (const block of bashBlocks) {
|
||
const lines = block.split('\n');
|
||
|
||
for (const line of lines) {
|
||
const trimmed = line.trimStart();
|
||
if (trimmed.startsWith('#')) continue;
|
||
if (!trimmed.includes('*')) continue;
|
||
// Skip lines where * is inside find -name, git pathspecs, or $(find)
|
||
if (/\bfind\b/.test(trimmed)) continue;
|
||
if (/\bgit\b/.test(trimmed)) continue;
|
||
if (/\$\(find\b/.test(trimmed)) continue;
|
||
|
||
// Check 1: "for VAR in <glob>" must use $(find ...) — caught above by the
|
||
// $(find check, so any surviving for-in with a glob pattern is a violation
|
||
if (/\bfor\s+\w+\s+in\b/.test(trimmed) && /\*\./.test(trimmed)) {
|
||
throw new Error(
|
||
`Unsafe for-in glob in ${skill.dir}/SKILL.md: "${trimmed}". ` +
|
||
`Use \`for f in $(find ... -name '*.ext')\` for zsh compatibility.`
|
||
);
|
||
}
|
||
|
||
// Check 2: ls/cat/rm/grep with glob file args must have setopt guard
|
||
const isGlobCmd = /\b(?:ls|cat|rm|grep)\b/.test(trimmed) &&
|
||
/(?:\/\*[a-z.*]|\*\.[a-z])/.test(trimmed);
|
||
if (isGlobCmd) {
|
||
expect(block).toContain('setopt +o nomatch');
|
||
}
|
||
}
|
||
}
|
||
}
|
||
});
|
||
|
||
test('preamble-using skills have correct skill name in telemetry', () => {
|
||
const PREAMBLE_SKILLS = [
|
||
{ dir: '.', name: 'gstack' },
|
||
{ dir: 'ship', name: 'ship' },
|
||
{ dir: 'review', name: 'review' },
|
||
{ dir: 'qa', name: 'qa' },
|
||
{ dir: 'retro', name: 'retro' },
|
||
];
|
||
for (const skill of PREAMBLE_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
// The skill name now travels as --skill into gstack-skill-start (the
|
||
// preamble fence) and gstack-skill-end (the telemetry epilogue) — the
|
||
// scripts write it into the JSONL events.
|
||
expect(content, `${skill.dir} preamble fence must pass its own name`)
|
||
.toMatch(new RegExp(`--skill "${skill.name}" --model`));
|
||
expect(content, `${skill.dir} epilogue must pass its own name`)
|
||
.toContain(`gstack-skill-end --skill "${skill.name}"`);
|
||
}
|
||
});
|
||
|
||
test('qa and qa-only load the shared QA_METHODOLOGY through exploration', () => {
|
||
const qaSkeletonTmpl = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(qaSkeletonTmpl).toContain('{{SECTION:exploratory}}');
|
||
expect(qaSkeletonTmpl).not.toContain('{{QA_METHOD_READS}}');
|
||
expect(qaSkeletonTmpl).not.toContain('{{QA_METHODOLOGY}}');
|
||
const qaSectionTmpl = fs.readFileSync(path.join(ROOT, 'qa', 'sections', 'qa-patterns.md.tmpl'), 'utf-8');
|
||
expect(qaSectionTmpl).toContain('{{QA_METHODOLOGY}}');
|
||
|
||
const qaOnlyTmpl = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(qaOnlyTmpl).not.toContain('{{QA_METHODOLOGY}}');
|
||
expect(qaOnlyTmpl).toContain('{{SECTION:exploratory}}');
|
||
expect(qaOnlyTmpl).not.toContain('{{QA_METHOD_READS}}');
|
||
expect(qaOnlyTmpl).toContain('Load the shared preparation gate now');
|
||
for (const skill of ['qa', 'qa-only']) {
|
||
expect(fs.readFileSync(path.join(ROOT, skill, 'sections/exploratory.md.tmpl'), 'utf8'))
|
||
.toContain('{{QA_EXPLORATORY}}');
|
||
const entry = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf8');
|
||
const explorer = fs.readFileSync(path.join(ROOT, skill, 'sections/exploratory.md'), 'utf8');
|
||
const directoryRead = skill === 'qa'
|
||
? 'Read `sections/scope.md` relative to the installed `qa`/`gstack-qa` SKILL.md directory'
|
||
: "Use this host's installed `qa`/`gstack-qa` SKILL.md directory for these reads";
|
||
expect(entry).toContain('sections/exploratory.md');
|
||
expect(explorer).toContain(directoryRead);
|
||
for (const method of ['system-functional', 'qa-patterns']) {
|
||
const methodRead = `Read \`sections/${method}.md\` in full.`;
|
||
expect(entry).not.toContain(methodRead);
|
||
expect(explorer).toContain(methodRead);
|
||
expect(explorer.split(methodRead)).toHaveLength(2);
|
||
expect(explorer.indexOf(directoryRead)).toBeLessThan(explorer.indexOf(methodRead));
|
||
const directory = path.resolve(ROOT, skill, skill === 'qa-only' ? '../qa' : '.');
|
||
expect(fs.realpathSync(path.join(directory, 'sections', `${method}.md`)))
|
||
.toBe(path.join(ROOT, 'qa', 'sections', `${method}.md`));
|
||
}
|
||
expect(explorer).toContain('**Browser surfaces only:**\nRead `sections/qa-patterns.md` in full.');
|
||
expectMentions(explorer, [['before','complete','charters']], 'explorer');
|
||
expectMentions(explorer, [['do not','invocation','completed']], 'explorer');
|
||
expect(explorer.indexOf('Read `sections/qa-patterns.md`')).toBeLessThan(explorer.indexOf('Write a **charter**'));
|
||
}
|
||
});
|
||
|
||
test('QA_METHODOLOGY is expanded in the shared resource referenced by qa and qa-only', () => {
|
||
const qaContent = readSkillUnion('qa'); // carved: methodology lives in qa/sections/qa-patterns.md
|
||
const qaOnlyUnion = readSkillUnion('qa-only');
|
||
expect(qaOnlyUnion).toContain("Use this host's installed `qa`/`gstack-qa` SKILL.md directory for these reads");
|
||
expect(qaOnlyUnion).toContain('**Browser surfaces only:**\nRead `sections/qa-patterns.md` in full.');
|
||
expect(qaOnlyUnion).not.toContain('Health Score Rubric');
|
||
const qaOnlyContent = qaOnlyUnion + fs.readFileSync(path.join(ROOT, 'qa/sections/qa-patterns.md'), 'utf8');
|
||
|
||
// Both should contain the health score rubric
|
||
expect(qaContent).toContain('Health Score Rubric');
|
||
expect(qaOnlyContent).toContain('Health Score Rubric');
|
||
|
||
// Both should contain framework guidance
|
||
expect(qaContent).toContain('Framework-Specific Guidance');
|
||
expect(qaOnlyContent).toContain('Framework-Specific Guidance');
|
||
|
||
// Both should contain the important rules
|
||
expect(qaContent).toContain('Important Rules');
|
||
expect(qaOnlyContent).toContain('Important Rules');
|
||
|
||
// Both should contain the 6 phases
|
||
expect(qaContent).toContain('Phase 1');
|
||
expect(qaOnlyContent).toContain('Phase 1');
|
||
expect(qaContent).toContain('Phase 6');
|
||
expect(qaOnlyContent).toContain('Phase 6');
|
||
});
|
||
|
||
test('qa-only has no-fix guardrails', () => {
|
||
const qaOnlyContent = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md'), 'utf-8');
|
||
expect(extractMarkdownSection(qaOnlyContent, '## Additional Rules (qa-only specific)')).toMatch(/never fix/i);
|
||
// Should not have Edit, Glob, or Grep in allowed-tools.
|
||
// Scope to frontmatter (between the first two --- lines) — the body can
|
||
// legitimately mention these tool names in prose (e.g., Claude model
|
||
// overlay names the host's dedicated file and search tools).
|
||
const fmMatch = qaOnlyContent.match(/^---\n([\s\S]*?)\n---/);
|
||
expect(fmMatch).not.toBeNull();
|
||
const frontmatter = fmMatch![1];
|
||
expect(frontmatter).toMatch(/allowed-tools:/);
|
||
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Edit/);
|
||
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Glob/);
|
||
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Grep/);
|
||
});
|
||
|
||
test('qa has fix-loop tools and phases', () => {
|
||
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
|
||
// Should have Edit, Glob, Grep in allowed-tools
|
||
expect(qaContent).toContain('Edit');
|
||
expect(qaContent).toContain('Glob');
|
||
expect(qaContent).toContain('Grep');
|
||
// Should have fix-loop phases
|
||
expect(qaContent).toContain('Phase 7');
|
||
expect(qaContent).toContain('Phase 8');
|
||
expect(qaContent).toContain('Fix Loop');
|
||
expect(qaContent).toContain('Triage');
|
||
expect(qaContent).toMatch(/every 5 fixes/i);
|
||
});
|
||
});
|
||
|
||
describe('BASE_BRANCH_DETECT resolver', () => {
|
||
// Find a generated SKILL.md that uses the placeholder (ship is guaranteed to)
|
||
const shipContent = readShipUnion();
|
||
|
||
test('resolver output contains PR base detection command', () => {
|
||
expect(shipContent).toContain('gh pr view --json baseRefName');
|
||
});
|
||
|
||
test('resolver output contains repo default branch detection command', () => {
|
||
expect(shipContent).toContain('gh repo view --json defaultBranchRef');
|
||
});
|
||
|
||
test('resolver output contains fallback to main', () => {
|
||
expect(shipContent).toMatch(/fall\s*back\s+to\s+`main`/i);
|
||
});
|
||
|
||
test('resolver output uses "the base branch" phrasing', () => {
|
||
expect(shipContent).toContain('the base branch');
|
||
});
|
||
|
||
test('resolver output contains GitLab CLI commands', () => {
|
||
expect(shipContent).toContain('glab');
|
||
});
|
||
|
||
test('resolver output contains git-native fallback', () => {
|
||
expect(shipContent).toContain('git symbolic-ref');
|
||
});
|
||
|
||
test('resolver output mentions GitLab platform', () => {
|
||
expect(shipContent).toMatch(/gitlab/i);
|
||
});
|
||
});
|
||
|
||
describe('GitLab support in generated skills', () => {
|
||
const retroContent = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
|
||
const shipSkillContent = readShipUnion();
|
||
|
||
test('retro contains GitLab MR number extraction', () => {
|
||
expect(retroContent).toContain('[#!]');
|
||
});
|
||
|
||
test('retro uses BASE_BRANCH_DETECT (contains glab)', () => {
|
||
expect(retroContent).toContain('glab');
|
||
});
|
||
|
||
test('ship contains glab mr create', () => {
|
||
expect(shipSkillContent).toContain('glab mr create');
|
||
});
|
||
|
||
test('ship checks .gitlab-ci.yml', () => {
|
||
expect(shipSkillContent).toContain('.gitlab-ci.yml');
|
||
});
|
||
});
|
||
|
||
/**
|
||
* Quality evals — catch description regressions.
|
||
*
|
||
* These test that generated output is *useful for an AI agent*,
|
||
* not just structurally valid. Each test targets a specific
|
||
* regression we actually shipped and caught in review.
|
||
*/
|
||
describe('description quality evals', () => {
|
||
// Regression: snapshot flags lost value hints (-d <N>, -s <sel>, -o <path>)
|
||
// Browse carve: the flag reference renders into browse/sections/command-list.md.
|
||
test('snapshot flags with values include value hints in output', () => {
|
||
const content = readSkillUnion('browse');
|
||
for (const flag of SNAPSHOT_FLAGS) {
|
||
if (flag.takesValue) {
|
||
expect(flag.valueHint).toBeDefined();
|
||
expect(content).toContain(`${flag.short} ${flag.valueHint}`);
|
||
}
|
||
}
|
||
});
|
||
|
||
// Regression: "is" lost the valid states enum
|
||
test('is command lists valid state values', () => {
|
||
const desc = COMMAND_DESCRIPTIONS['is'].description;
|
||
for (const state of ['visible', 'hidden', 'enabled', 'disabled', 'checked', 'editable', 'focused']) {
|
||
expect(desc).toContain(state);
|
||
}
|
||
});
|
||
|
||
// Regression: "press" lost common key examples
|
||
test('press command lists example keys', () => {
|
||
const desc = COMMAND_DESCRIPTIONS['press'].description;
|
||
expect(desc).toContain('Enter');
|
||
expect(desc).toContain('Tab');
|
||
expect(desc).toContain('Escape');
|
||
});
|
||
|
||
// Regression: "console" lost --errors filter note
|
||
test('console command describes --errors behavior', () => {
|
||
const desc = COMMAND_DESCRIPTIONS['console'].description;
|
||
expect(desc).toContain('--errors');
|
||
});
|
||
|
||
// Regression: snapshot -i lost "@e refs" context
|
||
test('snapshot -i mentions @e refs', () => {
|
||
const flag = SNAPSHOT_FLAGS.find(f => f.short === '-i')!;
|
||
expect(flag.description).toContain('@e');
|
||
});
|
||
|
||
// Regression: snapshot -C lost "@c refs" context
|
||
test('snapshot -C mentions @c refs', () => {
|
||
const flag = SNAPSHOT_FLAGS.find(f => f.short === '-C')!;
|
||
expect(flag.description).toContain('@c');
|
||
});
|
||
|
||
// Guard: every description must be at least 8 chars (catches empty or stub descriptions)
|
||
test('all command descriptions have meaningful length', () => {
|
||
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
||
expect(meta.description.length).toBeGreaterThanOrEqual(8);
|
||
}
|
||
});
|
||
|
||
// Guard: snapshot flag descriptions must be at least 10 chars
|
||
test('all snapshot flag descriptions have meaningful length', () => {
|
||
for (const flag of SNAPSHOT_FLAGS) {
|
||
expect(flag.description.length).toBeGreaterThanOrEqual(10);
|
||
}
|
||
});
|
||
|
||
// Guard: descriptions must not contain pipe (breaks markdown table cells)
|
||
// Usage strings are backtick-wrapped in the table so pipes there are safe.
|
||
test('no command description contains pipe character', () => {
|
||
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
||
expect(meta.description).not.toContain('|');
|
||
}
|
||
});
|
||
|
||
// Guard: generated output uses → not ->
|
||
test('generated SKILL.md uses unicode arrows', () => {
|
||
// P2 (v1.2.0): the browse body moved out of the top-level router into
|
||
// browse/SKILL.md. Guard arrow style on the browse body (sliced from its
|
||
// H1 so the auto-generated `-->` header comments are excluded).
|
||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||
const h1 = content.search(/^# browse: /m);
|
||
expect(h1).toBeGreaterThan(-1);
|
||
const body = content.slice(h1);
|
||
expect(body).toContain('→');
|
||
expect(body).not.toContain('->');
|
||
});
|
||
});
|
||
|
||
describe('REVIEW_DASHBOARD resolver', () => {
|
||
const REVIEW_SKILLS = ['plan-ceo-review', 'plan-eng-review', 'plan-design-review'];
|
||
|
||
for (const skill of REVIEW_SKILLS) {
|
||
test(`review dashboard appears in ${skill} generated file`, () => {
|
||
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
||
expect(content).toContain('gstack-review');
|
||
expect(content).toContain('REVIEW READINESS DASHBOARD');
|
||
});
|
||
}
|
||
|
||
test('review dashboard appears in ship generated file', () => {
|
||
const content = readShipUnion();
|
||
// The dashboard reads the review log through its reader; the literal
|
||
// reviews.jsonl filename used to come only from the Context Recovery fence.
|
||
expect(extractMarkdownSection(content, '## Review Readiness Dashboard')).toContain('bin/gstack-review-read');
|
||
expect(content).toContain('REVIEW READINESS DASHBOARD');
|
||
});
|
||
|
||
test('dashboard treats review as a valid Eng Review source', () => {
|
||
const content = readShipUnion();
|
||
expect(content).toContain('| Eng Review | `review` or `plan-eng-review` | (DIFF) or (PLAN) |');
|
||
expect(content).toContain('**Content-first rule:** For `review`');
|
||
expect(content).toContain('**Plan records** (plan-ceo-review, plan-eng-review');
|
||
expect(content.replace(/\s+/g, ' ')).toContain('CLEARED requires the selected Eng Review to be `clean`, within 7 days and fresh under step 2');
|
||
});
|
||
|
||
test('shared dashboard propagates review source to plan-eng-review', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
expect(content).toContain('| Eng Review | `review` or `plan-eng-review` | (DIFF) or (PLAN) |');
|
||
expect(content).toContain('**Content-first rule:** For `review`');
|
||
});
|
||
|
||
test('resolver output contains key dashboard elements', () => {
|
||
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
||
expect(content).toContain('VERDICT');
|
||
expect(content).toContain('CLEARED');
|
||
expect(content).toContain('Eng Review');
|
||
expect(content).toContain('7 days');
|
||
expect(content).toContain('Design Review');
|
||
expect(content).toContain('skip_eng_review');
|
||
});
|
||
|
||
test('dashboard bash block includes git HEAD for staleness detection', () => {
|
||
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
||
expect(content).toContain('git rev-parse --short HEAD');
|
||
expect(content).toContain('---HEAD---');
|
||
});
|
||
|
||
test('dashboard includes staleness detection prose', () => {
|
||
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
||
expect(content).toContain('**2. Check freshness before choosing a verdict.**');
|
||
expect(content).toContain('commit');
|
||
});
|
||
|
||
for (const skill of REVIEW_SKILLS) {
|
||
test(`${skill} contains review chaining section`, () => {
|
||
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
||
expect(content).toContain('Review Chaining');
|
||
});
|
||
|
||
test(`${skill} Review Log includes commit field`, () => {
|
||
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
||
expect(content).toContain('"commit"');
|
||
});
|
||
}
|
||
|
||
test('plan-ceo-review chaining mentions eng and design reviews', () => {
|
||
// Carved skill: the chaining prose lives in sections/*.md. (It used to
|
||
// pass against the skeleton only because the preamble's routing-injection
|
||
// rules incidentally named these skills — that prose moved into
|
||
// bin/gstack-skill-start in token-reduction Phase 2.)
|
||
const content = readSkillUnion('plan-ceo-review');
|
||
expect(content).toContain('/plan-eng-review');
|
||
expect(content).toContain('/plan-design-review');
|
||
});
|
||
|
||
test('plan-eng-review chaining mentions design and ceo reviews', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
expect(content).toContain('/plan-design-review');
|
||
expect(content).toContain('/plan-ceo-review');
|
||
});
|
||
|
||
test('plan-design-review chaining mentions eng, ceo, and design skills', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('/plan-eng-review');
|
||
expect(content).toContain('/plan-ceo-review');
|
||
expect(content).toContain('/design-shotgun');
|
||
expect(content).toContain('/design-html');
|
||
});
|
||
|
||
test('ship does NOT contain review chaining', () => {
|
||
const content = readShipUnion();
|
||
expect(content).not.toContain('Review Chaining');
|
||
});
|
||
});
|
||
|
||
// ─── Test Coverage Audit Resolver Tests ─────────────────────
|
||
|
||
describe('TEST_COVERAGE_AUDIT placeholders', () => {
|
||
const planSkill = readSkillUnion('plan-eng-review'); // carved
|
||
const shipSkill = readShipUnion();
|
||
const reviewSkill = readSkillUnion('review'); // carved: Review Army moved to sections/review-army.md
|
||
|
||
test('plan and ship modes share codepath tracing methodology', () => {
|
||
// Review mode delegates test coverage to the Testing specialist subagent (Review Army)
|
||
const normalizedPlanSkill = planSkill.replace(/\s+/g, ' ');
|
||
const normalizedShipSkill = shipSkill.replace(/\s+/g, ' ');
|
||
for (const skill of [normalizedPlanSkill, normalizedShipSkill]) {
|
||
ordered(skill, ['concrete source and test files', '**Trace data flow.**', '**Diagram the execution.**', 'Quality scoring rubric']);
|
||
for (const marker of ['★★★', '★★', 'GAP']) expect(skill).toContain(marker);
|
||
}
|
||
// Plan mode traces the plan, not a git diff
|
||
expect(planSkill).toContain('Trace every codepath in the plan');
|
||
expect(planSkill).not.toContain('git diff origin');
|
||
// Ship mode traces the diff
|
||
expect(shipSkill).toContain('Trace every codepath changed');
|
||
});
|
||
|
||
test('review mode uses Review Army for specialist dispatch', () => {
|
||
expect(reviewSkill).toContain('Review Army');
|
||
expect(reviewSkill).toContain('Specialist Dispatch');
|
||
expect(reviewSkill).toContain('testing.md');
|
||
});
|
||
|
||
test('plan and ship modes include E2E decision matrix', () => {
|
||
// Review mode delegates to Testing specialist
|
||
for (const skill of [planSkill, shipSkill]) {
|
||
expect(skill).toContain('E2E Test Decision Matrix');
|
||
expect(skill).toContain('→E2E');
|
||
expect(skill).toContain('→EVAL');
|
||
}
|
||
});
|
||
|
||
test('plan and ship modes include regression rule', () => {
|
||
// Review mode delegates to Testing specialist
|
||
for (const skill of [planSkill, shipSkill]) {
|
||
expect(skill).toMatch(/^#{3,4} REGRESSION RULE/m);
|
||
}
|
||
expectMentions(planSkill, [['without','requirement','regression']], 'planSkill');
|
||
});
|
||
|
||
test('all five plan audit subheadings stay inside Test review while ship keeps its existing depth', () => {
|
||
const tests = planSkill.split('\n### 3. Test review\n')[1]!.split('\n### 4. Performance review\n')[0]!;
|
||
const instructions = tests.replace(/```[\s\S]*?```/g, '');
|
||
const headings = ['Test Framework Detection', 'E2E Test Decision Matrix', 'REGRESSION RULE (mandatory)',
|
||
'LLM/eval scope', 'Test Plan Artifact'];
|
||
expect([...instructions.matchAll(/^#### (.+)$/gm)].map(match => match[1])).toEqual(headings);
|
||
expect(instructions).not.toMatch(/^#{1,3} /m);
|
||
for (const heading of headings.filter(heading => heading !== 'LLM/eval scope')) {
|
||
expect(shipSkill).toContain(`\n### ${heading}\n`);
|
||
expect(shipSkill).not.toContain(`\n#### ${heading}\n`);
|
||
}
|
||
});
|
||
|
||
test('planned regression coverage gets its own approved contract without making coverage optional', () => {
|
||
const regression = planSkill.split('\n#### REGRESSION RULE (mandatory)\n')[1]!.split('**Step 4.')[0]!;
|
||
expect(regression).toContain('critical requirement');
|
||
expectMentions(regression, [['not','whether','cover']], 'regression');
|
||
expectMentions(regression, [['before','approved','contract']], 'regression');
|
||
expect(regression).not.toContain('No AskUserQuestion');
|
||
expect(regression).not.toContain('the diff broke');
|
||
expectMentions(regression, [['not','running','already']], 'regression');
|
||
});
|
||
|
||
test('plan gap additions wait for their test-contract decision', () => {
|
||
const action = planSkill.split('**Step 5. Add missing tests to the plan:**')[1]!.split('\n#### Test Plan Artifact\n')[0]!;
|
||
expectMentions(action, [['until','contracts','optional']], 'action');
|
||
expectMentions(action, [['wait','applying','answer']], 'action');
|
||
expect(action).not.toContain('explain what broke');
|
||
});
|
||
|
||
test('implemented regressions still require an immediate test without a question', () => {
|
||
const regression = shipSkill.split('### REGRESSION RULE (mandatory)')[1]!.split('**4.')[0]!;
|
||
expectMentions(regression, [['no','askuserquestion']], 'regression');
|
||
});
|
||
|
||
test('plan and ship modes include test framework detection', () => {
|
||
// Review mode delegates to Testing specialist
|
||
for (const skill of [planSkill, shipSkill]) {
|
||
expect(skill).toContain('Test Framework Detection');
|
||
expect(skill).toContain('CLAUDE.md');
|
||
}
|
||
});
|
||
|
||
test('plan framework absence keeps requirements without offering test generation', () => {
|
||
const detection = planSkill.split('\n#### Test Framework Detection\n')[1]!.split('**Step 1.')[0]!;
|
||
expect(detection).toContain('State that the framework is unknown');
|
||
expectMentions(detection, [['no','selection','framework']], 'detection');
|
||
expectMentions(detection, [['do not','framework','proposed']], 'detection');
|
||
expect(detection).not.toContain('skip test generation');
|
||
});
|
||
|
||
test('plan mode adds tests to plan + includes test plan artifact', () => {
|
||
expect(planSkill).toContain('Add missing tests to the plan');
|
||
expect(planSkill).toContain('eng-review-test-plan');
|
||
expect(planSkill).toContain('Test Plan Artifact');
|
||
const artifact = planSkill.split('\n#### Test Plan Artifact\n')[1]!;
|
||
// The artifact is written even when Test review choices are still unanswered (periodic run 37151477069).
|
||
const step5 = planSkill.split('**Step 5. Add missing tests to the plan:**')[1]!.split('\n#### Test Plan Artifact\n')[0]!;
|
||
expectMentions(artifact, [['not','implementation','unresolved']], 'artifact');
|
||
expect(artifact).toContain('TEST_PLAN_USER=$(whoami)');
|
||
expect(artifact).toContain('sanitized `BRANCH` from gstack-slug');
|
||
expect(artifact).toContain('without an origin, write `local-only`');
|
||
});
|
||
|
||
test('ship mode auto-generates tests + includes before/after count', () => {
|
||
expect(shipSkill).toContain('Generate tests for uncovered paths');
|
||
expect(shipSkill).toContain('Before/after test count');
|
||
expect(shipSkill).toContain('30 code paths max');
|
||
expect(shipSkill).toContain('ship-test-plan');
|
||
});
|
||
|
||
test('review mode uses Fix-First + Review Army for specialist coverage', () => {
|
||
expect(reviewSkill).toContain('Fix-First');
|
||
expect(reviewSkill).toContain('INFORMATIONAL');
|
||
// Review Army handles test coverage via Testing specialist subagent
|
||
expect(reviewSkill).toContain('Review Army');
|
||
expect(reviewSkill).toContain('Testing');
|
||
});
|
||
|
||
test('plan mode does NOT include ship-specific content', () => {
|
||
expect(planSkill).not.toContain('Before/after test count');
|
||
expect(planSkill).not.toContain('30 code paths max');
|
||
expect(planSkill).not.toContain('ship-test-plan');
|
||
});
|
||
|
||
test('review mode does NOT include test plan artifact', () => {
|
||
expect(reviewSkill).not.toContain('Test Plan Artifact');
|
||
expect(reviewSkill).not.toContain('eng-review-test-plan');
|
||
expect(reviewSkill).not.toContain('ship-test-plan');
|
||
});
|
||
|
||
test('review/specialists/ directory has all expected checklist files', () => {
|
||
const specDir = path.join(ROOT, 'review', 'specialists');
|
||
const expected = [
|
||
'testing.md',
|
||
'maintainability.md',
|
||
'security.md',
|
||
'performance.md',
|
||
'data-migration.md',
|
||
'api-contract.md',
|
||
'simplification.md',
|
||
'red-team.md',
|
||
];
|
||
for (const f of expected) {
|
||
expect(fs.existsSync(path.join(specDir, f))).toBe(true);
|
||
}
|
||
});
|
||
|
||
// Regression pins for the simplification specialist (advisory carve-out edits
|
||
// the pre-existing quality_score instruction, so the rendered contract is
|
||
// pinned statically — the carve-out and the early-out line must both survive
|
||
// regeneration verbatim).
|
||
test('simplification advisory carve-out and early-out render into review docs', () => {
|
||
const reviewArmySection = fs.readFileSync(
|
||
path.join(ROOT, 'review', 'sections', 'review-army.md'),
|
||
'utf-8',
|
||
);
|
||
expect(reviewArmySection).toContain('"advisory": true');
|
||
expectTokens(reviewArmySection, ['`quality_score`'], 'reviewArmySection');
|
||
expectMentions(reviewArmySection, [['do not', 'specialist', 'findings']], 'reviewArmySection');
|
||
expect(reviewArmySection).toContain('Use the merged NON-advisory specialist findings for both counts and score');
|
||
expect(reviewArmySection).toContain('net: -N lines possible');
|
||
expect(reviewArmySection).toContain('--simplification');
|
||
// The specialist itself must never carry a verdict-shaped zero-findings line.
|
||
const spec = fs.readFileSync(
|
||
path.join(ROOT, 'review', 'specialists', 'simplification.md'),
|
||
'utf-8',
|
||
);
|
||
expect(spec).toContain('NO FINDINGS');
|
||
expect(spec).not.toContain('Lean already. Ship.');
|
||
// Closed tag vocabulary: the disavowed yagni: frame must not appear.
|
||
expect(spec).toContain('speculative');
|
||
expect(spec.toLowerCase()).not.toContain('"yagni"');
|
||
});
|
||
|
||
test('each specialist file has standard header with scope and output format', () => {
|
||
const specDir = path.join(ROOT, 'review', 'specialists');
|
||
const files = fs.readdirSync(specDir).filter(f => f.endsWith('.md'));
|
||
for (const f of files) {
|
||
const content = fs.readFileSync(path.join(specDir, f), 'utf-8');
|
||
// All specialist files must have Scope and Output/JSON in header
|
||
expect(content).toContain('Scope:');
|
||
expect(content.toLowerCase()).toMatch(/output|json/);
|
||
// Must define NO FINDINGS behavior
|
||
expect(content).toContain('NO FINDINGS');
|
||
}
|
||
});
|
||
|
||
// Regression guard: ship output contains key phrases from before the refactor
|
||
test('ship SKILL.md regression guard — key phrases preserved', () => {
|
||
const regressionPhrases = [
|
||
'Coverage goal: every changed behavior is protected by a test that would catch a real regression.',
|
||
'ASCII coverage diagram',
|
||
'processPayment',
|
||
'refundPayment',
|
||
'billing.test.ts',
|
||
'checkout.e2e.ts',
|
||
'COVERAGE:',
|
||
'QUALITY:',
|
||
'GAPS:',
|
||
'Code paths:',
|
||
'User flows:',
|
||
];
|
||
for (const phrase of regressionPhrases) {
|
||
expect(shipSkill).toContain(phrase);
|
||
}
|
||
});
|
||
|
||
test('ship SKILL.md contains review army specialist dispatch', () => {
|
||
expect(shipSkill).toContain('Specialist Dispatch');
|
||
expect(shipSkill).toContain('Step 9.1');
|
||
expect(shipSkill).toContain('Step 9.2');
|
||
});
|
||
|
||
test('ship SKILL.md contains cross-review finding dedup', () => {
|
||
expect(shipSkill).toContain('Cross-review finding dedup');
|
||
expect(shipSkill).toContain('Step 9.3');
|
||
});
|
||
|
||
test('ship SKILL.md contains re-run idempotency behavior', () => {
|
||
expect(shipSkill).toContain('**Route:**');
|
||
expect(shipSkill.replace(/\s+/g, ' ')).toContain('integrate (1–3) → test and review (4–11.5) → prepare the release (12–15) → verify frozen content (16) → push and publish (17–21)');
|
||
expectMentions(shipSkill.replace(/\s+/g, ' '), [['never','verification','duplicate']], 'shipSkill.replace(/\s+/g, )');
|
||
});
|
||
});
|
||
|
||
// --- {{TEST_FAILURE_TRIAGE}} resolver tests ---
|
||
|
||
describe('TEST_FAILURE_TRIAGE resolver', () => {
|
||
const shipSkill = readShipUnion();
|
||
|
||
test('contains all 4 triage steps', () => {
|
||
expect(shipSkill).toContain('Step T1: Classify each failure');
|
||
expect(shipSkill).toContain('Step T2: Handle in-branch failures');
|
||
expect(shipSkill).toContain('Step T3: Handle pre-existing failures');
|
||
expect(shipSkill).toContain('Step T4: Execute the chosen action');
|
||
});
|
||
|
||
test('T1 includes classification criteria (in-branch vs pre-existing)', () => {
|
||
expect(shipSkill).toContain('In-branch');
|
||
expect(shipSkill).toContain('Likely pre-existing');
|
||
expect(shipSkill).toContain('git diff origin/');
|
||
});
|
||
|
||
test('T3 branches on REPO_MODE (solo vs collaborative)', () => {
|
||
expect(shipSkill).toContain('REPO_MODE');
|
||
expect(shipSkill).toContain('solo');
|
||
expect(shipSkill).toContain('collaborative');
|
||
});
|
||
|
||
test('solo mode offers fix-now, TODO, and skip options', () => {
|
||
expect(shipSkill).toContain('Investigate and fix now');
|
||
expect(shipSkill).toContain('Add as P0 TODO');
|
||
expect(shipSkill).toContain('Skip');
|
||
});
|
||
|
||
test('collaborative mode offers blame + assign option', () => {
|
||
expect(shipSkill).toContain('Blame + assign GitHub issue');
|
||
expect(shipSkill).toContain('gh issue create');
|
||
});
|
||
|
||
test('defaults ambiguous failures to in-branch (safety)', () => {
|
||
expect(shipSkill).toContain('When ambiguous, default to in-branch');
|
||
});
|
||
});
|
||
|
||
// --- {{PLAN_FILE_REVIEW_REPORT}} resolver tests ---
|
||
|
||
describe('PLAN_FILE_REVIEW_REPORT resolver', () => {
|
||
const REVIEW_SKILLS = ['plan-ceo-review', 'plan-eng-review', 'plan-design-review', 'codex'];
|
||
|
||
for (const [label, skill] of [['Eng', 'plan-eng-review'], ['CEO', 'plan-ceo-review']]) {
|
||
test(`${label} report example is fenced Markdown, and the original escaped fence is rejected`, async () => {
|
||
const { marked } = await import('marked');
|
||
const content = readSkillUnion(skill!);
|
||
const tokens = marked.lexer(content);
|
||
const headings = tokens.filter(token => token.type === 'heading' && token.depth === 2);
|
||
expect(headings.map(token => token.text)).not.toContain('GSTACK REVIEW REPORT');
|
||
const example = tokens.find(token => token.type === 'code'
|
||
&& token.lang === 'markdown' && token.text.startsWith('## GSTACK REVIEW REPORT\n'));
|
||
expect(example).toBeDefined();
|
||
if (example?.type !== 'code') throw new Error('Missing fenced report example');
|
||
expect(example.text).toContain('| Review | Trigger | Why | Runs | Status | Findings |');
|
||
expect(example.text).toContain('`/' + skill + '`');
|
||
// Reproduce the observed literal-backslash delimiters without changing
|
||
// the parser or preprocessing malformed Markdown into valid fences.
|
||
const broken = content.replace(example.raw, example.raw.replaceAll('`', '\\`'));
|
||
expect(broken).not.toBe(content);
|
||
const brokenTokens = marked.lexer(broken);
|
||
expect(brokenTokens.some(token => token.type === 'code' && token.lang === 'markdown'
|
||
&& token.text.startsWith('## GSTACK REVIEW REPORT\n'))).toBe(false);
|
||
expect(brokenTokens.filter(token => token.type === 'heading' && token.depth === 2)
|
||
.map(token => token.text)).toContain('GSTACK REVIEW REPORT');
|
||
});
|
||
}
|
||
|
||
test('Eng renderers use real code delimiters without changing confidence rules or report fields', async () => {
|
||
const {generateConfidenceCalibration} = await import('../scripts/resolvers/confidence');
|
||
const {generateReviewDashboard, generatePlanFileReviewReport} = await import('../scripts/resolvers/review-dashboard');
|
||
const {generateCodexPlanReview} = await import('../scripts/resolvers/outside-voice-steps');
|
||
const {HOST_PATHS} = await import('../scripts/resolvers/types');
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const ctx = {skillName: 'plan-eng-review', tmplPath: 'plan-eng-review/SKILL.md.tmpl',
|
||
host: host.name, paths: HOST_PATHS[host.name]!};
|
||
const confidence = generateConfidenceCalibration(ctx);
|
||
const dashboard = generateReviewDashboard(ctx);
|
||
const report = generatePlanFileReviewReport(ctx);
|
||
const outside = generateCodexPlanReview(ctx);
|
||
// These four resolver fragments contain no intentional escaped-backtick
|
||
// example; the preamble does, and is deliberately outside this check.
|
||
for (const output of [confidence, dashboard, report, outside]) expect(output).not.toContain('\\`');
|
||
expect(confidence).toBe(generateConfidenceCalibration({...ctx, skillName: 'plan-ceo-review'}).replaceAll('\\`', '`'));
|
||
const ceoDashboard = generateReviewDashboard({...ctx, skillName: 'plan-ceo-review'}).replaceAll('\\`', '`');
|
||
const ceoVoiceSource = 'Below the dashboard, group `autoplan-voices` and `design-outside-voices` by workflow run and phase';
|
||
expect(dashboard.replace(/\s+/g, ' ')).toContain(ceoVoiceSource);
|
||
expect(ceoDashboard.replace(/\s+/g, ' ')).toContain(ceoVoiceSource);
|
||
expect(ceoDashboard).toBe(dashboard);
|
||
for (const field of ['status', 'unresolved', 'critical_gaps', 'issues_found', 'mode', 'commit']) {
|
||
expect(report).toContain('`' + field + '`');
|
||
}
|
||
expect(outside).toContain(host.name === 'codex' ? 'claude auth login' : 'codex login');
|
||
expectMentions(outside, [['never','supplies','coverage']], 'outside');
|
||
}
|
||
});
|
||
|
||
for (const skill of REVIEW_SKILLS) {
|
||
test(`plan file review report appears in ${skill} generated file`, () => {
|
||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('GSTACK REVIEW REPORT');
|
||
});
|
||
}
|
||
|
||
test('resolver output contains key report elements', () => {
|
||
const content = readSkillUnion('plan-ceo-review'); // carved: report writer moved to section
|
||
expect(content).toContain('Trigger');
|
||
expect(content).toContain('Findings');
|
||
expect(content).toContain('VERDICT');
|
||
expect(content).toContain('/plan-ceo-review');
|
||
expect(content).toContain('/plan-eng-review');
|
||
expect(content).toContain('/plan-design-review');
|
||
expect(content).toContain('/codex review');
|
||
});
|
||
|
||
test('Eng task output follows the same write policy as its report', () => {
|
||
const content = readSkillUnion('plan-eng-review');
|
||
// The task example itself has an H2; bound by the following real section.
|
||
const tasks = content.slice(content.indexOf('## Implementation Tasks'), content.indexOf('## Plan File Review Report'));
|
||
expectMentions(tasks, [['only','permits','review']], 'tasks');
|
||
expectMentions(tasks, [['not','persisted','complete']], 'tasks');
|
||
expect(tasks).not.toContain('always write');
|
||
});
|
||
});
|
||
|
||
// --- {{PLAN_COMPLETION_AUDIT}} resolver tests ---
|
||
|
||
describe('PLAN_COMPLETION_AUDIT placeholders', () => {
|
||
const shipSkill = readShipUnion();
|
||
const reviewSkill = readSkillUnion('review'); // carved: plan-completion audit moved to sections/plan-completion.md
|
||
|
||
test('ship SKILL.md contains plan completion audit step', () => {
|
||
expect(shipSkill).toContain('Plan Completion Audit');
|
||
expect(shipSkill).toContain('Step 8');
|
||
});
|
||
|
||
test('review SKILL.md contains plan completion in scope drift', () => {
|
||
expect(reviewSkill).toContain('Plan File Discovery');
|
||
expect(reviewSkill).toContain('Actionable Item Extraction');
|
||
expect(reviewSkill).toContain('Integration with Scope Drift Detection');
|
||
});
|
||
|
||
test('both modes share plan file discovery methodology', () => {
|
||
expect(shipSkill).toContain('Plan File Discovery');
|
||
expect(reviewSkill).toContain('Plan File Discovery');
|
||
// Both should have conversation context first
|
||
expect(shipSkill).toContain('Conversation context (primary)');
|
||
expect(reviewSkill).toContain('Conversation context (primary)');
|
||
// Both should have grep fallback
|
||
expect(shipSkill).toContain('Content-based search (fallback)');
|
||
expect(reviewSkill).toContain('Content-based search (fallback)');
|
||
});
|
||
|
||
test('ship mode has gate logic for NOT DONE items', () => {
|
||
expect(shipSkill).toContain('NOT DONE');
|
||
expect(shipSkill).toContain('Stop — implement the missing items');
|
||
expect(shipSkill).toContain('Ship anyway — defer');
|
||
expect(shipSkill).toContain('intentionally dropped');
|
||
});
|
||
|
||
test('review mode is INFORMATIONAL only', () => {
|
||
expect(reviewSkill).toContain('INFORMATIONAL');
|
||
expect(reviewSkill).toContain('MISSING REQUIREMENTS');
|
||
expect(reviewSkill).toContain('SCOPE CREEP');
|
||
});
|
||
|
||
test('item extraction has 50-item cap', () => {
|
||
expect(shipSkill).toContain('at most 50 items');
|
||
});
|
||
|
||
test('uses file-level traceability (not commit-level)', () => {
|
||
expect(shipSkill).toContain('Cite the specific file');
|
||
expect(shipSkill).not.toContain('commit-level traceability');
|
||
});
|
||
});
|
||
|
||
// --- {{PLAN_VERIFICATION_EXEC}} resolver tests ---
|
||
|
||
describe('PLAN_VERIFICATION_EXEC placeholder', () => {
|
||
const shipSkill = readShipUnion();
|
||
|
||
test('ship SKILL.md contains plan verification step', () => {
|
||
expect(shipSkill).toContain('Step 8.1');
|
||
expect(shipSkill).toContain('Plan Verification');
|
||
});
|
||
|
||
test('references the shared explorer without invoking an entire QA workflow', () => {
|
||
const resource = "From the installed /ship SKILL.md's directory, Read `../qa/sections/exploratory.md` in full";
|
||
expect(shipSkill).toContain(resource);
|
||
const load = shipSkill.indexOf(resource);
|
||
const preflight = shipSkill.indexOf('Run the shared preflight;');
|
||
const probes = shipSkill.indexOf('**3. Run smoke and plan checks.**');
|
||
expect(load).toBeGreaterThan(-1);
|
||
expect(preflight).toBeGreaterThan(load);
|
||
expect(probes).toBeGreaterThan(preflight);
|
||
const shared = fs.readFileSync(path.join(ROOT, 'qa/sections/exploratory.md'), 'utf8');
|
||
const selection = shared.indexOf('in full and select the surfaces');
|
||
const methods = shared.indexOf('Read `sections/system-functional.md`');
|
||
expect(selection).toBeGreaterThan(shared.indexOf('Read `sections/scope.md`'));
|
||
expect(methods).toBeGreaterThan(selection);
|
||
expect(shared.indexOf('Write a **charter**')).toBeGreaterThan(methods);
|
||
expect(shipSkill.slice(preflight, probes)).toContain("For browsers, Read QA's `sections/browser-setup.md`");
|
||
expectMentions(shipSkill, [['do not','invoke','entire']], 'shipSkill');
|
||
});
|
||
|
||
test('keeps declared browser URLs separate from native functional probes', () => {
|
||
expectMentions(shipSkill, [['without','discovering','functional']], 'shipSkill');
|
||
expectMentions(shipSkill.replace(/\s+/g, ' '), [['not', 'automatically', 'page']], 'shipSkill.replace(/\s+/g, )');
|
||
});
|
||
|
||
test('retains automatic exploration when there is no plan or verification section', () => {
|
||
expectMentions(shipSkill, [['no','verification','section']], 'shipSkill');
|
||
expect(shipSkill).toContain('Automatic diff-scoped QA still runs');
|
||
});
|
||
|
||
test('blocks unavailable required checks instead of silently skipping them', () => {
|
||
const flat = shipSkill.replace(/\s+/g, ' ');
|
||
expect(flat).toContain('Noninteractive runs return blocked');
|
||
expectMentions(flat, [['never','required-probe','silently']], 'flat');
|
||
expect(flat).toContain('Missing/unreadable assets block required QA');
|
||
expect(flat).toContain('Keep actual outcomes and incomplete flags; VERIFY_RESULT stays fail');
|
||
});
|
||
});
|
||
|
||
// --- Coverage gate tests ---
|
||
|
||
describe('Coverage gate in ship', () => {
|
||
const shipSkill = readShipUnion();
|
||
const reviewSkill = readSkillUnion('review'); // carved: testing.md specialist ref lives in sections/review-army.md
|
||
|
||
test('ship SKILL.md contains coverage gate with thresholds', () => {
|
||
expect(shipSkill).toContain('Coverage gate');
|
||
expect(shipSkill).toContain('>= target');
|
||
expect(shipSkill).toContain('< minimum');
|
||
});
|
||
|
||
test('ship SKILL.md supports configurable thresholds via CLAUDE.md', () => {
|
||
expect(shipSkill).toContain('## Test Coverage');
|
||
expect(shipSkill).toContain('Minimum:');
|
||
expect(shipSkill).toContain('Target:');
|
||
});
|
||
|
||
test('coverage gate skips on parse failure (not block)', () => {
|
||
expect(shipSkill).toContain('could not determine percentage — skipping');
|
||
});
|
||
|
||
test('review SKILL.md delegates coverage to Testing specialist', () => {
|
||
// Coverage audit moved to Testing specialist subagent in Review Army
|
||
expect(reviewSkill).toContain('testing.md');
|
||
expect(reviewSkill).toContain('INFORMATIONAL');
|
||
});
|
||
});
|
||
|
||
// --- Ship metrics logging ---
|
||
|
||
describe('Ship metrics logging', () => {
|
||
const shipSkill = readShipUnion();
|
||
|
||
test('ship SKILL.md contains metrics persistence step', () => {
|
||
expect(shipSkill).toContain('Step 20');
|
||
expect(shipSkill).toContain('coverage_pct');
|
||
expect(shipSkill).toContain('plan_items_total');
|
||
expect(shipSkill).toContain('plan_items_done');
|
||
expect(shipSkill).toContain('verification_result');
|
||
});
|
||
});
|
||
|
||
// --- Plan file discovery shared helper ---
|
||
|
||
describe('Plan file discovery shared helper', () => {
|
||
// The shared helper should appear in ship (via PLAN_COMPLETION_AUDIT_SHIP)
|
||
// and in review (via PLAN_COMPLETION_AUDIT_REVIEW)
|
||
const shipSkill = readShipUnion();
|
||
const reviewSkill = readSkillUnion('review'); // carved: plan-completion audit moved to sections/plan-completion.md
|
||
|
||
test('plan file discovery appears in both ship and review', () => {
|
||
expect(shipSkill).toContain('Plan File Discovery');
|
||
expect(reviewSkill).toContain('Plan File Discovery');
|
||
});
|
||
|
||
test('both include conversation context first', () => {
|
||
expect(shipSkill).toContain('Conversation context (primary)');
|
||
expect(reviewSkill).toContain('Conversation context (primary)');
|
||
});
|
||
|
||
test('both include content-based fallback', () => {
|
||
expect(shipSkill).toContain('Content-based search (fallback)');
|
||
expect(reviewSkill).toContain('Content-based search (fallback)');
|
||
});
|
||
});
|
||
|
||
// --- Retro plan completion ---
|
||
|
||
describe('Retro plan completion section', () => {
|
||
// Carved: the narrative report format (incl. Plan Completion) lives in
|
||
// retro/sections/report-format.md — read the skeleton+sections union.
|
||
const retroSkill = readSkillUnion('retro');
|
||
|
||
test('retro SKILL.md contains plan completion section', () => {
|
||
expect(retroSkill).toContain('### Plan Completion');
|
||
expect(retroSkill).toContain('plan_items_total');
|
||
expect(retroSkill).toContain('Plan Completion This Period');
|
||
});
|
||
});
|
||
|
||
// --- Plan status footer in preamble ---
|
||
|
||
describe('Plan status footer in preamble', () => {
|
||
test('preamble contains plan status footer as neutral forward reference to EXIT PLAN MODE GATE', () => {
|
||
// Read any skill that uses PREAMBLE
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
expect(content).toContain('Plan Status Footer');
|
||
expect(content).toContain('GSTACK REVIEW REPORT');
|
||
expect(content).toContain('ExitPlanMode');
|
||
expect(content).toContain('EXIT PLAN MODE GATE');
|
||
// The preamble must NOT impose review-report rules on operational skills
|
||
// that have no review report. It's a forward reference, not enforcement.
|
||
expect(content).not.toContain('NO REVIEWS YET');
|
||
});
|
||
});
|
||
|
||
// --- make-pdf setup ordering ---
|
||
|
||
describe('make-pdf setup ordering', () => {
|
||
test('MAKE-PDF SETUP appears before generic preamble footer sections', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'make-pdf', 'SKILL.md'), 'utf-8');
|
||
const preambleIdx = content.indexOf('## Preamble (run first)');
|
||
const setupIdx = content.indexOf('## MAKE-PDF SETUP');
|
||
const planModeIdx = content.indexOf('## Plan Mode Safe Operations');
|
||
const telemetryIdx = content.indexOf('## Telemetry (run last)');
|
||
const workflowIdx = content.indexOf('# make-pdf: publication-quality PDFs from markdown');
|
||
|
||
expect(preambleIdx).toBeGreaterThanOrEqual(0);
|
||
expect(setupIdx).toBeGreaterThan(preambleIdx);
|
||
expect(setupIdx).toBeLessThan(planModeIdx);
|
||
expect(setupIdx).toBeLessThan(telemetryIdx);
|
||
expect(setupIdx).toBeLessThan(workflowIdx);
|
||
expect(content.match(/^## MAKE-PDF SETUP/gm)?.length ?? 0).toBe(1);
|
||
});
|
||
});
|
||
|
||
// --- Skill invocation during plan mode in preamble ---
|
||
|
||
describe('Skill invocation during plan mode in preamble', () => {
|
||
test('preamble contains skill invocation plan mode section', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
expect(content).toContain('Skill Invocation During Plan Mode');
|
||
expect(content).toContain('Do not continue the workflow');
|
||
expect(content).toContain('cancel the skill or leave plan mode');
|
||
});
|
||
|
||
// #2851: a skill cannot grant permissions or redefine the host's mode; the
|
||
// host's restrictions and the user's scope win, in every generated skill.
|
||
test('no generated skill claims precedence over host plan-mode restrictions (#2851)', () => {
|
||
let checked = 0;
|
||
for (const file of ['SKILL.md', ...fs.readdirSync(ROOT).map(d => path.join(d, 'SKILL.md'))]) {
|
||
if (!fs.existsSync(path.join(ROOT, file))) continue;
|
||
const content = fs.readFileSync(path.join(ROOT, file), 'utf-8');
|
||
if (!content.includes('## Skill Invocation During Plan Mode')) continue;
|
||
checked++;
|
||
expect(content, file).not.toContain('takes precedence over generic plan mode');
|
||
expect(content, file).not.toContain('overrides generic plan mode');
|
||
expect(content, file).not.toContain('allowed because they inform the plan');
|
||
expect(content, file).not.toContain('"PLAN MODE EXCEPTION — ALWAYS RUN" execute.');
|
||
expect(content, file).not.toContain('Execute "PLAN MODE EXCEPTION — ALWAYS RUN" commands');
|
||
expectMentions(content, [['only','restrictions','precedence']], 'content');
|
||
expect(content, file).toMatch(/"PLAN MODE EXCEPTION — ALWAYS RUN" (?:commands )?(?:run )?only where the host permits/);
|
||
}
|
||
expect(checked).toBeGreaterThan(40);
|
||
});
|
||
});
|
||
|
||
// --- {{SPEC_REVIEW_LOOP}} resolver tests ---
|
||
|
||
describe('SPEC_REVIEW_LOOP resolver', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
const { generateSpecReviewLoop } = require('../scripts/resolvers/spec-review');
|
||
const { HOST_PATHS } = require('../scripts/resolvers/types');
|
||
const render = (skillName: string, host = 'claude') => generateSpecReviewLoop({
|
||
skillName,
|
||
tmplPath: `${skillName}/SKILL.md.tmpl`,
|
||
host,
|
||
paths: HOST_PATHS[host],
|
||
});
|
||
|
||
test('office-hours prepares a complete reviewer prompt on every host', () => {
|
||
const { renderOfficeHoursReviewerPrompt } = require('../lib/office-hours-review');
|
||
for (const host of Object.keys(HOST_PATHS)) {
|
||
const output = render('office-hours', host);
|
||
expect(output).toContain('gstack-office-hours-review prepare --design');
|
||
expect(output).toContain('--out-dir "<review-directory>"');
|
||
ordered(output, ['mktemp -d "<design-path>.review.XXXXXX"', 'prepare --design', '`round-N.prompt.md`', '`dispatch`']);
|
||
}
|
||
const prompt = renderOfficeHoursReviewerPrompt({ document: '/tmp/design.md', verdictPath: '/tmp/round-1.json' });
|
||
expect(prompt).toContain('Document: /tmp/design.md');
|
||
expect(prompt).toContain('Verdict: /tmp/round-1.json');
|
||
expect(prompt).toContain('"document": "/tmp/design.md"');
|
||
expect(prompt).toMatch(/design and coaching document/i);
|
||
for (const section of ["'The Assignment'", "'What I noticed about how you think'"]) expect(prompt).toContain(section);
|
||
expect(prompt).toMatch(/never invent customer answers/i);
|
||
expect(prompt).toMatch(/including minor findings/i);
|
||
expect(prompt).toMatch(/identical JSON/i);
|
||
expect(prompt).toMatch(/classify every preceding finding/i);
|
||
ordered(prompt, ['1. **Completeness**', '2. **Consistency**', '3. **Clarity**', '4. **Scope**', '5. **Feasibility**']);
|
||
expect(prompt).toMatch(/distinguished from committed behavior/i);
|
||
expect(prompt).toContain('"version": 1');
|
||
expect(prompt).toContain('"prior": []');
|
||
});
|
||
|
||
test('office-hours checks stop conditions before edits and preserves concerns mechanically', () => {
|
||
const output = render('office-hours');
|
||
const step2 = output.slice(output.indexOf('**Step 2:'), output.indexOf('**Step 3:'));
|
||
expectMentions(step2, [['before','dispatching','findings']], 'step2');
|
||
ordered(step2, ['gstack-office-hours-review check', '- PASS:', '- CONVERGENCE:', '- MAX_ITERATIONS: round 3', '- CONTINUE:', 'gstack-office-hours-review finalize --design']);
|
||
expectMentions(step2, [['do not','re-dispatch','again']], 'step2');
|
||
expectMentions(step2, [['do not','generated','section']], 'step2');
|
||
expect(step2.slice(step2.indexOf('finalize --design'))).toMatch(/existing user approval/i);
|
||
});
|
||
|
||
test('office-hours finalizes the report after writing it and labels derivable metrics', () => {
|
||
const output = render('office-hours');
|
||
expect(output).toContain('--report "<report-path>"');
|
||
expect(output).toMatch(/after the report exists/i);
|
||
expectMentions(output, [['do not','summarize','replace']], 'output');
|
||
expectMentions(output, [['approval','assignment','preserve']], 'output');
|
||
expect(output).toMatch(/explicit later reviewer evidence/i);
|
||
});
|
||
|
||
test('office-hours errors stay explicit and preserve preceding valid evidence', () => {
|
||
const output = render('office-hours');
|
||
expect(output).toMatch(/--unreviewed "<actual failure cause>"`\s+and only the\s+preceding valid round files/);
|
||
expect(output).toMatch(/do not fabricate JSON/i);
|
||
expect(output).toMatch(/not an approval gate/i);
|
||
});
|
||
|
||
test('CEO keeps its loop limits, scoring, failure handling, and reporting', () => {
|
||
const ceo = render('plan-ceo-review');
|
||
const report = ceo.split('**Step 3:')[1]!.replace(/\s+/g, ' ');
|
||
expect(report).toMatch(/use JSON `null`/i);
|
||
expect(ceo).toMatch(/implementable without follow-up questions/i);
|
||
expect(ceo).not.toContain('design and coaching document');
|
||
expect(ceo).not.toContain('gstack-office-hours-review');
|
||
const dispatch = ceo.slice(ceo.indexOf('**Step 1:'), ceo.indexOf('**Step 2:')).replace(/\s+/g, ' ');
|
||
expect(dispatch).toContain('`run_in_background: false`');
|
||
expectMentions(dispatch, [['wait','tool']], 'dispatch');
|
||
expectMentions(dispatch, [['do not','waiting','advance']], 'dispatch');
|
||
const fixes = ceo.slice(ceo.indexOf('**Step 2:'), ceo.indexOf('**Step 3:')).replace(/\s+/g, ' ');
|
||
ordered(fixes, ['use 0D', 'amend the working plan', 're-dispatch']);
|
||
expectMentions(fixes, [['stop','review','after']], 'fixes');
|
||
expect(fixes).toMatch(/stop the loop/i);
|
||
expectMentions(fixes, [['not','successful','reviewer']], 'fixes');
|
||
expectMentions(fixes, [['does not','missing','require']], 'fixes');
|
||
expect(report).toContain('"## Reviewer Concerns"');
|
||
expect(report).toMatch(/SCORE is the latest attempt's reported 1–10 grade/i);
|
||
expect(report).toContain('"prior review score"');
|
||
expect(report).toMatch(/as not persisted/i);
|
||
expect(report).toContain('**0H spec-review metrics**');
|
||
expectMentions(report, [['stop','completion','required']], 'report');
|
||
expect(report).toContain('GSTACK_STATE_ROOT=$(~/.claude/skills/gstack/bin/gstack-paths --get GSTACK_STATE_ROOT); : "${GSTACK_STATE_ROOT:?gstack-paths failed; reinstall with ./setup or /gstack-upgrade}"');
|
||
expect(report).toContain('mkdir -p "$GSTACK_STATE_ROOT/analytics" || exit 1');
|
||
expect(report).toContain('>> "$GSTACK_STATE_ROOT/analytics/spec-review.jsonl" || exit 1');
|
||
expect(report).not.toContain('Your doc survived');
|
||
});
|
||
|
||
test('CEO spec report separates findings, confirmed fixes, and unresolved concerns', () => {
|
||
const report = render('plan-ceo-review').split('**Step 3: Report and persist metrics**')[1]!.replace(/\s+/g, ' ');
|
||
expect(report).toMatch(/ITERATIONS counts actual reviewer launches/i);
|
||
ordered(report, ['FOUND, FIXED and REMAINING', 'reviewer-confirmed fixes']);
|
||
expect(report).toMatch(/actual counts, never estimates/i);
|
||
expect(report).not.toContain('M issues caught and fixed');
|
||
});
|
||
|
||
test('CEO spec review receives the full source plan as well as the scope artifact on every host', () => {
|
||
for (const host of Object.keys(HOST_PATHS)) {
|
||
const output = render('plan-ceo-review', host).replace(/\s+/g, ' ');
|
||
expect(output).toMatch(/all five dimensions/i);
|
||
expect(output).toMatch(/unsupported accepted expansions/i);
|
||
}
|
||
const source = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf8');
|
||
expect(source).toContain('## Plan under review\n{working plan path, or');
|
||
const template = source.replace(/\s+/g, ' ');
|
||
expectMentions(template, [['cannot','summary','serve']], 'template');
|
||
});
|
||
|
||
test('CEO shares both inputs after spec review and owns unresolved concerns in its scope document', () => {
|
||
const template = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf8');
|
||
const persist = (template.split('### 0H.')[1]?.split('### 0I.')[0] ?? '').replace(/\s+/g, ' ');
|
||
const review = persist.indexOf('{{SPEC_REVIEW_LOOP}}');
|
||
const sharing = persist.search(/present both inputs for final scope-document approval/i);
|
||
expect(review).toBeGreaterThanOrEqual(0);
|
||
expect(sharing).toBeGreaterThan(review);
|
||
const output = render('plan-ceo-review');
|
||
const processing = output.split('**Step 2:')[1]!.split('**Step 3:')[0]!.replace(/\s+/g, ' ');
|
||
const report = output.split('**Step 3:')[1] ?? '';
|
||
expect(report).toContain('CEO summary');
|
||
expect(report).toContain('Reviewer Concerns');
|
||
expect(report).toMatch(/owning\s+input/);
|
||
});
|
||
|
||
test('CEO reviewer receives one complete prompt without duplicate scoring instructions', () => {
|
||
const output = render('plan-ceo-review');
|
||
const dispatch = (output.split('**Step 1: Dispatch reviewer subagent**')[1]?.split('**Step 2:')[0] ?? '').replace(/\s+/g, ' ');
|
||
expect(dispatch.match(/Read both inputs in full/gi)).toHaveLength(1);
|
||
expect(dispatch.match(/quality score/gi)).toHaveLength(1);
|
||
expect(dispatch).toMatch(/quality score \(1-10\)/i);
|
||
expect(dispatch).toMatch(/overall PASS only if all dimensions pass/i);
|
||
expect(dispatch).toMatch(/PASS or numbered issues/i);
|
||
expect(dispatch).toMatch(/cite input and requirement/i);
|
||
});
|
||
|
||
test('contains all 5 review dimensions', () => {
|
||
for (const dim of ['Completeness', 'Consistency', 'Clarity', 'Scope', 'Feasibility']) {
|
||
expect(content).toContain(dim);
|
||
}
|
||
});
|
||
|
||
test('references Agent tool for subagent dispatch', () => {
|
||
expect(content).toMatch(/Agent.*tool/i);
|
||
});
|
||
|
||
test('specifies max 3 iterations', () => {
|
||
expect(content).toMatch(/3.*iteration|maximum.*3/i);
|
||
});
|
||
|
||
test('includes quality score', () => {
|
||
expect(content).toContain('quality score');
|
||
});
|
||
|
||
test('includes metrics path', () => {
|
||
expect(content).toContain('spec-review.jsonl');
|
||
});
|
||
|
||
test('includes convergence guard', () => {
|
||
expect(content).toMatch(/[Cc]onvergence/);
|
||
});
|
||
|
||
test('includes graceful failure handling', () => {
|
||
expect(content).toMatch(/skip.*review|unavailable/i);
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_SKETCH}} resolver tests ---
|
||
|
||
describe('DESIGN_SKETCH resolver', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
|
||
test('references DESIGN.md for design system constraints', () => {
|
||
expect(content).toContain('DESIGN.md');
|
||
});
|
||
|
||
test('contains wireframe or sketch terminology', () => {
|
||
expect(content).toMatch(/wireframe|sketch/i);
|
||
});
|
||
|
||
test('wireframes render through gstack-render (Aside first)', () => {
|
||
expect(content).toContain('gstack-render.ts');
|
||
});
|
||
|
||
test('references screenshot capture', () => {
|
||
expect(content).toContain('--screenshot');
|
||
});
|
||
|
||
test('specifies rough aesthetic', () => {
|
||
expect(content).toMatch(/[Rr]ough|hand-drawn/);
|
||
});
|
||
|
||
test('includes skip conditions', () => {
|
||
expect(content).toMatch(/no UI component|skip/i);
|
||
});
|
||
});
|
||
|
||
// --- {{CODEX_SECOND_OPINION}} resolver tests ---
|
||
|
||
describe('CODEX_SECOND_OPINION resolver', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
const codexContent = readExternalSkillUnion(EXTERNAL_OUT, '.agents', 'office-hours');
|
||
|
||
test('Phase 3.5 section appears in office-hours SKILL.md', () => {
|
||
expect(content).toContain('Phase 3.5: Cross-Model Second Opinion');
|
||
});
|
||
|
||
test('contains codex exec invocation', () => {
|
||
expect(content).toContain('codex exec');
|
||
});
|
||
|
||
test('contains cross-model synthesis instructions', () => {
|
||
expect(content).toMatch(/[Ss]ynthesis/);
|
||
});
|
||
|
||
test('contains Claude subagent fallback', () => {
|
||
expect(content).toContain('CODEX_MODE: not_installed');
|
||
expect(content).toContain('If preflight is not ready');
|
||
expect(content).toContain('Agent tool');
|
||
expect(content).toContain('SECOND OPINION (Claude subagent)');
|
||
});
|
||
|
||
test('contains premise revision check', () => {
|
||
expect(content).toContain('Codex challenged premise');
|
||
});
|
||
|
||
test('contains error handling for auth, timeout, and empty', () => {
|
||
expect(content).toMatch(/[Aa]uth.*fail/);
|
||
expect(content).toMatch(/[Tt]imeout/);
|
||
expect(content).toMatch(/[Ee]mpty response/);
|
||
});
|
||
|
||
test('Codex host runs Phase 3.5 through Claude Code', () => {
|
||
expect(codexContent).toContain('Phase 3.5: Cross-Model Second Opinion');
|
||
expect(codexContent).toContain('gstack-claude-code');
|
||
expect(codexContent).toContain('Claude Code');
|
||
expect(codexContent).not.toMatch(/codex\s+(?:exec|review)\s/);
|
||
});
|
||
});
|
||
|
||
// --- Codex filesystem boundary tests ---
|
||
|
||
describe('Codex filesystem boundary', () => {
|
||
// Skills that call codex exec/review and should contain boundary text
|
||
const CODEX_CALLING_SKILLS = [
|
||
'codex', // /codex skill — 3 modes
|
||
'autoplan', // /autoplan — CEO/design/eng voices
|
||
'review', // /review — adversarial step resolver
|
||
'ship', // /ship — adversarial step resolver
|
||
'plan-eng-review', // outside voice resolver
|
||
'plan-ceo-review', // outside voice resolver
|
||
'office-hours', // second opinion resolver
|
||
];
|
||
|
||
const BOUNDARY_MARKER = /do not read or execute any/i;
|
||
|
||
test('boundary instruction appears in all skills that call codex', () => {
|
||
for (const skill of CODEX_CALLING_SKILLS) {
|
||
// Union: ship's codex call lives in sections/adversarial.md after the carve.
|
||
const content = readSkillUnion(skill);
|
||
expect(content).toMatch(BOUNDARY_MARKER);
|
||
}
|
||
});
|
||
|
||
test('codex skill has Filesystem Boundary section', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('## Filesystem Boundary');
|
||
});
|
||
|
||
test('codex skill has rabbit-hole detection rule', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Detect skill-file rabbit holes');
|
||
expect(content).toContain('gstack-update-check');
|
||
expect(content).toContain('Consider retrying');
|
||
});
|
||
|
||
test('outside-voice-steps.ts CODEX_BOUNDARY constant is interpolated into resolver output', () => {
|
||
// The adversarial step resolver should include boundary text in codex exec
|
||
// prompts. Carved: the adversarial step lives in sections/adversarial.md.
|
||
const reviewContent = readSkillUnion('review');
|
||
// Boundary should appear near codex exec invocations
|
||
const boundaryIdx = reviewContent.search(BOUNDARY_MARKER);
|
||
const codexExecIdx = reviewContent.indexOf('codex exec');
|
||
// Both must exist and boundary must come before a codex exec call
|
||
expect(boundaryIdx).toBeGreaterThan(-1);
|
||
expect(codexExecIdx).toBeGreaterThan(-1);
|
||
});
|
||
|
||
test('autoplan boundary text avoids host-specific paths for cross-host compatibility', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'autoplan', 'SKILL.md.tmpl'), 'utf-8');
|
||
// autoplan template uses generic 'skills/gstack' pattern instead of host-specific
|
||
// paths like ~/.claude/ or .agents/skills (which break Codex/Claude output tests)
|
||
const boundaryStart = content.indexOf('Filesystem Boundary');
|
||
const boundaryEnd = content.indexOf('---', boundaryStart + 1);
|
||
const boundarySection = content.slice(boundaryStart, boundaryEnd);
|
||
expect(boundarySection).not.toContain('~/.claude/');
|
||
expect(boundarySection).not.toContain('.agents/skills');
|
||
expect(boundarySection).toContain('skills/gstack');
|
||
expect(boundarySection).toMatch(BOUNDARY_MARKER);
|
||
});
|
||
});
|
||
|
||
// --- Codex outside-voice availability probe ---
|
||
|
||
describe('Codex outside voice is offered behind an executed availability probe', () => {
|
||
test.each(['office-hours', 'plan-ceo-review', 'plan-design-review', 'plan-eng-review'])('%s probes codex and has a not-installed fallback', skill => {
|
||
const content = readSkillUnion(skill);
|
||
expect(content).toContain('command -v codex');
|
||
expect(content).toContain('not_installed');
|
||
});
|
||
});
|
||
|
||
// --- {{BENEFITS_FROM}} resolver tests ---
|
||
|
||
describe('BENEFITS_FROM resolver', () => {
|
||
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||
const engContent = readSkillUnion('plan-eng-review'); // carved
|
||
|
||
test('plan-ceo-review contains prerequisite skill offer', () => {
|
||
expect(ceoContent).toContain('Prerequisite Skill Offer');
|
||
expect(ceoContent).toContain('/office-hours');
|
||
});
|
||
|
||
test('plan-ceo-review offers /office-hours when no design doc exists and detects a lost user mid-session', () => {
|
||
const offer = extractMarkdownSection(ceoContent, '## Prerequisite Skill Offer');
|
||
expect(offer.replace(/\s+/g, ' ')).toMatch(/when the design doc check above prints "No design doc found,?"/i);
|
||
expect(offer).toContain('A) Run /office-hours now');
|
||
expect(offer).toContain('B) Skip — proceed with standard review');
|
||
expectMentions(offer, [['do not','re-offer','session']], 'offer');
|
||
expect(ceoContent).toContain('**Mid-session detection (0A):**');
|
||
const midSession = ceoContent.indexOf('**Mid-session detection (0A):**');
|
||
expect(ceoContent.slice(midSession, midSession + 300)).toMatch(/offer `\/office-hours`/);
|
||
});
|
||
|
||
test('plan-eng-review contains prerequisite skill offer', () => {
|
||
expect(engContent).toContain('Prerequisite Skill Offer');
|
||
expect(engContent).toContain('/office-hours');
|
||
});
|
||
|
||
test('the optional Eng prerequisite ends before mandatory engineering review', () => {
|
||
const offer = extractMarkdownSection(engContent, '## Prerequisite Skill Offer');
|
||
expect(offer).toContain('B) Skip — proceed with standard review');
|
||
expectMentions(offer, [['do not','re-offer','session']], 'offer');
|
||
expect(offer).not.toContain('### Step 0: Scope Challenge');
|
||
const review = extractMarkdownSection(engContent, '## Engineering review');
|
||
expect(review).toContain('### Step 0: Scope Challenge');
|
||
expectMentions(review, [['before','challenge','mandatory']], 'review');
|
||
expect(review).toContain('sections/review-sections.md');
|
||
});
|
||
|
||
test('Eng initial and prerequisite recheck both discover the canonical override and fall back on slug failure', () => {
|
||
const initial = extractMarkdownSection(engContent, '### Design Doc Check').match(/```bash\n([\s\S]*?)\n```/)![1]!;
|
||
const offer = extractMarkdownSection(engContent, '## Prerequisite Skill Offer');
|
||
expectTokens(offer, ['**Design Doc Check**'], 'offer');
|
||
expectMentions(offer, [['block', 'office-hours', 'completes']], 'offer');
|
||
expect(offer).toContain('This is a fresh execution');
|
||
expectMentions(offer, [['do not','prerequisite','preamble']], 'offer');
|
||
expect(offer).not.toContain('_REVIEW_SLUG=');
|
||
const recheck = initial; // Execute the one canonical block again after the prerequisite.
|
||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'eng-design-slug-'));
|
||
try {
|
||
const home = path.join(dir, 'home'), cwd = path.join(dir, 'project');
|
||
const helper = path.join(home, '.claude/skills/gstack/bin/gstack-slug');
|
||
fs.mkdirSync(path.dirname(helper), {recursive: true});
|
||
fs.mkdirSync(cwd);
|
||
fs.copyFileSync(path.join(ROOT, 'bin/gstack-slug'), helper);
|
||
fs.chmodSync(helper, 0o755);
|
||
fs.copyFileSync(path.join(ROOT, 'bin/gstack-state-root.sh'), path.join(path.dirname(helper), 'gstack-state-root.sh'));
|
||
fs.copyFileSync(path.join(ROOT, 'bin/gstack-remote-identity.sh'), path.join(path.dirname(helper), 'gstack-remote-identity.sh'));
|
||
for (const bin of ['gstack-paths', 'gstack-design-doc-find']) {
|
||
fs.copyFileSync(path.join(ROOT, 'bin', bin), path.join(path.dirname(helper), bin));
|
||
fs.chmodSync(path.join(path.dirname(helper), bin), 0o755);
|
||
}
|
||
const expected = path.join(home, '.gstack/projects/canonical-override/session-unknown-design-current.md');
|
||
const wrong = path.join(home, '.gstack/projects/project/session-unknown-design-wrong.md');
|
||
for (const file of [expected, wrong]) {
|
||
fs.mkdirSync(path.dirname(file), {recursive: true});
|
||
fs.writeFileSync(file, '# Design fixture\n');
|
||
}
|
||
const env = {...process.env, HOME: home, GSTACK_HOME: path.join(home, '.gstack'),
|
||
GSTACK_PROJECT_SLUG: 'canonical-override', GIT_CEILING_DIRECTORIES: dir};
|
||
for (const command of [initial, recheck]) {
|
||
// No eval (#2763): worktree-isolated Claude Code sessions refuse it.
|
||
expect(command).toContain('if SLUG=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG); then');
|
||
expect(command).toContain('BRANCH=$(~/.claude/skills/gstack/bin/gstack-slug --get BRANCH)');
|
||
expect(command).not.toContain('eval ');
|
||
expect(command).toContain('echo "No design doc found"');
|
||
expect(command).not.toContain('remote-slug');
|
||
const result = runCapturedCommand('bash', ['-c', command], {cwd, env, timeout: 5000, captureStdout: true});
|
||
expect(result.status, result.stderr).toBe(0);
|
||
expect(result.stdout).toContain(`Design doc found: ${expected}`);
|
||
expect(result.stdout).not.toContain(wrong);
|
||
}
|
||
fs.writeFileSync(helper, '#!/usr/bin/env bash\necho "slug fixture failed" >&2\nexit 23\n', {mode: 0o755});
|
||
for (const command of [initial, recheck]) {
|
||
const result = runCapturedCommand('bash', ['-c', command], {cwd, env, timeout: 5000, captureStdout: true});
|
||
expect(result.status).toBe(0);
|
||
expect(result.stderr).toContain('slug fixture failed');
|
||
expect(result.stdout).not.toContain('Design doc found:');
|
||
expect(result.stdout).toContain('No design doc found');
|
||
}
|
||
} finally { fs.rmSync(dir, {recursive: true, force: true}); }
|
||
});
|
||
|
||
test('offer includes graceful decline', () => {
|
||
expect(ceoContent).toContain('No worries');
|
||
});
|
||
|
||
test('skills without benefits-from do NOT have prerequisite offer', () => {
|
||
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
|
||
expect(qaContent).not.toContain('Prerequisite Skill Offer');
|
||
});
|
||
|
||
test('inline invocation — no "another window" language', () => {
|
||
expect(ceoContent).not.toContain('another window');
|
||
expect(engContent).not.toContain('another window');
|
||
});
|
||
|
||
test('inline invocation — read-and-follow path present', () => {
|
||
expect(ceoContent).toContain('office-hours/SKILL.md');
|
||
expect(engContent).toContain('office-hours/SKILL.md');
|
||
});
|
||
|
||
test('BENEFITS_FROM delegates to INVOKE_SKILL pattern', () => {
|
||
// Should contain the INVOKE_SKILL-style loading prose (not the old manual skip list)
|
||
expect(engContent).toContain('skipping these sections');
|
||
});
|
||
});
|
||
|
||
// --- {{INVOKE_SKILL}} resolver tests ---
|
||
|
||
describe('INVOKE_SKILL resolver', () => {
|
||
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||
|
||
test('plan-ceo-review uses INVOKE_SKILL for mid-session office-hours fallback', () => {
|
||
// The mid-session detection path should use INVOKE_SKILL-generated prose
|
||
expect(ceoContent).toContain('office-hours/SKILL.md');
|
||
});
|
||
|
||
test('INVOKE_SKILL output includes default skip list', () => {
|
||
expect(ceoContent).toContain('Preamble (run first)');
|
||
expect(ceoContent).toContain('Telemetry (run last)');
|
||
expect(ceoContent).toContain('AskUserQuestion Format');
|
||
});
|
||
|
||
test('INVOKE_SKILL output includes error handling', () => {
|
||
expect(ceoContent).toContain('If unreadable');
|
||
expect(ceoContent).toContain('Could not load');
|
||
});
|
||
|
||
test('template uses {{INVOKE_SKILL:office-hours}} placeholder', () => {
|
||
const tmpl = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(tmpl).toContain('{{INVOKE_SKILL:office-hours}}');
|
||
});
|
||
});
|
||
|
||
// --- {{CHANGELOG_WORKFLOW}} resolver tests ---
|
||
|
||
describe('CHANGELOG_WORKFLOW resolver', () => {
|
||
const shipContent = readShipUnion();
|
||
|
||
test('ship SKILL.md contains changelog workflow', () => {
|
||
expect(shipContent).toContain('CHANGELOG (auto-generate)');
|
||
expect(shipContent).toContain('git log origin/<base>..HEAD --oneline');
|
||
});
|
||
|
||
test('changelog workflow includes cross-check step', () => {
|
||
expect(shipContent).toContain('Cross-check');
|
||
const crossCheck = shipContent.slice(shipContent.indexOf('**Cross-check:**'), shipContent.indexOf('**Do NOT ask the user to describe changes.**'));
|
||
expect(crossCheck).toContain('commit list from step 2');
|
||
expect(crossCheck).toMatch(/user-facing/i);
|
||
});
|
||
|
||
test('changelog workflow includes voice guidance', () => {
|
||
expect(shipContent).toContain('Lead with what the user can now **do**');
|
||
});
|
||
|
||
test('template uses {{CHANGELOG_WORKFLOW}} placeholder', () => {
|
||
// Post-carve (T9): the skeleton points to the changelog section, which carries
|
||
// the resolver. Neither should inline the old changelog content.
|
||
const skel = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md.tmpl'), 'utf-8');
|
||
const changelogSection = fs.readFileSync(path.join(ROOT, 'ship', 'sections', 'changelog.md.tmpl'), 'utf-8');
|
||
expect(skel).toContain('{{SECTION:changelog}}');
|
||
expect(changelogSection).toContain('{{CHANGELOG_WORKFLOW}}');
|
||
expect(skel + changelogSection).not.toContain('Group commits by theme');
|
||
});
|
||
|
||
test('changelog workflow includes keep-changelog format', () => {
|
||
expect(shipContent).toContain('### Added');
|
||
expect(shipContent).toContain('### Fixed');
|
||
});
|
||
});
|
||
|
||
// --- Duplicate {{PREAMBLE}} guard (#2508/#2362) ---
|
||
|
||
describe('assertSinglePreamble', () => {
|
||
test('one {{PREAMBLE}} passes', () => {
|
||
expect(() => assertSinglePreamble('a\n{{PREAMBLE}}\nb', 'x/SKILL.md.tmpl')).not.toThrow();
|
||
});
|
||
|
||
test('zero {{PREAMBLE}} passes (sections have none)', () => {
|
||
expect(() => assertSinglePreamble('no macro here', 'x/sections/y.md.tmpl')).not.toThrow();
|
||
});
|
||
|
||
test('a second occurrence throws with the template path — even in prose', () => {
|
||
// The original #2508 bug WAS a prose mention: "emitted by {{PREAMBLE}}'s
|
||
// preamble bash". Resolution is context-blind, so the guard must be too.
|
||
const tmpl = '{{PREAMBLE}}\n\n...later: emitted by {{PREAMBLE}}\'s preamble bash';
|
||
expect(() => assertSinglePreamble(tmpl, 'spec/SKILL.md.tmpl')).toThrow(/spec\/SKILL\.md\.tmpl.*2 times/);
|
||
});
|
||
});
|
||
|
||
// --- Parameterized resolver infrastructure tests ---
|
||
|
||
describe('parameterized resolver support', () => {
|
||
test('gen-skill-docs regex handles colon-separated args', () => {
|
||
// Verify the template containing {{INVOKE_SKILL:office-hours}} was processed
|
||
// without leaving unresolved placeholders
|
||
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||
expect(ceoContent).not.toMatch(/\{\{INVOKE_SKILL:[^}]+\}\}/);
|
||
});
|
||
|
||
test('templates with parameterized resolvers pass unresolved check', () => {
|
||
// All generated SKILL.md files should have no unresolved {{...}} placeholders
|
||
const skillDirs = fs.readdirSync(ROOT).filter(d =>
|
||
fs.existsSync(path.join(ROOT, d, 'SKILL.md'))
|
||
);
|
||
for (const dir of skillDirs) {
|
||
const content = fs.readFileSync(path.join(ROOT, dir, 'SKILL.md'), 'utf-8');
|
||
const unresolved = content.match(/\{\{[A-Z_]+(?::[^}]*)?\}\}/g);
|
||
if (unresolved) {
|
||
throw new Error(`${dir}/SKILL.md has unresolved placeholders: ${unresolved.join(', ')}`);
|
||
}
|
||
}
|
||
});
|
||
});
|
||
|
||
// --- Preamble routing injection tests ---
|
||
|
||
describe('preamble routing injection (bin/gstack-skill-start emission layer)', () => {
|
||
// Token-reduction Phase 2: the routing-injection prose left the rendered
|
||
// preamble entirely — bin/gstack-skill-start probes, gates, and emits the
|
||
// whole flow as a GSTACK_INSTRUCTION block (with the AUQ, the routing rules
|
||
// to append, and the decline ack all INSIDE the block). Absence from the
|
||
// renders is pinned by test/onboarding-moved-literals.test.ts (tombstone);
|
||
// this suite pins the gate structure and the emitted block's content.
|
||
const routingBlock = (() => {
|
||
const start = SKILL_START_SCRIPT.indexOf('_emit_block routing-injection');
|
||
expect(start).toBeGreaterThan(0);
|
||
return SKILL_START_SCRIPT.slice(start, SKILL_START_SCRIPT.indexOf('\nEOI', start));
|
||
})();
|
||
|
||
test('routing probe checks CLAUDE.md and AGENTS.md (now in gstack-skill-start)', () => {
|
||
// #2500: the probe iterates CLAUDE.md AND AGENTS.md — non-Claude hosts
|
||
// route skills via AGENTS.md, the cross-harness convention file.
|
||
expect(SKILL_START_SCRIPT).toContain('for _RF in CLAUDE.md AGENTS.md');
|
||
expect(SKILL_START_SCRIPT).toContain('grep -q "## Skill routing" "$_RF"');
|
||
expect(SKILL_START_SCRIPT).toContain('echo "HAS_ROUTING: $_HAS_ROUTING"');
|
||
});
|
||
|
||
test('script reads and echoes routing_declined config', () => {
|
||
expect(SKILL_START_SCRIPT).toMatch(/_ROUTING_DECLINED=\$\("\$_BIN\/gstack-config" get routing_declined/);
|
||
expect(SKILL_START_SCRIPT).toContain('echo "ROUTING_DECLINED: $_ROUTING_DECLINED"');
|
||
});
|
||
|
||
test('emitted block carries the routing injection AskUserQuestion', () => {
|
||
expect(routingBlock).toContain('Add routing rules to CLAUDE.md');
|
||
expect(routingBlock).toContain("I'll invoke skills manually");
|
||
});
|
||
|
||
test('routing injection respects prior decline (gate + in-block ack)', () => {
|
||
expect(SKILL_START_SCRIPT).toContain('[ "$_ROUTING_DECLINED" = "false" ]');
|
||
expect(routingBlock).toMatch(/routing_declined.*true/);
|
||
expect(routingBlock).toContain('re-enable with `__BIN__/gstack-config set routing_declined false`');
|
||
});
|
||
|
||
test('routing injection only fires when all conditions met', () => {
|
||
// Must be: HAS_ROUTING=no AND ROUTING_DECLINED=false AND PROACTIVE_PROMPTED=yes
|
||
expect(SKILL_START_SCRIPT).toContain(
|
||
'if [ "$_HAS_ROUTING" = "no" ] && [ "$_ROUTING_DECLINED" = "false" ] && [ "$_PROACTIVE_PROMPTED" = "yes" ]; then',
|
||
);
|
||
});
|
||
|
||
test('routing section content includes key routing rules', () => {
|
||
expect(routingBlock).toContain('invoke /office-hours');
|
||
expect(routingBlock).toContain('invoke /investigate');
|
||
expect(routingBlock).toContain('invoke /ship');
|
||
expect(routingBlock).toContain('invoke /qa');
|
||
});
|
||
|
||
test('routing section uses renamed checkpoint skills (not stale /checkpoint)', () => {
|
||
expect(routingBlock).toContain('invoke /context-save');
|
||
expect(routingBlock).toContain('invoke /context-restore');
|
||
expect(routingBlock).not.toContain('invoke checkpoint');
|
||
});
|
||
|
||
test('routing section uses soft "when in doubt" policy, not hard "ALWAYS invoke"', () => {
|
||
expect(routingBlock).toContain('When in doubt, invoke the skill');
|
||
expect(routingBlock).not.toContain('Do NOT answer directly');
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_OUTSIDE_VOICES}} resolver tests ---
|
||
|
||
describe('DESIGN_OUTSIDE_VOICES resolver', () => {
|
||
test('plan-design-review contains outside voices section', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Design Outside Voices');
|
||
expect(content).toContain('CODEX_MODE: ready');
|
||
expect(content).toContain('LITMUS SCORECARD');
|
||
});
|
||
|
||
test('design-review contains outside voices section', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Design Outside Voices');
|
||
expect(content).toContain('source audit');
|
||
});
|
||
|
||
test('design-consultation contains outside voices section', () => {
|
||
const content = readSkillUnion('design-consultation');
|
||
expect(content).toContain('Design Outside Voices');
|
||
expect(content).toContain('design direction');
|
||
});
|
||
|
||
test('consultation dispatch shares the complete brief without executing product text', () => {
|
||
const content = readSkillUnion('design-consultation');
|
||
const command = [...content.matchAll(/```bash\n([\s\S]*?)```/g)]
|
||
.map(match => match[1]).find(block => block.includes('codex exec'))!;
|
||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'design-brief-contract-'));
|
||
const quote = (value: string) => "'" + value.replaceAll("'", "'\\''") + "'";
|
||
try {
|
||
const brief = path.join(dir, 'product.md');
|
||
const capture = path.join(dir, 'argv.json');
|
||
const marker = path.join(dir, 'must-not-execute');
|
||
const product = `A product with $(touch ${marker}), \`touch ${marker}\`, and "quotes".\nUsers: builders.\n`;
|
||
fs.writeFileSync(brief, product);
|
||
// Fake codex: the free sandbox preflight succeeds; exec writes its final message to -o.
|
||
fs.writeFileSync(path.join(dir, 'codex'), `#!${process.execPath}\nconst fs = require('fs'); const a = process.argv.slice(2); if (a[0] === 'sandbox') process.exit(0);\nfs.writeFileSync(process.env.CAPTURE, JSON.stringify({ args: a, stdin: a[1] === '-' ? fs.readFileSync(0, 'utf8') : '' }));\nconst msg = 'Recommendation: choose a clear hierarchy because builders need to find their work.';\nif (a.includes('-o')) fs.writeFileSync(a[a.indexOf('-o') + 1], msg);\nconsole.log(msg);\n`, { mode: 0o700 });
|
||
const prepare = (file: string) => command.replace("'<prepared-prompt-file>'", quote(file))
|
||
.replaceAll('$HOME/.claude/skills/gstack', ROOT);
|
||
const env = { ...process.env, PATH: dir + path.delimiter + process.env.PATH, CAPTURE: capture,
|
||
CODEX_THREAD_ID: '', CODEX_SANDBOX: '', CLAUDECODE: '1', GSTACK_ACTIVE_HOST: 'claude',
|
||
GSTACK_HOME: path.join(dir, 'state'), GSTACK_STATE_ROOT: '' };
|
||
const result = spawnSync('bash', ['-c', prepare(brief)], { cwd: ROOT, env, encoding: 'utf8', timeout: 5_000 });
|
||
expect(result.status, result.stderr).toBe(0);
|
||
const { args, stdin } = JSON.parse(fs.readFileSync(capture, 'utf8'));
|
||
expect(args[0]).toBe('exec');
|
||
expect(args[1]).toBe('-');
|
||
expect(stdin).toBe(product);
|
||
expect(args).toContain('read-only');
|
||
expect(fs.readFileSync(brief, 'utf8')).toBe(product);
|
||
expect(fs.existsSync(marker)).toBe(false);
|
||
fs.unlinkSync(capture);
|
||
const missing = spawnSync('bash', ['-c', prepare(path.join(dir, 'absent'))], { cwd: ROOT, env, encoding: 'utf8', timeout: 5_000 });
|
||
expect(missing.status).not.toBe(0);
|
||
expect(missing.stderr).toContain('No such file');
|
||
expect(fs.existsSync(capture)).toBe(false);
|
||
expect(content).toContain('await both before synthesis');
|
||
expect(content).toContain('use only the native voice');
|
||
expect(content).toContain('give the native Agent its absolute path');
|
||
expect(content).toContain('Read the complete product brief at [the absolute DESIGN_BRIEF path printed above]');
|
||
expect(content).toContain('Omit faces you cannot verify');
|
||
expect(content).toContain('a face may serve multiple roles');
|
||
expect(content).not.toContain('a single question that covers everything');
|
||
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
|
||
});
|
||
|
||
test('branches correctly per skillName — different prompts', () => {
|
||
const planContent = readSkillUnion('plan-design-review');
|
||
const consultContent = readSkillUnion('design-consultation');
|
||
// plan-design-review uses analytical prompt (high reasoning)
|
||
expect(planContent).toContain('model_reasoning_effort="high"');
|
||
// design-consultation uses creative prompt (medium reasoning)
|
||
expect(consultContent).toContain('model_reasoning_effort="medium"');
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_HARD_RULES}} resolver tests ---
|
||
|
||
describe('DESIGN_HARD_RULES resolver', () => {
|
||
test('plan-design-review Pass 4 contains hard rules', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Design Hard Rules');
|
||
expect(content).toContain('Classifier');
|
||
expect(content).toContain('MARKETING/LANDING PAGE');
|
||
expect(content).toContain('APP UI');
|
||
});
|
||
|
||
test('design-review contains hard rules', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Design Hard Rules');
|
||
});
|
||
|
||
test('includes all 3 rule sets', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Landing page rules');
|
||
expect(content).toContain('App UI rules');
|
||
expect(content).toContain('Universal rules');
|
||
});
|
||
|
||
test('classifier names the four visitor modes and keeps the legacy aliases', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
for (const mode of ['PERSUADE', 'OPERATE', 'READ', 'EXPERIENCE', 'HYBRID']) expect(content).toContain(`**${mode}**`);
|
||
expect(content).toContain('Read rules');
|
||
expect(content).toContain('Experience rules');
|
||
expectMentions(content, [['not', 'classify', 'section']], 'content');
|
||
});
|
||
|
||
test('carries the craft-floor reflexes and the three-looks calibration', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Reflexes no detector catches');
|
||
expect(content).toContain('Browser surfaces carry the design');
|
||
expect(content).toContain('One authored motion moment');
|
||
expect(content).toContain('Depth has an offset');
|
||
expect(content).toContain('Light or dark comes from the use scene');
|
||
expect(content).toContain('Calibration: the three looks');
|
||
});
|
||
|
||
test('slop section lists detector rule ids and judgment tells outside design-review', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('nested-cards: Nested cards');
|
||
expectMentions(content, [['no', 'judgment', 'detector']], 'content');
|
||
// Never a bracketed gstack-only id.
|
||
expect(content).not.toContain('[hero-metrics]');
|
||
});
|
||
|
||
test('design-consultation carries the font procedure, role-scoped lists, color strategies, and catalog bullets', () => {
|
||
const content = readSkillUnion('design-consultation');
|
||
expect(content).toContain('Choosing faces: a procedure, not a menu');
|
||
expect(content).toContain('**Overused as display**');
|
||
expect(content).toContain('**Banned in any role:** Papyrus');
|
||
expect(content).toContain('Restrained (1 accent + neutrals');
|
||
expectMentions(content, [['not', 'light', 'dials']], 'content');
|
||
expect(content).toContain('Calibration: the three looks');
|
||
// Bullets are prose only: never a bracketed rule id in the proposal skill.
|
||
expect(content).toContain('- A card inside a card is always wrong.');
|
||
expect(content).not.toMatch(/^- \[[a-z-]+\] /m);
|
||
// The old menu is gone.
|
||
expect(content).not.toContain('Font recommendations by purpose');
|
||
});
|
||
|
||
test('design-html blacklist lines carry catalog ids', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
||
expect(content).toMatch(/AI slop blacklist/i);
|
||
expect(content).toContain('Purple/blue gradients as default <!-- ai-color-palette -->');
|
||
expect(content).toContain('lib/design-catalog.ts');
|
||
});
|
||
|
||
test('design-review renders the catalog once: Methodology category 9 carries it, Hard Rules points at it', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expect(content.split('### Design Hard Rules').length - 1).toBe(1);
|
||
// Category 9 lists the rule once (ids only); Typography points at the same id from its overused-face item.
|
||
expect(content.split('[overused-font]').length - 1).toBe(2);
|
||
expect(content).toContain('are Methodology category 9');
|
||
expect(content).toContain('**9. AI Slop Detection**');
|
||
expect(content).toContain('Detector rules (ids only;');
|
||
expect(content).toContain('[nested-cards] nested cards');
|
||
expect(content).toContain('Judgment tells (no detector rule');
|
||
// The legacy blacklist is not repeated as a numbered list in design-review.
|
||
expect(content).not.toMatch(/^1\. Purple\/violet\/indigo/m);
|
||
});
|
||
|
||
test('references shared AI slop blacklist items', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('3-column feature grid');
|
||
expect(content).toContain('Purple/violet/indigo');
|
||
});
|
||
|
||
test('includes OpenAI hard rejection criteria', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Generic SaaS card grid');
|
||
expect(content).toContain('Carousel with no narrative purpose');
|
||
});
|
||
|
||
test('includes OpenAI litmus checks', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Brand/product unmistakable');
|
||
});
|
||
});
|
||
|
||
describe('Design approval reconciliation', () => {
|
||
test('token alignment reuses the approved outcome while preserving first decisions and new tradeoffs', () => {
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-design-review/sections/review-sections.md'), 'utf8');
|
||
const pass = extractMarkdownSection(section, '### Pass 5: Design System Alignment');
|
||
const ask = pass.search(/stop\W+AskUserQuestion once per issue/i);
|
||
expect(ask).toBeGreaterThan(0);
|
||
const beforeQuestion = pass.slice(0, ask);
|
||
expect(beforeQuestion).toMatch(/already approved that outcome/i);
|
||
expect(beforeQuestion).toMatch(/not a new issue/i);
|
||
expectMentions(beforeQuestion, [['only','evidence','again']], 'beforeQuestion');
|
||
});
|
||
|
||
test('loaded review section distinguishes individual issue decisions from navigation', () => {
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-design-review/sections/review-sections.md'), 'utf8').replace(/\s+/g, ' ');
|
||
expectMentions(section, [['no','next-step','remedies']], 'section');
|
||
expectMentions(section, [['never','askuserquestion','issue-approval']], 'section');
|
||
expectMentions(section, [['no','unresolved','findings']], 'section');
|
||
});
|
||
|
||
test('Design blocking Exit checklist reconciles approvals and refreshes stale output after a decision', () => {
|
||
const main = fs.readFileSync(path.join(ROOT, 'plan-design-review/SKILL.md'), 'utf8');
|
||
const check = extractMarkdownSection(main, '## Section self-check');
|
||
expect(check).toMatch(/run approval check 0/i);
|
||
const gate = extractMarkdownSection(main, '## EXIT PLAN MODE GATE (BLOCKING)').replace(/\s+/g, ' ');
|
||
ordered(gate, ['0. Approvals:', '1. Read the plan file']);
|
||
const approvals = gate.slice(gate.indexOf('0. Approvals:'), gate.indexOf('1. Read the plan file'));
|
||
expect(approvals).toMatch(/never group distinct issues/i);
|
||
expect(approvals).toMatch(/navigation are not approval/i);
|
||
expect(approvals).toContain('preamble-authorized');
|
||
expect(approvals).toMatch(/deferrals remain unresolved/i);
|
||
expectMentions(approvals, [['wait','pending','drafts']], 'approvals');
|
||
ordered(approvals, ['refresh the plan and report', 'Read-back gate', 'rerun this gate']);
|
||
});
|
||
|
||
test('CEO readiness binds actual approvals before its read-only exit verification', () => {
|
||
const main = fs.readFileSync(path.join(ROOT, 'plan-ceo-review/SKILL.md'), 'utf8');
|
||
const check = extractMarkdownSection(main, '## Section self-check').replace(/\s+/g, ' ');
|
||
const gate = extractMarkdownSection(main, '## EXIT PLAN MODE GATE (BLOCKING)').replace(/\s+/g, ' ');
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-ceo-review/sections/review-sections.md'), 'utf8');
|
||
const readiness = extractMarkdownSection(section, '## Approval readiness').replace(/\s+/g, ' ');
|
||
expect(section.indexOf('## Approval readiness')).toBeLessThan(section.indexOf('## Required Outputs'));
|
||
expectMentions(readiness, [['approval','preamble-authorized','exact']], 'readiness');
|
||
expectMentions(readiness, [['not','navigation','approvals']], 'readiness');
|
||
expectMentions(readiness, [['only','commitments','approves']], 'readiness');
|
||
expectMentions(readiness, [['only','applies','scope']], 'readiness');
|
||
expect(readiness).toMatch(/delivery-scope deferral is settled/i);
|
||
expect(readiness).toMatch(/leaves that choice unresolved/i);
|
||
expect(readiness).toContain('`Approval readiness: PASS`');
|
||
expect(readiness).toMatch(/navigation alone does not/i);
|
||
expect(main.replace(/\s+/g, ' ')).toMatch(/default to implementation-ready/i);
|
||
ordered(gate, ['Read-only verification', 'Verify `Approval readiness: PASS`', '1. Read the plan file']);
|
||
expectMentions(gate, [['only','readiness','return']], 'gate');
|
||
expect(gate).toContain('**Gate outcome: Blocked**');
|
||
expect(main).toContain('**Pass with log-only gaps:**');
|
||
expectMentions(gate, [['without','exitplanmode','telemetry']], 'gate');
|
||
expect(check).toContain('`sections/review-sections.md`');
|
||
expectMentions(check, [['stop','review','read']], 'check');
|
||
const decisions = main.slice(main.indexOf('### 0D.'), main.indexOf('### 0E.')).replace(/\s+/g, ' ');
|
||
ordered(decisions, ["**Choose the question's route first:**", '**Admin question:**', '**Plan decision:**']);
|
||
expect(decisions).toMatch(/reuse exact prior approvals/i);
|
||
expect(decisions).toContain('`D<N> — <ROW-ID>: <one-line question>`');
|
||
const row = decisions.slice(decisions.indexOf('**Pre-question checkpoint:**'), decisions.indexOf('Effort/risk must'));
|
||
for (const field of ['ID', 'owner', 'Current/Proposed', 'Status', 'Exact approval and scope']) expect(row).toContain(field);
|
||
expectMentions(decisions.slice(decisions.indexOf('**4. Ask')), [['stop','actual','answer']], 'section');
|
||
expect(decisions).toMatch(/amend only (?:what it authorizes|authorized work)/i);
|
||
expectMentions(decisions, [['only','contradictions','instructions']], 'decisions');
|
||
expect(decisions).toContain('"No issues, moving on."');
|
||
const outside = extractMarkdownSection(section, '**Cross-model tension:**').replace(/\s+/g, ' ');
|
||
expectMentions(outside, [['only','completed','external']], 'outside');
|
||
expect(outside).toContain('CROSS-MODEL');
|
||
const findings = extractMarkdownSection(section, '**Integrate reviewer findings:**').replace(/\s+/g, ' ');
|
||
expect(findings).toMatch(/bounded native fallback/i);
|
||
expect(findings).toContain('Outside Voice Integration Rule');
|
||
});
|
||
|
||
test('Eng cannot exit with unasked findings listed only in an unresolved-decisions report', () => {
|
||
const main = fs.readFileSync(path.join(ROOT, 'plan-eng-review/SKILL.md'), 'utf8');
|
||
const rawGate = extractMarkdownSection(main, '## EXIT PLAN MODE GATE (BLOCKING)');
|
||
const gate = rawGate.replace(/\s+/g, ' ');
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-eng-review/sections/review-sections.md'), 'utf8');
|
||
const readiness = extractMarkdownSection(section, '## Approval readiness');
|
||
const normalizedReadiness = readiness.replace(/\s+/g, ' ');
|
||
expect(section.indexOf('## Approval readiness')).toBeLessThan(section.indexOf('## Required outputs'));
|
||
expect(normalizedReadiness).toContain('`Approval readiness: PASS`');
|
||
expect(normalizedReadiness).toMatch(/navigation alone does not/i);
|
||
expect(readiness).not.toMatch(/^ {3}\S/m);
|
||
expect(gate).toMatch(/confirm approval readiness passed/i);
|
||
expectMentions(gate, [['not','verification','read-only']], 'gate');
|
||
const approvalParagraph = rawGate.slice(rawGate.indexOf('Confirm Approval readiness'), rawGate.indexOf('Verify all five checks'));
|
||
expect(approvalParagraph).not.toMatch(/^ {3}\S/m);
|
||
expect([...rawGate.matchAll(/^([1-5])\. /gm)].map(match => match[1])).toEqual(['1', '2', '3', '4', '5']);
|
||
ordered(gate, ['Confirm Approval readiness', '1. Read the report file']);
|
||
expect(normalizedReadiness).toMatch(/every accepted remedy/i);
|
||
expectMentions(normalizedReadiness, [['approval','auto-decision','authorized']], 'normalizedReadiness');
|
||
expect(normalizedReadiness).toMatch(/navigation do not count/i);
|
||
expect(normalizedReadiness).toMatch(/deferrals remain unresolved/i);
|
||
expect(normalizedReadiness).toMatch(/one dedicated decision/i);
|
||
expect(normalizedReadiness).toMatch(/mark that draft pending/i);
|
||
const decisionStart = section.indexOf('\n## Decision procedure\n');
|
||
const decisionEnd = section.indexOf('\n## Scope Challenge\n', decisionStart);
|
||
expect(decisionStart).toBeGreaterThan(0);
|
||
expect(decisionEnd).toBeGreaterThan(decisionStart);
|
||
const decisions = section.slice(decisionStart, decisionEnd).replace(/\s+/g, ' ');
|
||
expect(decisions).toContain('AskUserQuestion({ questions: [currentDecision] })');
|
||
const ask = decisions.indexOf('AskUserQuestion({ questions: [currentDecision] })');
|
||
const stop = decisions.search(/stop until the actual answer arrives/i);
|
||
expect(stop).toBeGreaterThan(ask);
|
||
expect(stop).toBeLessThan(decisions.indexOf('### Record the answer'));
|
||
expectMentions(gate, [['stop','verification','report']], 'gate');
|
||
expect(gate).toContain('**Recovery routing → Late change or missing work**');
|
||
const recovery = readSkillUnion('plan-eng-review').split('**Late change or missing work:**')[1]!.split('**Blocked outcome:**')[0]!.replace(/\s+/g, ' ');
|
||
ordered(recovery, ['Repeat Approval readiness', 'Required outputs', 'navigation']);
|
||
expect(gate).toContain('**Blocked outcome**');
|
||
const report = extractMarkdownSection(section, '### Write to the report file');
|
||
expect(report).toContain('**Blocked outcome**');
|
||
expect(report.replace(/\s+/g, ' ')).toMatch(/follow \*\*Blocked outcome\*\* before Review Log or decision logging/i);
|
||
});
|
||
|
||
test('approval entry does not alter other review Exit checklists', () => {
|
||
for (const skill of ['plan-devex-review']) {
|
||
const gate = extractMarkdownSection(readSkillUnion(skill), '## EXIT PLAN MODE GATE (BLOCKING)');
|
||
expect(gate).not.toContain('0. Approvals:');
|
||
expect(gate).toContain('1. Read the plan file');
|
||
expect(gate).toContain('5. If a plan file is in context');
|
||
}
|
||
});
|
||
|
||
test('task and decision reporting require approval while preserving unanswered findings', () => {
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-design-review/sections/review-sections.md'), 'utf8').replace(/\s+/g, ' ');
|
||
const reconcile = section.search(/before synthesizing tasks or the completion summary/i);
|
||
expect(reconcile).toBeGreaterThan(0);
|
||
expect(reconcile).toBeLessThan(section.indexOf('## Implementation Tasks'));
|
||
expectMentions(section, [['only','individually','decisions']], 'section');
|
||
expectMentions(section, [['not','including','finding']], 'section');
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_DETECTOR}} resolver tests ---
|
||
|
||
describe('DESIGN_DETECTOR resolver', () => {
|
||
const designReview = () => fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
const designHtml = () => fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
||
const bashBlocksOf = (content: string) => [...content.matchAll(/```bash\n([\s\S]*?)```/g)].map(m => m[1]);
|
||
|
||
test('design-review carries the probe, Phase 0, the DOM dump, and the run id', () => {
|
||
const c = designReview();
|
||
expect(c).toContain('gstack-design-detect.ts probe --host claude');
|
||
expect(c).toContain('IMPECCABLE_READY');
|
||
// the consent-gated install: offered once, only on the probe's say-so, never in spawned sessions, never via npx
|
||
expect(c).toContain('DESIGN_DETECTOR_INSTALL_OFFER');
|
||
expect(c).toContain('gstack-design-detect.ts install --host claude');
|
||
expect(c).toContain("Install impeccable's design detector engine?");
|
||
expect(c).toContain('gstack-config set design_detector_install_prompted true');
|
||
expect(c).toContain('`SESSION_KIND: spawned` or a headless run, never install and never ask');
|
||
expect(c).toContain('**Phase 0: mechanical scan**');
|
||
expect(c).toContain('scan --changed <base> --format gstack --host claude');
|
||
expect(c).toContain('### DOM dump (DOM mode only');
|
||
expect(c).toContain('data-gstack-dom-css');
|
||
expect(c).toContain(`$B js '('"$_DUMP"')()' --out "$_TMP/{page}.dom.html" --raw`);
|
||
expect(c).toContain('DOM_DUMP_OK');
|
||
expect(c).toContain('DOM_DUMP_REDACTION_BLOCKED');
|
||
expect(c).toContain('DOM_DUMP_TOO_LARGE');
|
||
expect(c).toContain('REPORT_DIR="$GSTACK_STATE_ROOT/projects/$SLUG/designs/design-audit-$(date +%Y%m%d)"');
|
||
expect(c).toContain('RUN_ID="$(date +%H%M%S)-$$"');
|
||
expect(c).toContain('"schemaVersion": 2');
|
||
expect(c).toContain('engine changed X → Y; rule set may differ');
|
||
expect(c).toContain('Detector: N → M');
|
||
expect(c).toContain('/impeccable typeset');
|
||
});
|
||
|
||
test('the DOM-dump script is loaded from lib/dom-dump.js, never inlined in the prose', () => {
|
||
const c = designReview();
|
||
expect(c).not.toMatch(/```js\n/);
|
||
expect(c).not.toContain('document.documentElement.cloneNode');
|
||
expect(c).toContain('_DUMP=$(cat "$HOME/.claude/skills/gstack/lib/dom-dump.js")');
|
||
expect(c).toContain(`const html = await pg.evaluate('"$_DUMP"');`);
|
||
expect(c).toContain('_TMP=$(mktemp -d "${TMPDIR:-/tmp}/gstack-dom.XXXXXX"); _DUMP=$(cat "$HOME/.claude/skills/gstack/lib/dom-dump.js")');
|
||
});
|
||
|
||
test('every rendered Aside script is single-quoted: a page-controlled <url> is never inside a double-quoted bash string', () => {
|
||
const files = [...fs.readdirSync(ROOT).filter(d => fs.existsSync(path.join(ROOT, d, 'SKILL.md'))).map(d => path.join(ROOT, d, 'SKILL.md')),
|
||
...fs.readdirSync(ROOT).flatMap(d => fs.existsSync(path.join(ROOT, d, 'sections')) ? fs.readdirSync(path.join(ROOT, d, 'sections')).filter(f => f.endsWith('.md')).map(f => path.join(ROOT, d, 'sections', f)) : [])];
|
||
expect(files.length).toBeGreaterThan(10);
|
||
for (const f of files) {
|
||
const c = fs.readFileSync(f, 'utf-8');
|
||
expect(c, path.relative(ROOT, f)).not.toMatch(/^aside repl "/m);
|
||
}
|
||
});
|
||
|
||
test('the E2E fixture slice markers exist in the rendered design skills (a template rename fails here, not in paid CI)', () => {
|
||
const dr = designReview();
|
||
const dh = fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
||
for (const [a, b] of [['**Design detector (optional, deterministic):**', '**Create output directories:**'], ['**Phase 0: mechanical scan**', '## Phases 1-6'], ['### DOM dump (DOM mode only', '### Auth Detection']]) {
|
||
expect(sliceBetween(dr, a, b).length, `${a} .. ${b}`).toBeGreaterThan(100);
|
||
}
|
||
for (const [a, b] of [['**Design detector (optional, deterministic):**', '## Step 0: Input Detection'], ['### Slop Gate (bounded, never a loop)', '### Verification Screenshots']]) {
|
||
expect(sliceBetween(dh, a, b).length, `${a} .. ${b}`).toBeGreaterThan(100);
|
||
}
|
||
});
|
||
|
||
test('design-html carries the probe and the bounded slop gate', () => {
|
||
const c = designHtml();
|
||
expect(c).toContain('gstack-design-detect.ts probe --host claude');
|
||
expect(c).toContain('### Slop Gate (bounded, never a loop)');
|
||
expect(c).toContain('One pass, not a loop.');
|
||
expect(c).toContain('impeccable-disable <rule>: <reason>');
|
||
});
|
||
|
||
test('ship and review unions reach the detector through review-lite and the checklist', () => {
|
||
const ship = readSkillUnion('ship');
|
||
expect(ship).toContain('**Mechanical pass first.**');
|
||
expect(ship).toContain('scan --changed <base> --format gstack --host claude');
|
||
expect(ship).toContain('"detector":D');
|
||
expect(ship).toContain('Detector: "clean" | "N findings');
|
||
const review = readSkillUnion('review');
|
||
const checklist = fs.readFileSync(path.join(ROOT, 'review', 'design-checklist.md'), 'utf-8');
|
||
expect(checklist).toContain('**0. Mechanical pass first.**');
|
||
expect(checklist).toContain('IMPECCABLE_READY');
|
||
});
|
||
|
||
test('every rendered invocation uses bun --no-env-file and ends a scan with the exit echo; no bash block runs npx impeccable', () => {
|
||
for (const content of [designReview(), designHtml(), readSkillUnion('ship'), readSkillUnion('review'), fs.readFileSync(path.join(ROOT, 'review', 'design-checklist.md'), 'utf-8')]) {
|
||
for (const block of bashBlocksOf(content)) {
|
||
expect(block).not.toContain('npx impeccable');
|
||
for (const line of block.split('\n')) {
|
||
if (!line.includes('gstack-design-detect.ts')) continue;
|
||
expect(line).toContain('bun --no-env-file run ');
|
||
if (/gstack-design-detect\.ts scan /.test(line)) expect(line).toContain('echo "DETECT_EXIT_CODE=$?"');
|
||
}
|
||
}
|
||
}
|
||
});
|
||
|
||
test('--host is rendered per host', () => {
|
||
// Fresh codex render into a temp out-dir: the tracked tree is Claude-only and
|
||
// the gitignored .agents/ copy may be stale.
|
||
const out = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-detector-host-'));
|
||
try {
|
||
const r = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--out-dir', out], { cwd: ROOT, timeout: 120_000 });
|
||
expect(r.exitCode).toBe(0);
|
||
const codex = fs.readFileSync(path.join(out, '.agents', 'skills', 'gstack-design-review', 'SKILL.md'), 'utf-8');
|
||
expect(codex).toContain('gstack-design-detect.ts probe --host codex');
|
||
expect(codex).not.toContain('probe --host claude');
|
||
expect(codex).toContain('$GSTACK_ROOT/lib/dom-dump.js');
|
||
} finally {
|
||
fs.rmSync(out, { recursive: true, force: true });
|
||
}
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_MD_CHECK}} resolver + open DESIGN.md adoption ---
|
||
|
||
describe('DESIGN_MD_CHECK resolver and open DESIGN.md adoption', () => {
|
||
test('design-consultation asks the conversion question once and writes the spec form', () => {
|
||
const c = readSkillUnion('design-consultation');
|
||
expect(c).toContain('gstack-design-md.ts check DESIGN.md');
|
||
expect(c).toContain('DESIGN_MD_FORMAT: spec');
|
||
expect(c).toContain('mark legacy-keep');
|
||
expect(c).toContain('convert --write');
|
||
expect(c).toContain('# gstack: design-md-format=spec');
|
||
expect(c).toContain("## Do's and Don'ts");
|
||
expect(c).toContain('## Elevation & Depth');
|
||
expect(c).toContain('fontFeature: tnum');
|
||
expect(c).toContain('"{colors.primary}"');
|
||
// the legacy template is gone
|
||
expect(c).not.toContain('## Product Context\n- **What this is:**');
|
||
});
|
||
|
||
test('design-review calibrates against tokens and never re-offers conversion; design-html writes the spec form', () => {
|
||
const dr = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expect(dr).toContain('gstack-design-md.ts check DESIGN.md');
|
||
expect(dr).toContain('gstack-design-md.ts tokens DESIGN.md');
|
||
expect(dr).toMatch(/never offer a conversion/i);
|
||
expect(dr).not.toContain('mark legacy-keep');
|
||
const dh = fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
||
expect(dh).toContain('# gstack: design-md-format=spec');
|
||
const pdr = readSkillUnion('plan-design-review');
|
||
expect(pdr).toContain('{colors.primary}');
|
||
const checklist = fs.readFileSync(path.join(ROOT, 'review', 'design-checklist.md'), 'utf-8');
|
||
expect(checklist).toContain('gstack-design-md.ts tokens DESIGN.md');
|
||
expect(readSkillUnion('ship')).toContain('gstack-design-md.ts tokens DESIGN.md');
|
||
});
|
||
|
||
test('every rendered gstack-design-md invocation uses bun --no-env-file', () => {
|
||
for (const content of [readSkillUnion('design-consultation'), fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8'), readSkillUnion('ship')]) {
|
||
for (const line of content.split('\n')) {
|
||
if (line.includes('gstack-design-md.ts')) expect(line).toContain('bun --no-env-file run ');
|
||
}
|
||
}
|
||
});
|
||
});
|
||
|
||
// --- PRODUCT.md prefill + /impeccable handoffs ---
|
||
|
||
describe('PRODUCT.md prefill and /impeccable handoffs', () => {
|
||
test('design-consultation and design-shotgun read PRODUCT.md and never open the impeccable skill', () => {
|
||
for (const skill of ['design-consultation', 'design-shotgun']) {
|
||
const c = readSkillUnion(skill);
|
||
expect(c).toContain('cat PRODUCT.md 2>/dev/null | head -120 || echo "NO_PRODUCT_MD"');
|
||
expect(c).toContain('do not re-ask');
|
||
expect(c).toContain('Never open `.claude/skills/impeccable/**`');
|
||
}
|
||
});
|
||
|
||
test('handoffs are gated on IMPECCABLE_SKILL: present in review-lite and design-review', () => {
|
||
expect(readSkillUnion('ship')).toContain('IMPECCABLE_SKILL: present`, end each NEEDS INPUT detector row with the `handoff=` command');
|
||
const dr = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expectTokens(dr, ['`handoff=`', '`IMPECCABLE_SKILL: present`'], 'dr');
|
||
expectTokens(dr, ['`/impeccable`'], 'dr');
|
||
});
|
||
});
|
||
|
||
// --- Extended DESIGN_SKETCH resolver tests ---
|
||
|
||
describe('DESIGN_SKETCH extended with outside voices', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
|
||
test('contains outside design voices step', () => {
|
||
expect(content).toContain('Outside design voices');
|
||
});
|
||
|
||
test('offers opt-in via AskUserQuestion', () => {
|
||
expect(content).toContain('outside design perspectives');
|
||
});
|
||
|
||
test('still contains original wireframe steps', () => {
|
||
expect(content).toContain('wireframe');
|
||
expect(content).toContain('gstack-render.ts');
|
||
});
|
||
});
|
||
|
||
// --- Extended DESIGN_REVIEW_LITE resolver tests ---
|
||
|
||
describe('DESIGN_REVIEW_LITE extended with Codex', () => {
|
||
const content = readShipUnion();
|
||
|
||
test('contains Codex design voice block', () => {
|
||
expect(content).toContain('Codex design voice');
|
||
expect(content).toContain('CODEX (design)');
|
||
});
|
||
|
||
test('still contains original checklist steps', () => {
|
||
expect(content).toContain('design-checklist.md');
|
||
expect(content).toContain('SCOPE_FRONTEND');
|
||
});
|
||
|
||
test('design-checklist path uses installed gstack/review root (#2694)', () => {
|
||
// #2694: generateDesignReviewLite used to emit
|
||
// `.claude/skills/review/design-checklist.md` (missing the gstack/ segment).
|
||
// After install the file lives at ~/.claude/skills/gstack/review/design-checklist.md.
|
||
// The bad relative form must not appear — the good path does not contain it
|
||
// as a substring because `gstack/` sits between `skills/` and `review/`.
|
||
expect(content).toContain('~/.claude/skills/gstack/review/design-checklist.md');
|
||
expect(content).not.toContain('.claude/skills/review/design-checklist.md');
|
||
});
|
||
|
||
});
|
||
|
||
// ─── Codex Generation Tests ─────────────────────────────────
|
||
|
||
describe('Codex generation (--host codex)', () => {
|
||
// .agents/ is gitignored (v0.11.2.0) — read the module-level out-dir render
|
||
// (--host all covers codex) instead of regenerating the live tree in place.
|
||
const AGENTS_DIR = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
||
|
||
// Dynamic discovery of expected Codex skills: all templates except /codex.
|
||
// The out-dir is a fresh mkdtemp, so the vendored-dev-mode symlink loop
|
||
// (.agents/skills/{name} → repo root) that made the generator skip skills
|
||
// in-place can never occur here — every template renders.
|
||
const CODEX_SKILLS = (() => {
|
||
const skills: Array<{ dir: string; codexName: string }> = [];
|
||
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
||
skills.push({ dir: '.', codexName: 'gstack' });
|
||
}
|
||
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
||
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||
if (entry.name === 'codex') continue; // /codex is excluded from Codex output
|
||
if (!fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) continue;
|
||
const codexName = entry.name.startsWith('gstack-') ? entry.name : `gstack-${entry.name}`;
|
||
skills.push({ dir: entry.name, codexName });
|
||
}
|
||
return skills;
|
||
})();
|
||
|
||
test('--host codex generates correct output paths', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const skillMd = path.join(AGENTS_DIR, skill.codexName, 'SKILL.md');
|
||
expect(fs.existsSync(skillMd)).toBe(true);
|
||
}
|
||
});
|
||
|
||
test('root gstack bundle has OpenAI metadata for Codex skill browsing', () => {
|
||
const rootMetadata = path.join(ROOT, 'agents', 'openai.yaml');
|
||
expect(fs.existsSync(rootMetadata)).toBe(true);
|
||
const content = fs.readFileSync(rootMetadata, 'utf-8');
|
||
expect(content).toContain('display_name: "gstack"');
|
||
expect(content).toContain('allow_implicit_invocation: true');
|
||
});
|
||
|
||
test('externalSkillName mapping: root is gstack, others are gstack-{dir}', () => {
|
||
// Root → gstack
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack', 'SKILL.md'))).toBe(true);
|
||
// Subdirectories → gstack-{dir}
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'))).toBe(true);
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'))).toBe(true);
|
||
// gstack-upgrade doesn't double-prefix
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-upgrade', 'SKILL.md'))).toBe(true);
|
||
// No double-prefix: gstack-gstack-upgrade must NOT exist
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-gstack-upgrade', 'SKILL.md'))).toBe(false);
|
||
});
|
||
|
||
test('Codex frontmatter has ONLY name + description', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
expect(content.startsWith('---\n')).toBe(true);
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
expect(fmEnd).toBeGreaterThan(0);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
// Must have name and description
|
||
expect(frontmatter).toContain('name:');
|
||
expect(frontmatter).toContain('description:');
|
||
// Must NOT have allowed-tools, version, or hooks
|
||
expect(frontmatter).not.toContain('allowed-tools:');
|
||
expect(frontmatter).not.toContain('version:');
|
||
expect(frontmatter).not.toContain('hooks:');
|
||
}
|
||
});
|
||
|
||
test('all Codex skills have agents/openai.yaml metadata', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const metadata = path.join(AGENTS_DIR, skill.codexName, 'agents', 'openai.yaml');
|
||
expect(fs.existsSync(metadata)).toBe(true);
|
||
const content = fs.readFileSync(metadata, 'utf-8');
|
||
expect(content).toContain(`display_name: "${skill.codexName}"`);
|
||
expect(content).toContain('short_description:');
|
||
expect(content).toContain('allow_implicit_invocation: true');
|
||
}
|
||
});
|
||
|
||
test('no .claude/skills/ in Codex output', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('.claude/skills');
|
||
}
|
||
});
|
||
|
||
test('no ~/.claude/ runtime paths in Codex output', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
// Outside prompts may explicitly forbid reading Claude's skill directory.
|
||
// Every executable/runtime path must still use the selected host root.
|
||
const withoutBoundary = content.replace(/^.*do not read or execute any files under[^\n]*$/gim, '');
|
||
expect(withoutBoundary).not.toContain('~/.claude/');
|
||
}
|
||
});
|
||
|
||
test('/codex skill excluded from Codex output', () => {
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-codex', 'SKILL.md'))).toBe(false);
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-codex'))).toBe(false);
|
||
});
|
||
|
||
test('Codex output includes the renamed Claude Code skill and restricted runner', () => {
|
||
const content = readExternalSkillUnion(EXTERNAL_OUT, '.agents', 'claude-code');
|
||
expect(content).toContain('name: claude-code');
|
||
expect(content).toContain('gstack-claude-code');
|
||
expect(content).toContain('--access none');
|
||
expect(content).toContain('--access read-only');
|
||
expect(content).toContain('--timeout-ms');
|
||
expect(content).toContain('claude-session-id');
|
||
expect(content).not.toContain('$HOME/.claude/.credentials.json');
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-claude'))).toBe(false);
|
||
});
|
||
|
||
test('Codex ship and review preserve outside review through Claude Code', () => {
|
||
for (const skill of ['ship', 'review']) {
|
||
const content = readExternalSkillUnion(EXTERNAL_OUT, '.agents', skill);
|
||
expect(content).not.toMatch(/codex\s+(?:exec|review)\s/);
|
||
expect(content).toContain('gstack-claude-code');
|
||
expect(content).toContain('codex_reviews');
|
||
expect(content).toContain('adversarial-review');
|
||
}
|
||
});
|
||
|
||
test('--host codex --dry-run freshness', () => {
|
||
// Dry-run against the out-dir render: determinism/idempotency check
|
||
// (regenerating produces the same bytes the module-level render did).
|
||
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
// Every Codex skill should be FRESH
|
||
for (const skill of CODEX_SKILLS) {
|
||
expect(output).toContain(`FRESH: .agents/skills/${skill.codexName}/SKILL.md`);
|
||
}
|
||
expect(output).not.toContain('STALE');
|
||
});
|
||
|
||
test('--host agents alias produces same output as --host codex', () => {
|
||
const codexResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
const agentsResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'agents', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
expect(codexResult.exitCode).toBe(0);
|
||
expect(agentsResult.exitCode).toBe(0);
|
||
// Both should produce the same output (same FRESH lines)
|
||
expect(codexResult.stdout.toString()).toBe(agentsResult.stdout.toString());
|
||
});
|
||
|
||
test('multiline descriptions preserved in Codex output', () => {
|
||
// office-hours has a multiline description — verify it survives the frontmatter transform
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-office-hours', 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
// Description should span multiple lines (block scalar)
|
||
const descLines = frontmatter.split('\n').filter(l => l.startsWith(' '));
|
||
expect(descLines.length).toBeGreaterThan(1);
|
||
// Verify key phrases survived
|
||
expect(frontmatter).toContain('YC Office Hours');
|
||
});
|
||
|
||
test('hook skills have safety prose and no hooks: in frontmatter', () => {
|
||
const HOOK_SKILLS = ['gstack-careful', 'gstack-freeze', 'gstack-guard'];
|
||
for (const skillName of HOOK_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skillName, 'SKILL.md'), 'utf-8');
|
||
// Must have safety advisory prose
|
||
expect(content).toContain('Safety Advisory');
|
||
// Must NOT have hooks: in frontmatter
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
expect(frontmatter).not.toContain('hooks:');
|
||
}
|
||
});
|
||
|
||
test('all Codex SKILL.md files have auto-generated header', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('AUTO-GENERATED from SKILL.md.tmpl');
|
||
expect(content).toContain('Regenerate: bun run gen:skill-docs');
|
||
}
|
||
});
|
||
|
||
test('Codex preamble resolves runtime assets from repo-local or global gstack roots', () => {
|
||
// Check a skill that has a preamble (review is a good candidate)
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('GSTACK_ROOT');
|
||
// C1: one shared root resolution — repo-local first, then CODEX_HOME's global root.
|
||
expect(content).toContain('_r=$(git rev-parse --show-toplevel 2>/dev/null)/.agents/skills/gstack');
|
||
expect(content).toContain('_r=${CODEX_HOME:-~/.codex}/skills/gstack');
|
||
// Phase 1/2: config reads moved into gstack-skill-start — the fence itself
|
||
// is the bin asset the preamble must resolve through $GSTACK_BIN, and the
|
||
// question-preference runtime call still resolves the same way.
|
||
expect(content).toContain('$GSTACK_BIN/gstack-skill-start');
|
||
expect(content).toContain('$GSTACK_BIN/gstack-question-preference');
|
||
// The upgrade-skill doc reference moved into the script's upgrade-flow
|
||
// block, resolved $0-relative ($_ROOT_DIR) — host-neutral by construction,
|
||
// so the Codex render no longer needs its own copy.
|
||
expect(SKILL_START_SCRIPT).toContain('$_ROOT_DIR/gstack-upgrade/SKILL.md');
|
||
expect(SKILL_START_SCRIPT).toContain('_ROOT_DIR=$(dirname "$_BIN")');
|
||
expect(content).not.toContain('~/.codex/skills/gstack/bin/gstack-config get telemetry');
|
||
});
|
||
|
||
// ─── Path rewriting regression tests ─────────────────────────
|
||
|
||
test('sidecar paths resolve through $GSTACK_ROOT (not gstack-review/)', () => {
|
||
// #2518: templates now anchor sidecars at the installed skill root
|
||
// (~/.claude/skills/gstack/review/...), which the codex path rewrite turns
|
||
// into $GSTACK_ROOT/review/... — resolved by the preamble against the
|
||
// repo-local .agents root or the global install. The old repo-relative
|
||
// form (.claude/skills/review/) only resolved inside gstack's own checkout.
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('$GSTACK_ROOT/review/checklist.md');
|
||
// design-checklist.md is now referenced via Review Army specialist (Claude only, stripped for Codex)
|
||
// Wrong: must NOT reference gstack-review/checklist.md (file doesn't exist there)
|
||
expect(content).not.toContain('.agents/skills/gstack-review/checklist.md');
|
||
});
|
||
|
||
test('sidecar paths in ship skill point to gstack/review/ for pre-landing review', () => {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
// Ship references the review checklist in its pre-landing review step
|
||
if (content.includes('checklist.md')) {
|
||
expect(content).toContain('.agents/skills/gstack/review/');
|
||
expect(content).not.toContain('.agents/skills/gstack-review/checklist');
|
||
}
|
||
});
|
||
|
||
test('greptile-triage sidecar path is correct', () => {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
if (content.includes('greptile-triage')) {
|
||
expect(content).toContain('$GSTACK_ROOT/review/greptile-triage.md');
|
||
expect(content).not.toContain('.agents/skills/gstack-review/greptile-triage');
|
||
}
|
||
});
|
||
|
||
test('all four path rewrite rules produce correct output', () => {
|
||
// Test each of the 4 path rewrite rules individually
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
|
||
// Rule 1: ~/.claude/skills/gstack → $GSTACK_ROOT
|
||
expect(content).not.toContain('~/.claude/skills/gstack');
|
||
expect(content).toContain('$GSTACK_ROOT');
|
||
|
||
// Rule 2: .claude/skills/gstack → .agents/skills/gstack
|
||
expect(content).not.toContain('.claude/skills/gstack');
|
||
|
||
// Rule 3: .claude/skills/review → .agents/skills/gstack/review
|
||
expect(content).not.toContain('.claude/skills/review');
|
||
|
||
// Rule 4: .claude/skills → .agents/skills (catch-all)
|
||
expect(content).not.toContain('.claude/skills');
|
||
});
|
||
|
||
test('path rewrite rules apply to all Codex skills with sidecar references', () => {
|
||
// Verify across ALL generated skills, not just review
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
// No skill should reference Claude paths
|
||
expect(content).not.toContain('~/.claude/skills');
|
||
expect(content).not.toContain('.claude/skills');
|
||
if (content.includes('gstack-config') || content.includes('gstack-update-check') || content.includes('gstack-telemetry-log')) {
|
||
expect(content).toContain('$GSTACK_ROOT');
|
||
}
|
||
// If a skill references checklist.md, it must use the correct sidecar path
|
||
if (content.includes('checklist.md') && !content.includes('design-checklist.md')) {
|
||
expect(content).not.toContain('gstack-review/checklist.md');
|
||
}
|
||
}
|
||
});
|
||
|
||
// ─── Claude output regression guard ─────────────────────────
|
||
|
||
test('Claude output uses installed-root review paths (#2518)', () => {
|
||
// Codex changes must NOT affect Claude output; the Claude form is the
|
||
// installed-root anchor, not the old repo-relative path that only
|
||
// resolved inside gstack's own checkout.
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('~/.claude/skills/gstack/review/checklist.md');
|
||
expect(content).toContain('~/.claude/skills/gstack');
|
||
// Must NOT contain Codex HOST paths. `~/.codex/sessions/` is exempt: the
|
||
// timeout-wrapper guidance documents the Codex CLI's own rollout-log
|
||
// location (a user-facing CLI path, same class as ~/.codex/logs/ in the
|
||
// codex skill), not the gstack Codex host install path.
|
||
// `~/.codex/config.toml` is the same user-facing class: the shared
|
||
// codexPreflight's model_unusable branch (#2477) points at the CLI's own
|
||
// config file, where the rejected `model =` pin lives.
|
||
expect(content).not.toContain('.agents/skills');
|
||
expect(
|
||
content
|
||
.replaceAll('~/.codex/sessions/', '')
|
||
.replaceAll('~/.codex/config.toml', ''),
|
||
).not.toContain('~/.codex/');
|
||
});
|
||
|
||
test('Claude output unchanged: ship skill still uses .claude/skills/ paths', () => {
|
||
const content = readShipUnion();
|
||
expect(content).toContain('~/.claude/skills/gstack');
|
||
expect(content).not.toContain('.agents/skills');
|
||
// ~/.codex/sessions/ is the Codex CLI's rollout-log path (user-facing),
|
||
// documented by the adversarial-pass timeout guidance; ~/.codex/config.toml
|
||
// is the CLI's own config file (model_unusable guidance, #2477) — see the
|
||
// review test above.
|
||
expect(
|
||
content
|
||
.replaceAll('~/.codex/sessions/', '')
|
||
.replaceAll('~/.codex/config.toml', ''),
|
||
).not.toContain('~/.codex/');
|
||
});
|
||
|
||
test('Claude output unchanged: all Claude skills have zero Codex paths', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
// pair-agent legitimately documents how Codex agents store credentials.
|
||
// codex + autoplan document the Codex CLI auth file (~/.codex/auth.json)
|
||
// and log path (~/.codex/logs/) — those are user-facing Codex CLI paths,
|
||
// not the gstack Codex host install path. ~/.codex/sessions/ (rollout
|
||
// logs, referenced by the review/ship timeout guidance) and
|
||
// ~/.codex/config.toml (the model_unusable guidance in the shared
|
||
// codexPreflight, #2477) are the same user-facing class, so they are
|
||
// scrubbed before the ban.
|
||
if (skill.dir !== 'pair-agent' && skill.dir !== 'codex' && skill.dir !== 'autoplan') {
|
||
expect(
|
||
content
|
||
.replaceAll('~/.codex/sessions/', '')
|
||
.replaceAll('~/.codex/config.toml', ''),
|
||
).not.toContain('~/.codex/');
|
||
}
|
||
// gstack-upgrade legitimately references .agents/skills for cross-platform detection
|
||
if (skill.dir !== 'gstack-upgrade') {
|
||
expect(content).not.toContain('.agents/skills');
|
||
}
|
||
}
|
||
});
|
||
|
||
// ─── Design outside voices: Codex host guard ─────────────────
|
||
|
||
test('codex host retains design outside voices through Claude Code', () => {
|
||
const codexContent = readExternalSkillUnion(EXTERNAL_OUT, '.agents', 'design-review');
|
||
expect(codexContent).toContain('Design Outside Voices');
|
||
expect(codexContent).toContain('gstack-claude-code');
|
||
});
|
||
|
||
test('codex host does not include Codex design block in ship', () => {
|
||
const codexContent = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
expect(codexContent).not.toContain('Codex design voice');
|
||
});
|
||
|
||
// ─── Explicit --model override wins over the host default ────
|
||
// Without --model the codex host renders its defaultModel (gpt) — pinned by
|
||
// the golden test. This pins the OTHER direction through the real CLI:
|
||
// `./setup --host codex --model <id>` depends on it. The override renders
|
||
// into its OWN out-dir, so no restore pass is needed — the host-default
|
||
// render (EXTERNAL_OUT) is untouched and asserted directly.
|
||
test('explicit --model overrides the codex host default', () => {
|
||
const overrideOut = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-model-override-'));
|
||
try {
|
||
const override = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--model', 'claude', '--out-dir', overrideOut], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
expect(override.exitCode).toBe(0);
|
||
const content = fs.readFileSync(path.join(overrideOut, '.agents', 'skills', 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Model-Specific Behavioral Patch (claude)');
|
||
// The overlay now travels as --model into gstack-skill-start, which
|
||
// echoes MODEL_OVERLAY at runtime.
|
||
expect(content).toContain('--model "claude"');
|
||
} finally {
|
||
fs.rmSync(overrideOut, { recursive: true, force: true });
|
||
}
|
||
// Host-default direction: the untouched EXTERNAL_OUT render carries gpt.
|
||
const hostDefault = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
expect(hostDefault).toContain('Model-Specific Behavioral Patch (gpt)');
|
||
expect(hostDefault).toContain('--model "gpt"');
|
||
});
|
||
});
|
||
|
||
// ─── Factory generation tests ────────────────────────────────
|
||
|
||
describe('Factory generation (--host factory)', () => {
|
||
// .factory/ is gitignored — read the module-level out-dir render
|
||
// (--host all covers factory) instead of regenerating in place.
|
||
const FACTORY_DIR = path.join(EXTERNAL_OUT, '.factory', 'skills');
|
||
|
||
// Fresh out-dir → the vendored-dev-mode symlink loop can never occur, so
|
||
// every template renders (see the Codex discovery note above).
|
||
const FACTORY_SKILLS = (() => {
|
||
const skills: Array<{ dir: string; factoryName: string }> = [];
|
||
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
||
skills.push({ dir: '.', factoryName: 'gstack' });
|
||
}
|
||
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
||
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||
if (entry.name === 'codex') continue;
|
||
if (!fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) continue;
|
||
const factoryName = entry.name.startsWith('gstack-') ? entry.name : `gstack-${entry.name}`;
|
||
skills.push({ dir: entry.name, factoryName });
|
||
}
|
||
return skills;
|
||
})();
|
||
|
||
test('--host factory generates correct output paths', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const skillMd = path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md');
|
||
expect(fs.existsSync(skillMd)).toBe(true);
|
||
}
|
||
});
|
||
|
||
test('Factory frontmatter has name + description + user-invocable', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
expect(frontmatter).toContain('name:');
|
||
expect(frontmatter).toContain('description:');
|
||
expect(frontmatter).toContain('user-invocable: true');
|
||
expect(frontmatter).not.toContain('allowed-tools:');
|
||
expect(frontmatter).not.toContain('preamble-tier:');
|
||
expect(frontmatter).not.toContain('sensitive:');
|
||
}
|
||
});
|
||
|
||
test('sensitive skills have disable-model-invocation', () => {
|
||
const SENSITIVE = ['gstack-ship', 'gstack-land-and-deploy', 'gstack-guard', 'gstack-careful', 'gstack-freeze', 'gstack-unfreeze'];
|
||
for (const name of SENSITIVE) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, name, 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
expect(frontmatter).toContain('disable-model-invocation: true');
|
||
}
|
||
});
|
||
|
||
test('non-sensitive skills lack disable-model-invocation', () => {
|
||
const NON_SENSITIVE = ['gstack-qa', 'gstack-review', 'gstack-investigate', 'gstack-browse'];
|
||
for (const name of NON_SENSITIVE) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, name, 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
expect(frontmatter).not.toContain('disable-model-invocation');
|
||
}
|
||
});
|
||
|
||
test('no .claude/skills/ in Factory output', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('.claude/skills');
|
||
}
|
||
});
|
||
|
||
test('no ~/.claude/skills/ paths in Factory output', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
||
// ~/.claude/skills should be rewritten, but ~/.claude/plans is legitimate
|
||
// (plan directory lookup) and ~/.claude/ in codex prompts is intentional
|
||
expect(content).not.toContain('~/.claude/skills');
|
||
}
|
||
});
|
||
|
||
test('Factory installs both outside-review wrapper skills', () => {
|
||
for (const skill of ['gstack-codex', 'gstack-claude-code']) {
|
||
expect(fs.existsSync(path.join(FACTORY_DIR, skill, 'SKILL.md'))).toBe(true);
|
||
}
|
||
});
|
||
|
||
test('Factory keeps Codex integration blocks', () => {
|
||
// Factory users CAN use Codex second opinions (codex exec is a standalone binary)
|
||
const shipContent = fs.readFileSync(path.join(FACTORY_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
expect(shipContent).toContain('codex');
|
||
});
|
||
|
||
test('no agents/openai.yaml in Factory output', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const yamlPath = path.join(FACTORY_DIR, skill.factoryName, 'agents', 'openai.yaml');
|
||
expect(fs.existsSync(yamlPath)).toBe(false);
|
||
}
|
||
});
|
||
|
||
test('--host droid alias works', () => {
|
||
const factoryResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
|
||
});
|
||
const droidResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'droid', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
|
||
});
|
||
expect(factoryResult.exitCode).toBe(0);
|
||
expect(droidResult.exitCode).toBe(0);
|
||
expect(factoryResult.stdout.toString()).toBe(droidResult.stdout.toString());
|
||
});
|
||
|
||
test('--host factory --dry-run freshness', () => {
|
||
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
|
||
});
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
for (const skill of FACTORY_SKILLS) {
|
||
expect(output).toContain(`FRESH: .factory/skills/${skill.factoryName}/SKILL.md`);
|
||
}
|
||
expect(output).not.toContain('STALE');
|
||
});
|
||
|
||
test('Factory preamble uses .factory paths', () => {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('GSTACK_ROOT');
|
||
expect(content).toContain('_r=$(git rev-parse --show-toplevel 2>/dev/null)/.factory/skills/gstack');
|
||
expect(content).toContain('$GSTACK_BIN/gstack-config');
|
||
});
|
||
});
|
||
|
||
// ─── Parameterized host smoke tests (config-driven) ─────────
|
||
|
||
import { ALL_HOST_CONFIGS, getExternalHosts } from '../hosts/index';
|
||
import { sliceBetween } from './helpers/skill-fixture';
|
||
|
||
describe('Parameterized host smoke tests', () => {
|
||
// Every external host was rendered up front by the module-level
|
||
// `--host all --out-dir EXTERNAL_OUT` render, so the per-host `--dry-run`
|
||
// freshness checks are deterministic: they compare a regeneration against
|
||
// that render — an idempotency/determinism check that catches
|
||
// non-deterministic gen without ever writing (or depending on) the live
|
||
// gitignored host dirs. The tracked-claude freshness test
|
||
// (`generated files are fresh`) runs earlier and is unaffected.
|
||
for (const hostConfig of getExternalHosts()) {
|
||
describe(`${hostConfig.displayName} (--host ${hostConfig.name})`, () => {
|
||
const hostDir = path.join(EXTERNAL_OUT, hostConfig.hostSubdir, 'skills');
|
||
|
||
test('generates output that exists on disk', () => {
|
||
// The module-level --host all render must have produced this host's tree.
|
||
expect(fs.existsSync(hostDir)).toBe(true);
|
||
const skills = fs.readdirSync(hostDir).filter(d =>
|
||
fs.existsSync(path.join(hostDir, d, 'SKILL.md'))
|
||
);
|
||
expect(skills.length).toBeGreaterThan(0);
|
||
});
|
||
|
||
test('no .claude/skills path leakage outside repo-root sidecar symlinks', () => {
|
||
expect(fs.existsSync(hostDir)).toBe(true);
|
||
const skills = fs.readdirSync(hostDir);
|
||
for (const skill of skills) {
|
||
// Dev installs may mount the repo root at host/skills/gstack as a runtime
|
||
// sidecar. The generator skips that symlink loop, so leakage checks should too.
|
||
if (isRepoRootSymlink(path.join(hostDir, skill))) continue;
|
||
const skillMd = path.join(hostDir, skill, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
const leaks = externalHostPathLeaks(content);
|
||
if (leaks.length > 0) {
|
||
throw new Error(`${skill}: .claude/skills leakage:\n${leaks.slice(0, 3).join('\n')}`);
|
||
}
|
||
}
|
||
});
|
||
|
||
test('frontmatter has name and description', () => {
|
||
expect(fs.existsSync(hostDir)).toBe(true);
|
||
const skills = fs.readdirSync(hostDir);
|
||
for (const skill of skills) {
|
||
const skillMd = path.join(hostDir, skill, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
expect(content).toMatch(/^---\n/);
|
||
expect(content).toMatch(/^name:\s/m);
|
||
expect(content).toMatch(/^description:\s/m);
|
||
}
|
||
});
|
||
|
||
test('--dry-run freshness check passes', () => {
|
||
const result = Bun.spawnSync(
|
||
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', hostConfig.name, '--dry-run', '--out-dir', EXTERNAL_OUT],
|
||
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 }
|
||
);
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
expect(output).not.toContain('STALE');
|
||
});
|
||
|
||
if (hostConfig.generation.skipSkills?.includes('codex')) {
|
||
test('/codex skill excluded', () => {
|
||
expect(fs.existsSync(path.join(hostDir, 'gstack-codex', 'SKILL.md'))).toBe(false);
|
||
});
|
||
}
|
||
});
|
||
}
|
||
});
|
||
|
||
// ─── --host all tests ────────────────────────────────────────
|
||
|
||
describe('--host all', () => {
|
||
// Same determinism guard as the parameterized block: the module-level
|
||
// `--host all --out-dir EXTERNAL_OUT` render is the comparison baseline, so
|
||
// this dry-run reports FRESH regardless of live-tree state — and proves the
|
||
// claude host plus every external host regenerate deterministically.
|
||
test('--host all generates for all registered hosts', () => {
|
||
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
|
||
});
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
// All hosts should appear in output
|
||
expect(output).toContain('FRESH: SKILL.md'); // claude
|
||
for (const hostConfig of getExternalHosts()) {
|
||
expect(output).toContain(`FRESH: ${hostConfig.hostSubdir}/skills/`);
|
||
}
|
||
});
|
||
});
|
||
|
||
// ─── Setup script validation ─────────────────────────────────
|
||
// These tests verify the setup script's install layout matches
|
||
// what the generator produces — catching the bug where setup
|
||
// installed Claude-format source dirs for Codex users.
|
||
|
||
describe('setup script validation', () => {
|
||
const setupContent = fs.readFileSync(path.join(ROOT, 'setup'), 'utf-8');
|
||
|
||
test('setup has separate link functions for Claude and Codex', () => {
|
||
expect(setupContent).toContain('link_claude_skill_dirs');
|
||
expect(setupContent).toContain('link_codex_skill_dirs');
|
||
// Old unified function must not exist
|
||
expect(setupContent).not.toMatch(/^link_skill_dirs\(\)/m);
|
||
});
|
||
|
||
test('Claude install uses link_claude_skill_dirs', () => {
|
||
// The Claude install section (section 4) should use the Claude function
|
||
const claudeSection = setupContent.slice(
|
||
setupContent.indexOf('# 4. Install for Claude'),
|
||
setupContent.indexOf('# 5. Install for Codex')
|
||
);
|
||
expect(claudeSection).toContain('link_claude_skill_dirs');
|
||
expect(claudeSection).not.toContain('link_codex_skill_dirs');
|
||
});
|
||
|
||
test('Codex install uses link_codex_skill_dirs', () => {
|
||
// The Codex install section (section 5) should use the Codex function
|
||
// End marker: the next numbered section header (a marker that doesn't
|
||
// exist slices to EOF and the assertion reads unrelated sections).
|
||
const codexSection = setupContent.slice(
|
||
setupContent.indexOf('# 5. Install for Codex'),
|
||
setupContent.indexOf('# 6. Install for Kiro')
|
||
);
|
||
expect(setupContent.indexOf('# 6. Install for Kiro')).toBeGreaterThan(-1);
|
||
expect(codexSection).toContain('create_codex_runtime_root');
|
||
expect(codexSection).toContain('link_codex_skill_dirs');
|
||
expect(codexSection).not.toContain('link_claude_skill_dirs');
|
||
expect(codexSection).not.toContain('_link_or_copy "$GSTACK_DIR" "$CODEX_GSTACK"');
|
||
});
|
||
|
||
test('Codex install prefers repo-local .agents/skills when setup runs from there', () => {
|
||
expect(setupContent).toContain('SKILLS_PARENT_BASENAME');
|
||
expect(setupContent).toContain('CODEX_REPO_LOCAL=0');
|
||
expect(setupContent).toContain('[ "$SKILLS_PARENT_BASENAME" = ".agents" ]');
|
||
expect(setupContent).toContain('CODEX_REPO_LOCAL=1');
|
||
expect(setupContent).toContain('CODEX_SKILLS="$INSTALL_SKILLS_DIR"');
|
||
});
|
||
|
||
test('setup separates install path from source path for symlinked repo-local installs', () => {
|
||
expect(setupContent).toContain('INSTALL_GSTACK_DIR=');
|
||
expect(setupContent).toContain('SOURCE_GSTACK_DIR=');
|
||
expect(setupContent).toContain('INSTALL_SKILLS_DIR=');
|
||
expect(setupContent).toContain('CODEX_GSTACK="$INSTALL_GSTACK_DIR"');
|
||
// Links come from the source checkout, or its per-install render (#1882).
|
||
expect(setupContent).toContain('link_codex_skill_dirs "${_CODEX_RENDER_ROOT:-$SOURCE_GSTACK_DIR}" "$CODEX_SKILLS"');
|
||
});
|
||
|
||
test('Codex installs always create sidecar runtime assets for the real skill target', () => {
|
||
expect(setupContent).toContain('if [ "$INSTALL_CODEX" -eq 1 ]; then');
|
||
expect(setupContent).toContain('create_agents_sidecar "$SOURCE_GSTACK_DIR"');
|
||
});
|
||
|
||
test('link_codex_skill_dirs reads from .agents/skills/', () => {
|
||
// The Codex link function must reference .agents/skills for generated Codex skills
|
||
const fnStart = setupContent.indexOf('link_codex_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('.agents/skills');
|
||
expect(fnBody).toContain('gstack*');
|
||
});
|
||
|
||
test('link_claude_skill_dirs creates real directories with absolute SKILL.md symlinks', () => {
|
||
// Claude links should be real directories with absolute SKILL.md symlinks
|
||
// to ensure Claude Code discovers them as top-level skills (not nested under gstack/)
|
||
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('mkdir -p "$target"');
|
||
// v1.36.0.0: routes through _link_or_copy helper for Windows fallback (cp on MSYS2/Git Bash).
|
||
// v1.67 (#2569): the source is render-aware — canonical SKILL.md, or the
|
||
// rendered :user variant from ${GSTACK_HOME}/render/claude when present.
|
||
expect(fnBody).toContain('_skill_md_src="$gstack_dir/$dir_name/SKILL.md"');
|
||
expect(fnBody).toContain('_link_or_copy "$_skill_md_src" "$target/SKILL.md"');
|
||
});
|
||
|
||
// REGRESSION: cleanup functions must handle both old symlinks AND new real-directory pattern
|
||
test('cleanup functions handle real directories with symlinked SKILL.md', () => {
|
||
// cleanup_old_claude_symlinks must detect and remove real dirs with SKILL.md symlinks
|
||
const cleanupOldStart = setupContent.indexOf('cleanup_old_claude_symlinks()');
|
||
const cleanupOldEnd = setupContent.indexOf('}', setupContent.indexOf('cleaned up old', cleanupOldStart));
|
||
const cleanupOldBody = setupContent.slice(cleanupOldStart, cleanupOldEnd);
|
||
expect(cleanupOldBody).toContain('-d "$old_target"');
|
||
expect(cleanupOldBody).toContain('-L "$old_target/SKILL.md"');
|
||
expect(cleanupOldBody).toContain('rm -rf "$old_target"');
|
||
|
||
// cleanup_prefixed_claude_symlinks must also handle the new pattern
|
||
const cleanupPrefixedStart = setupContent.indexOf('cleanup_prefixed_claude_symlinks()');
|
||
const cleanupPrefixedEnd = setupContent.indexOf('}', setupContent.indexOf('cleaned up prefixed', cleanupPrefixedStart));
|
||
const cleanupPrefixedBody = setupContent.slice(cleanupPrefixedStart, cleanupPrefixedEnd);
|
||
expect(cleanupPrefixedBody).toContain('-d "$prefixed_target"');
|
||
expect(cleanupPrefixedBody).toContain('-L "$prefixed_target/SKILL.md"');
|
||
expect(cleanupPrefixedBody).toContain('rm -rf "$prefixed_target"');
|
||
});
|
||
|
||
// REGRESSION: link function must upgrade old directory symlinks
|
||
test('link_claude_skill_dirs removes old directory symlinks before creating real dirs', () => {
|
||
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
// Must check for and remove old symlinks before mkdir
|
||
expect(fnBody).toContain('if [ -L "$target" ]');
|
||
expect(fnBody).toContain('rm -f "$target"');
|
||
});
|
||
|
||
test('setup links root gstack skill through a thin Claude wrapper alias', () => {
|
||
const fnStart = setupContent.indexOf('link_claude_root_skill_alias()');
|
||
const fnEnd = setupContent.indexOf('# ─── Helper: remove old unprefixed Claude skill entries', fnStart);
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('_gstack-command');
|
||
// #2511: the alias must be a rewritten COPY (unique frontmatter name),
|
||
// never a verbatim symlink of the canonical SKILL.md.
|
||
expect(fnBody).toContain('_install_alias_skill_md "$gstack_dir/SKILL.md" "$target" "_gstack-command"');
|
||
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir/SKILL.md"');
|
||
|
||
const claudeSection = setupContent.slice(
|
||
setupContent.indexOf('# 4. Install for Claude'),
|
||
setupContent.indexOf('# 5. Install for Codex')
|
||
);
|
||
expect(claudeSection).toContain('link_claude_root_skill_alias "$SOURCE_GSTACK_DIR" "$INSTALL_SKILLS_DIR"');
|
||
});
|
||
|
||
test('setup supports --host auto|claude|codex|kiro|opencode|cursor; slate is informational', () => {
|
||
expect(setupContent).toContain('--host');
|
||
// #2361: slate moved OUT of the install accept-list (it was accepted but
|
||
// never dispatched — a silent exit-0 no-op) into an informational arm.
|
||
expect(setupContent).toContain('claude|codex|kiro|factory|opencode|cursor|copilot|auto');
|
||
expect(setupContent).toMatch(/^ {2}slate\)/m);
|
||
});
|
||
|
||
test('auto mode detects claude, codex, kiro, and opencode binaries', () => {
|
||
expect(setupContent).toContain('command -v claude');
|
||
expect(setupContent).toContain('command -v codex');
|
||
expect(setupContent).toContain('command -v kiro-cli');
|
||
expect(setupContent).toContain('command -v opencode');
|
||
});
|
||
|
||
// T1: Sidecar skip guard — prevents .agents/skills/gstack from being linked as a skill
|
||
test('link_codex_skill_dirs skips the gstack sidecar directory', () => {
|
||
const fnStart = setupContent.indexOf('link_codex_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('[ "$skill_name" = "gstack" ] && continue');
|
||
});
|
||
|
||
// T2: Dynamic $GSTACK_ROOT paths in generated Codex preambles
|
||
test('generated Codex preambles use dynamic GSTACK_ROOT paths', () => {
|
||
// Read the module-level out-dir render (always present).
|
||
const codexSkillDir = path.join(EXTERNAL_OUT, '.agents', 'skills', 'gstack-ship');
|
||
const content = fs.readFileSync(path.join(codexSkillDir, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('GSTACK_ROOT=');
|
||
expect(content).toContain('$GSTACK_BIN/');
|
||
});
|
||
|
||
test('setup installs Kiro-native output and runtime assets', () => {
|
||
expect(setupContent).toContain('INSTALL_KIRO=');
|
||
expect(setupContent).toContain('kiro-cli');
|
||
expect(setupContent).toContain('KIRO_SKILLS=');
|
||
expect(setupContent).toContain('KIRO_SKILLS="$HOME/.kiro/skills"');
|
||
expect(setupContent).toContain('KIRO_DIR="$SOURCE_GSTACK_DIR/.kiro/skills"');
|
||
expect(setupContent).toContain('gen:skill-docs --host kiro');
|
||
expect(setupContent).toContain('$KIRO_GSTACK/lib');
|
||
});
|
||
|
||
test('setup supports --host opencode with install section and OpenCode skill path vars', () => {
|
||
expect(setupContent).toContain('INSTALL_OPENCODE=');
|
||
expect(setupContent).toContain('OPENCODE_SKILLS="$HOME/.config/opencode/skills"');
|
||
expect(setupContent).toContain('OPENCODE_GSTACK="$OPENCODE_SKILLS/gstack"');
|
||
});
|
||
|
||
// --host cursor full install slice (#1358, PR #2547 by @szsunyuan re-derived)
|
||
test('auto mode detects Cursor via binary or ~/.cursor directory', () => {
|
||
expect(setupContent).toContain('command -v cursor');
|
||
expect(setupContent).toContain('[ -d "$HOME/.cursor" ] && INSTALL_CURSOR=1');
|
||
});
|
||
|
||
test('setup supports --host cursor with install section and Cursor skill path vars', () => {
|
||
expect(setupContent).toContain('INSTALL_CURSOR=');
|
||
expect(setupContent).toContain('CURSOR_SKILLS="$HOME/.cursor/skills"');
|
||
expect(setupContent).toContain('CURSOR_GSTACK="$CURSOR_SKILLS/gstack"');
|
||
expect(setupContent).toContain('create_cursor_runtime_root');
|
||
expect(setupContent).toContain('create_cursor_sidecar');
|
||
expect(setupContent).toContain('link_cursor_skill_dirs');
|
||
expect(setupContent).toContain('gstack ready (cursor).');
|
||
});
|
||
|
||
test('create_cursor_runtime_root exposes only Cursor runtime assets', () => {
|
||
const fnStart = setupContent.indexOf('create_cursor_runtime_root()');
|
||
const fnEnd = setupContent.indexOf('create_cursor_sidecar()', fnStart);
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('gstack/SKILL.md');
|
||
expect(fnBody).toContain('browse/dist');
|
||
expect(fnBody).toContain('browse/bin');
|
||
expect(fnBody).toContain('gstack-upgrade/SKILL.md');
|
||
expect(fnBody).toContain('checklist.md');
|
||
expect(fnBody).toContain('TODOS-format.md');
|
||
// bin scripts import ../lib — the two must travel together.
|
||
expect(fnBody).toContain('$cursor_gstack/lib');
|
||
expect(fnBody).not.toContain('design-checklist.md');
|
||
expect(fnBody).not.toContain('greptile-triage.md');
|
||
expect(fnBody).not.toContain('review/specialists');
|
||
expect(fnBody).not.toContain('qa/templates');
|
||
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir" "$cursor_gstack"');
|
||
});
|
||
|
||
test('create_cursor_sidecar plants runtime assets without wiping generated SKILL.md', () => {
|
||
const fnStart = setupContent.indexOf('create_cursor_sidecar()');
|
||
const fnEnd = setupContent.indexOf('link_cursor_skill_dirs()', fnStart);
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('.cursor/skills/gstack');
|
||
expect(fnBody).toContain('bin');
|
||
expect(fnBody).toContain('browse/dist');
|
||
expect(fnBody).toContain('browse/bin');
|
||
expect(fnBody).toContain('ETHOS.md');
|
||
expect(fnBody).not.toContain('rm -rf');
|
||
});
|
||
|
||
test('link_cursor_skill_dirs skips the gstack runtime root directory', () => {
|
||
const fnStart = setupContent.indexOf('link_cursor_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('[ "$skill_name" = "gstack" ] && continue');
|
||
// #2444-aware guard: Windows bypass, else only replace symlink-or-missing.
|
||
expect(fnBody).toContain('[ "$IS_WINDOWS" -eq 1 ] || [ -L "$target" ] || [ ! -e "$target" ]');
|
||
});
|
||
|
||
// #2142 deleted existing ~/.cursor/skills/<name> dirs with `rm -rf "$target"`
|
||
// before relinking. That can wipe unowned Cursor skills. Only replace a
|
||
// symlink or a missing path; never the whole skills directory.
|
||
test('link_cursor_skill_dirs does not delete unowned Cursor skill directories', () => {
|
||
const fnStart = setupContent.indexOf('link_cursor_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).not.toContain('rm -rf "$target"');
|
||
expect(fnBody).not.toContain('rm -rf "$skills_dir"');
|
||
expect(setupContent).not.toContain('rm -rf "$CURSOR_SKILLS"');
|
||
});
|
||
|
||
test('Cursor install links generated skills before planting the sidecar', () => {
|
||
const cursorInstall = setupContent.slice(
|
||
setupContent.indexOf('# 6d. Install for Cursor'),
|
||
setupContent.indexOf('# 7. Create .agents/ sidecar'),
|
||
);
|
||
const linkCall = cursorInstall.indexOf('link_cursor_skill_dirs "$SOURCE_GSTACK_DIR"');
|
||
const sidecarCall = cursorInstall.indexOf('create_cursor_sidecar "$SOURCE_GSTACK_DIR"');
|
||
expect(linkCall).toBeGreaterThan(-1);
|
||
expect(sidecarCall).toBeGreaterThan(-1);
|
||
expect(linkCall).toBeLessThan(sidecarCall);
|
||
});
|
||
|
||
test('setup installs OpenCode skills into a nested gstack runtime root', () => {
|
||
expect(setupContent).toContain('create_opencode_runtime_root');
|
||
expect(setupContent).toContain('.opencode/skills');
|
||
expect(setupContent).toContain('review/specialists');
|
||
expect(setupContent).toContain('qa/templates');
|
||
expect(setupContent).toContain('qa/references');
|
||
expect(setupContent).toContain('dx-hall-of-fame.md');
|
||
expect(setupContent).toContain('$opencode_gstack/lib');
|
||
});
|
||
|
||
test('create_agents_sidecar links runtime assets', () => {
|
||
// Sidecar must link bin with its shared lib modules, plus browse, review, qa
|
||
const fnStart = setupContent.indexOf('create_agents_sidecar()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('bin');
|
||
expect(fnBody).toContain('lib');
|
||
expect(fnBody).toContain('browse');
|
||
expect(fnBody).toContain('review');
|
||
expect(fnBody).toContain('qa');
|
||
});
|
||
|
||
test('create_codex_runtime_root exposes only runtime assets', () => {
|
||
const fnStart = setupContent.indexOf('create_codex_runtime_root()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', setupContent.indexOf('review/', fnStart)));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('gstack/SKILL.md');
|
||
expect(fnBody).toContain('$codex_gstack/lib');
|
||
expect(fnBody).toContain('browse/dist');
|
||
expect(fnBody).toContain('browse/bin');
|
||
expect(fnBody).toContain('gstack-upgrade/SKILL.md');
|
||
// Review runtime assets (individual files, not the whole dir)
|
||
expect(fnBody).toContain('checklist.md');
|
||
expect(fnBody).toContain('design-checklist.md');
|
||
expect(fnBody).toContain('greptile-triage.md');
|
||
expect(fnBody).toContain('TODOS-format.md');
|
||
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir" "$codex_gstack"');
|
||
});
|
||
|
||
test('create_factory_runtime_root links shared lib modules beside bin', () => {
|
||
const fnStart = setupContent.indexOf('create_factory_runtime_root()');
|
||
const fnEnd = setupContent.indexOf('create_opencode_runtime_root()', fnStart);
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('$factory_gstack/bin');
|
||
expect(fnBody).toContain('$factory_gstack/lib');
|
||
});
|
||
|
||
test('direct Codex installs are migrated out of ~/.codex/skills/gstack', () => {
|
||
expect(setupContent).toContain('migrate_direct_codex_install');
|
||
expect(setupContent).toContain('$HOME/.gstack/repos/gstack');
|
||
expect(setupContent).toContain('avoid duplicate skill discovery');
|
||
});
|
||
|
||
// --- Symlink prefix tests (PR #503) ---
|
||
|
||
test('link_claude_skill_dirs applies gstack- prefix by default', () => {
|
||
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('SKILL_PREFIX');
|
||
expect(fnBody).toContain('link_name="gstack-$skill_name"');
|
||
});
|
||
|
||
test('link_claude_skill_dirs preserves already-prefixed dirs', () => {
|
||
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
// gstack-* dirs should keep their name (e.g., gstack-upgrade stays gstack-upgrade)
|
||
expect(fnBody).toContain('gstack-*) link_name="$skill_name"');
|
||
});
|
||
|
||
test('setup supports --no-prefix flag', () => {
|
||
expect(setupContent).toContain('--no-prefix');
|
||
expect(setupContent).toContain('SKILL_PREFIX=0');
|
||
});
|
||
|
||
test('cleanup_old_claude_symlinks removes only gstack-pointing symlinks', () => {
|
||
expect(setupContent).toContain('cleanup_old_claude_symlinks');
|
||
const fnStart = setupContent.indexOf('cleanup_old_claude_symlinks()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('removed[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
// Should check readlink before removing
|
||
expect(fnBody).toContain('readlink');
|
||
expect(fnBody).toContain('gstack/*');
|
||
// Should skip already-prefixed dirs
|
||
expect(fnBody).toContain('gstack-*) continue');
|
||
});
|
||
|
||
test('cleanup runs before link when prefix is enabled', () => {
|
||
// In the Claude install section, cleanup should happen before linking
|
||
const claudeInstallSection = setupContent.slice(
|
||
setupContent.indexOf('INSTALL_CLAUDE'),
|
||
setupContent.lastIndexOf('link_claude_skill_dirs')
|
||
);
|
||
expect(claudeInstallSection).toContain('cleanup_old_claude_symlinks');
|
||
});
|
||
|
||
// --- Persistent config + interactive prompt tests ---
|
||
|
||
test('setup reads skill_prefix from config', () => {
|
||
expect(setupContent).toContain('get skill_prefix');
|
||
expect(setupContent).toContain('GSTACK_CONFIG');
|
||
});
|
||
|
||
test('setup supports --prefix flag', () => {
|
||
expect(setupContent).toContain('--prefix)');
|
||
expect(setupContent).toContain('SKILL_PREFIX=1; SKILL_PREFIX_FLAG=1');
|
||
});
|
||
|
||
test('--prefix and --no-prefix persist to config', () => {
|
||
expect(setupContent).toContain('set skill_prefix');
|
||
});
|
||
|
||
test('interactive prompt shows when no config', () => {
|
||
expect(setupContent).toContain('Short names');
|
||
expect(setupContent).toContain('Namespaced');
|
||
expect(setupContent).toContain('Choice [1/2]');
|
||
});
|
||
|
||
test('non-TTY defaults to flat names', () => {
|
||
// Should check if stdin is a TTY before prompting
|
||
expect(setupContent).toContain('-t 0');
|
||
});
|
||
|
||
test('cleanup_prefixed_claude_symlinks exists and uses readlink', () => {
|
||
expect(setupContent).toContain('cleanup_prefixed_claude_symlinks');
|
||
const fnStart = setupContent.indexOf('cleanup_prefixed_claude_symlinks()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('removed[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('readlink');
|
||
expect(fnBody).toContain('gstack-$skill_name');
|
||
});
|
||
|
||
test('reverse cleanup runs before link when prefix is disabled', () => {
|
||
const claudeInstallSection = setupContent.slice(
|
||
setupContent.indexOf('INSTALL_CLAUDE'),
|
||
setupContent.lastIndexOf('link_claude_skill_dirs')
|
||
);
|
||
expect(claudeInstallSection).toContain('cleanup_prefixed_claude_symlinks');
|
||
});
|
||
|
||
test('welcome message references SKILL_PREFIX', () => {
|
||
// gstack-upgrade is always called gstack-upgrade (it's the actual dir name)
|
||
// but the welcome section should exist near the prefix logic
|
||
expect(setupContent).toContain('Run /gstack-upgrade anytime');
|
||
});
|
||
});
|
||
|
||
describe('discover-skills hidden directory filtering', () => {
|
||
test('discoverTemplates skips dot-prefixed directories', () => {
|
||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-discover-'));
|
||
try {
|
||
// Create a hidden dir with a template (should be excluded)
|
||
fs.mkdirSync(path.join(tmpDir, '.hidden'), { recursive: true });
|
||
fs.writeFileSync(path.join(tmpDir, '.hidden', 'SKILL.md.tmpl'), '---\nname: evil\n---\ntest');
|
||
// Create a visible dir with a template (should be included)
|
||
fs.mkdirSync(path.join(tmpDir, 'visible'), { recursive: true });
|
||
fs.writeFileSync(path.join(tmpDir, 'visible', 'SKILL.md.tmpl'), '---\nname: good\n---\ntest');
|
||
|
||
const { discoverTemplates } = require('../scripts/discover-skills');
|
||
const results = discoverTemplates(tmpDir);
|
||
const dirs = results.map((r: { tmpl: string }) => r.tmpl);
|
||
|
||
expect(dirs).toContain('visible/SKILL.md.tmpl');
|
||
expect(dirs).not.toContain('.hidden/SKILL.md.tmpl');
|
||
} finally {
|
||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('telemetry', () => {
|
||
test('telemetry start block lives in gstack-skill-start; render notes the handoff keys', () => {
|
||
// The start-block bash moved into the script (Phase 1): it reads the
|
||
// config, mints the session identity, and echoes the STATUS keys.
|
||
expect(SKILL_START_SCRIPT).toContain('_TEL_START=$(date +%s)');
|
||
expect(SKILL_START_SCRIPT).toContain('_SESSION_ID=');
|
||
expect(SKILL_START_SCRIPT).toContain('echo "TELEMETRY:');
|
||
expect(SKILL_START_SCRIPT).toContain('echo "TEL_PROMPTED:');
|
||
expect(SKILL_START_SCRIPT).toMatch(/gstack-config" get telemetry/);
|
||
// The render must tell the model to carry SESSION_ID/TEL_START to skill end.
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('SESSION_ID');
|
||
expect(content).toContain('TEL_START');
|
||
});
|
||
|
||
test('telemetry opt-in prompt lives in gstack-skill-start (marker-gated emit)', () => {
|
||
// Token-reduction Phase 2: the one-time consent prompt left the renders
|
||
// (absence pinned by test/onboarding-moved-literals.test.ts); the script
|
||
// gates it on the marker files and emits it as a GSTACK_INSTRUCTION block
|
||
// with all three config-set outcomes and the ack INSIDE the block.
|
||
expect(SKILL_START_SCRIPT).toContain(
|
||
'if [ "$_TEL_PROMPTED" = "no" ] && [ "$_LAKE_SEEN" = "yes" ]; then',
|
||
);
|
||
expect(SKILL_START_SCRIPT).toContain('_emit_block telemetry-prompt');
|
||
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry community');
|
||
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry anonymous');
|
||
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry off');
|
||
expect(SKILL_START_SCRIPT).toContain('touch "$_GH/.telemetry-prompted"');
|
||
});
|
||
|
||
test('generated SKILL.md contains telemetry epilogue (one gstack-skill-end call)', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Telemetry (run last)');
|
||
expect(content).toContain('gstack-skill-end --skill "gstack" --outcome OUTCOME');
|
||
expect(content).toContain('--tel-start "TEL_START"');
|
||
expect(content).toContain('PLAN MODE EXCEPTION');
|
||
// The duration math + remote-log dispatch moved into gstack-skill-end.
|
||
expect(SKILL_END_SCRIPT).toContain('_TEL_END');
|
||
expect(SKILL_END_SCRIPT).toContain('_TEL_DUR');
|
||
expect(SKILL_END_SCRIPT).toContain('SKILL_NAME');
|
||
expect(SKILL_END_SCRIPT).toContain('OUTCOME');
|
||
expect(SKILL_END_SCRIPT).toContain('gstack-telemetry-log');
|
||
});
|
||
|
||
test('pending marker handling lives in the scripts', () => {
|
||
// gstack-skill-start finalizes stale markers; gstack-skill-end clears the
|
||
// session's own marker.
|
||
expect(SKILL_START_SCRIPT).toContain("-name '.pending-*'");
|
||
expect(SKILL_START_SCRIPT).toContain('_pending_finalize');
|
||
expect(SKILL_END_SCRIPT).toContain('.pending-$SESSION_ID');
|
||
});
|
||
|
||
test('telemetry blocks appear in all skill files that use PREAMBLE', () => {
|
||
const skills = ['qa', 'ship', 'review', 'plan-ceo-review', 'plan-eng-review', 'retro'];
|
||
for (const skill of skills) {
|
||
const skillPath = path.join(ROOT, skill, 'SKILL.md');
|
||
if (fs.existsSync(skillPath)) {
|
||
const content = fs.readFileSync(skillPath, 'utf-8');
|
||
expect(content).toContain('Telemetry (run last)');
|
||
expect(content).toContain(`gstack-skill-end --skill "${skill}"`);
|
||
expect(content).toContain('--tel-start "TEL_START"');
|
||
}
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('community fixes wave', () => {
|
||
// Helper to get all generated SKILL.md files
|
||
function getAllSkillMds(): Array<{ name: string; content: string }> {
|
||
const results: Array<{ name: string; content: string }> = [];
|
||
const rootPath = path.join(ROOT, 'SKILL.md');
|
||
if (fs.existsSync(rootPath)) {
|
||
results.push({ name: 'root', content: fs.readFileSync(rootPath, 'utf-8') });
|
||
}
|
||
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
||
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||
const skillPath = path.join(ROOT, entry.name, 'SKILL.md');
|
||
if (fs.existsSync(skillPath)) {
|
||
results.push({ name: entry.name, content: fs.readFileSync(skillPath, 'utf-8') });
|
||
}
|
||
}
|
||
return results;
|
||
}
|
||
|
||
// #594 — Discoverability: every SKILL.md.tmpl description contains "gstack"
|
||
test('every SKILL.md.tmpl description contains "gstack"', () => {
|
||
for (const skill of ALL_SKILLS) {
|
||
const tmplPath = skill.dir === '.' ? path.join(ROOT, 'SKILL.md.tmpl') : path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
||
const content = fs.readFileSync(tmplPath, 'utf-8');
|
||
const desc = extractDescription(content);
|
||
expect(desc.toLowerCase()).toContain('gstack');
|
||
}
|
||
});
|
||
|
||
// #594 — Discoverability: first line of each description is under 120 chars
|
||
test('every SKILL.md.tmpl description first line is under 120 chars', () => {
|
||
for (const skill of ALL_SKILLS) {
|
||
const tmplPath = skill.dir === '.' ? path.join(ROOT, 'SKILL.md.tmpl') : path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
||
const content = fs.readFileSync(tmplPath, 'utf-8');
|
||
const desc = extractDescription(content);
|
||
const firstLine = desc.split('\n')[0];
|
||
expect(firstLine.length).toBeLessThanOrEqual(120);
|
||
}
|
||
});
|
||
|
||
// #573 — Feature signals: ship/SKILL.md contains feature signal detection
|
||
test('ship/SKILL.md contains feature signal detection in Step 4', () => {
|
||
const content = readShipUnion();
|
||
expect(content.toLowerCase()).toContain('feature signal');
|
||
});
|
||
|
||
// #510 — Context warnings: no SKILL.md contains "running low on context"
|
||
test('no generated SKILL.md contains "running low on context"', () => {
|
||
const skills = getAllSkillMds();
|
||
for (const { name, content } of skills) {
|
||
expect(content).not.toContain('running low on context');
|
||
}
|
||
});
|
||
|
||
// #510 — Context warnings: plan-eng-review has explicit anti-warning
|
||
test('plan-eng-review/SKILL.md explicitly forbids preemptive context warnings', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
expect(content.toLowerCase()).toContain('do not preemptively warn');
|
||
});
|
||
|
||
// #474 — Safety Net: no SKILL.md uses find with -delete
|
||
test('no generated SKILL.md contains find with -delete flag', () => {
|
||
const skills = getAllSkillMds();
|
||
for (const { name, content } of skills) {
|
||
// Match find commands that use -delete (but not prose mentioning the word "delete")
|
||
const lines = content.split('\n');
|
||
for (const line of lines) {
|
||
if (line.includes('find ') && line.includes('-delete')) {
|
||
throw new Error(`${name}/SKILL.md contains find with -delete: ${line.trim()}`);
|
||
}
|
||
}
|
||
}
|
||
});
|
||
|
||
// #467 — Telemetry: preamble JSONL writes are gated by telemetry setting
|
||
test('preamble JSONL writes are inside telemetry conditional', () => {
|
||
const preamble = fs.readFileSync(path.join(ROOT, 'scripts/resolvers/preamble.ts'), 'utf-8');
|
||
// Find all skill-usage.jsonl write lines
|
||
const lines = preamble.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
if (lines[i].includes('skill-usage.jsonl') && lines[i].includes('>>')) {
|
||
// Look backwards for a telemetry conditional within 5 lines
|
||
let foundConditional = false;
|
||
for (let j = i - 1; j >= Math.max(0, i - 5); j--) {
|
||
if (lines[j].includes('_TEL') && lines[j].includes('off')) {
|
||
foundConditional = true;
|
||
break;
|
||
}
|
||
}
|
||
expect(foundConditional).toBe(true);
|
||
}
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('codex commands must not use inline $(git rev-parse --show-toplevel) for cwd', () => {
|
||
// Regression test: inline $(git rev-parse --show-toplevel) in codex exec -C
|
||
// or codex review without cd evaluates in whatever cwd the background shell
|
||
// inherits, which may be a different project in Conductor workspaces.
|
||
// The fix is to resolve _REPO_ROOT eagerly at the top of each bash block.
|
||
|
||
// Scan all source files that could contain codex commands
|
||
// Use Bun.Glob to avoid ELOOP from .claude/skills/gstack symlink back to ROOT
|
||
const tmplGlob = new Bun.Glob('**/*.tmpl');
|
||
const sourceFiles = [
|
||
...Array.from(tmplGlob.scanSync({ cwd: ROOT, followSymlinks: false })),
|
||
...fs.readdirSync(path.join(ROOT, 'scripts/resolvers'))
|
||
.filter(f => f.endsWith('.ts'))
|
||
.map(f => `scripts/resolvers/${f}`),
|
||
'scripts/gen-skill-docs.ts',
|
||
];
|
||
|
||
test('no codex exec command uses inline $(git rev-parse --show-toplevel) in -C flag', () => {
|
||
const violations: string[] = [];
|
||
for (const rel of sourceFiles) {
|
||
const abs = path.join(ROOT, rel);
|
||
if (!fs.existsSync(abs)) continue;
|
||
const content = fs.readFileSync(abs, 'utf-8');
|
||
const lines = content.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
const line = lines[i];
|
||
if (line.includes('codex exec') && line.includes('-C') && line.includes('$(git rev-parse --show-toplevel)')) {
|
||
violations.push(`${rel}:${i + 1}`);
|
||
}
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
|
||
test('no generated SKILL.md has codex exec with inline $(git rev-parse --show-toplevel) in -C flag', () => {
|
||
const violations: string[] = [];
|
||
const skillMdGlob = new Bun.Glob('**/SKILL.md');
|
||
const skillMdFiles = Array.from(skillMdGlob.scanSync({ cwd: ROOT, followSymlinks: false }));
|
||
for (const rel of skillMdFiles) {
|
||
const abs = path.join(ROOT, rel);
|
||
if (!fs.existsSync(abs)) continue;
|
||
const content = fs.readFileSync(abs, 'utf-8');
|
||
const lines = content.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
const line = lines[i];
|
||
if (line.includes('codex exec') && line.includes('-C') && line.includes('$(git rev-parse --show-toplevel)')) {
|
||
violations.push(`${rel}:${i + 1}`);
|
||
}
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
|
||
test('codex review commands must be preceded by cd "$_REPO_ROOT" (no -C support)', () => {
|
||
// codex review does not support -C, so the pattern must be:
|
||
// _REPO_ROOT=$(git rev-parse --show-toplevel) || { ... }
|
||
// cd "$_REPO_ROOT"
|
||
// codex review ...
|
||
// NOT: codex review ... with inline $(git rev-parse --show-toplevel)
|
||
const allFiles = [
|
||
...Array.from(tmplGlob.scanSync({ cwd: ROOT, followSymlinks: false })),
|
||
...Array.from(new Bun.Glob('**/SKILL.md').scanSync({ cwd: ROOT, followSymlinks: false })),
|
||
...fs.readdirSync(path.join(ROOT, 'scripts/resolvers'))
|
||
.filter(f => f.endsWith('.ts'))
|
||
.map(f => `scripts/resolvers/${f}`),
|
||
'scripts/gen-skill-docs.ts',
|
||
];
|
||
const violations: string[] = [];
|
||
for (const rel of allFiles) {
|
||
const abs = path.join(ROOT, rel);
|
||
if (!fs.existsSync(abs)) continue;
|
||
const content = fs.readFileSync(abs, 'utf-8');
|
||
const lines = content.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
const line = lines[i];
|
||
// Skip non-executable lines (markdown table cells, prose references)
|
||
if (line.includes('|') && line.includes('`/codex review`')) continue;
|
||
if (line.includes('`codex review`')) continue;
|
||
// Check for codex review with inline $(git rev-parse)
|
||
if (line.includes('codex review') && line.includes('$(git rev-parse --show-toplevel)')) {
|
||
violations.push(`${rel}:${i + 1} — inline git rev-parse in codex review`);
|
||
}
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
|
||
test('codex review commands take their scope from a flag, never from prompt text', () => {
|
||
// `codex review` scope comes ONLY from --base/--commit/--uncommitted. The
|
||
// positional [PROMPT] is mutually exclusive with all three (#1428, #1479),
|
||
// and a prompt-only `codex review` silently falls back to the *uncommitted
|
||
// working-tree* scope (`git status --short; git diff`) — so describing the
|
||
// diff range in prompt text produces a confident review of the wrong
|
||
// changes, with no error. Both halves are pinned here:
|
||
// (a) every `codex review` invocation carries a scope flag, and
|
||
// (b) no invocation puts a positional prompt in front of that flag.
|
||
//
|
||
// This does NOT apply to `codex exec`, which is agentic and really does run
|
||
// the git command it's told to — the adversarial pass legitimately scopes
|
||
// itself in prompt text.
|
||
const checkedFiles = [
|
||
'scripts/resolvers/review-dashboard.ts',
|
||
'scripts/resolvers/plan-gates.ts',
|
||
'scripts/resolvers/spec-review.ts',
|
||
'scripts/resolvers/outside-voice-steps.ts',
|
||
'scripts/resolvers/review-scope.ts',
|
||
'review/SKILL.md',
|
||
'ship/SKILL.md',
|
||
'codex/SKILL.md.tmpl',
|
||
'codex/SKILL.md',
|
||
// codex's scoped invocations moved into the carved review-mode section
|
||
// (T9) — keep sweeping both the .tmpl source and the generated section.
|
||
'codex/sections/review-mode.md.tmpl',
|
||
'codex/sections/review-mode.md',
|
||
];
|
||
|
||
const violations: string[] = [];
|
||
for (const rel of checkedFiles) {
|
||
// ship's codex/adversarial command moved into sections/adversarial.md (T9 carve).
|
||
const content = rel === 'ship/SKILL.md' ? readShipUnion() : fs.readFileSync(path.join(ROOT, rel), 'utf-8');
|
||
const lines = content.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
const line = lines[i];
|
||
// Only inspect real shell invocations, not prose mentioning the command.
|
||
if (line.includes('`codex review`')) continue;
|
||
const match = line.match(/(?:^|[;&|]\s*|\s)codex\s+review\b(.*)$/);
|
||
if (!match) continue;
|
||
const rest = match[1];
|
||
const scopeFlag = /--base\b|--commit\b|--uncommitted\b/;
|
||
if (!scopeFlag.test(rest)) {
|
||
// A quoted prompt with no scope flag is the silent-wrong-scope bug.
|
||
if (/^\s*["'$]/.test(rest)) {
|
||
violations.push(`${rel}:${i + 1} — prompt-only codex review (falls back to working-tree scope)`);
|
||
}
|
||
continue;
|
||
}
|
||
const beforeFlag = rest.split(scopeFlag)[0].trim();
|
||
if (/^["'$]|^--\s*["']/.test(beforeFlag)) {
|
||
violations.push(`${rel}:${i + 1} — positional prompt passed alongside a scope flag`);
|
||
}
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
});
|
||
|
||
// ─── Learnings + Confidence Resolver Tests ─────────────────────
|
||
|
||
describe('LEARNINGS_SEARCH resolver', () => {
|
||
const SEARCH_SKILLS = ['review', 'ship', 'plan-eng-review', 'investigate', 'office-hours', 'plan-ceo-review'];
|
||
|
||
for (const skill of SEARCH_SKILLS) {
|
||
test(`${skill} generated SKILL.md contains learnings search`, () => {
|
||
const content = readSkillUnion(skill); // ship: moved to sections/plan-completion.md
|
||
expect(content).toContain('Prior Learnings');
|
||
expect(content).toContain('gstack-learnings-search');
|
||
});
|
||
}
|
||
|
||
test('learnings search includes cross-project config check', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('cross_project_learnings');
|
||
expect(content).toContain('--cross-project');
|
||
});
|
||
|
||
test('learnings search includes AskUserQuestion for first-time cross-project opt-in', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Enable cross-project learnings');
|
||
expect(content).toContain('project-scoped only');
|
||
});
|
||
|
||
test('Eng clarifies its full brief without changing any host options or other skills', async () => {
|
||
const { generateLearningsSearch } = await import('../scripts/resolvers/learnings');
|
||
const { ALL_HOST_CONFIGS } = await import('../hosts');
|
||
const { HOST_PATHS } = await import('../scripts/resolvers/types');
|
||
const fullBrief = 'Build a full decision brief from these facts and options using the preamble format, then ask and wait:';
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const ctx = { skillName: 'plan-eng-review', tmplPath: 'plan-eng-review/SKILL.md.tmpl', host: host.name, paths: HOST_PATHS[host.name]! };
|
||
const eng = generateLearningsSearch(ctx);
|
||
const standard = generateLearningsSearch({ ...ctx, skillName: 'review' });
|
||
expect(generateLearningsSearch({ ...ctx, skillName: 'plan-ceo-review' })).toBe(standard);
|
||
if (host.learningsMode === 'basic') {
|
||
expect(eng).toBe(standard);
|
||
expect(eng).not.toContain('CROSS_PROJECT');
|
||
} else {
|
||
expect(eng).toContain(fullBrief);
|
||
expect(eng.replace(fullBrief, 'Use AskUserQuestion:')).toBe(standard);
|
||
expect(eng).toContain('A) Enable cross-project learnings (recommended)');
|
||
expect(eng).toContain('B) Keep learnings project-scoped only');
|
||
expect(eng).toContain('gstack-config set cross_project_learnings true');
|
||
expect(eng).toContain('gstack-config set cross_project_learnings false');
|
||
}
|
||
}
|
||
});
|
||
|
||
test('learnings search mentions prior learning applied display format', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Prior learning applied');
|
||
});
|
||
});
|
||
|
||
describe('LEARNINGS_LOG resolver', () => {
|
||
const LOG_SKILLS = ['review', 'retro', 'investigate'];
|
||
|
||
for (const skill of LOG_SKILLS) {
|
||
test(`${skill} generated SKILL.md contains learnings log`, () => {
|
||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Capture Learnings');
|
||
expect(content).toContain('gstack-learnings-log');
|
||
});
|
||
}
|
||
|
||
test('learnings log documents all type values', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
for (const type of ['pattern', 'pitfall', 'preference', 'architecture', 'tool']) {
|
||
expect(content).toContain(type);
|
||
}
|
||
});
|
||
|
||
test('learnings log documents all source values', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
for (const source of ['observed', 'user-stated', 'inferred', 'cross-model']) {
|
||
expect(content).toContain(source);
|
||
}
|
||
});
|
||
|
||
test('learnings log includes files field for staleness detection', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('"files"');
|
||
expect(content).toContain('staleness detection');
|
||
});
|
||
});
|
||
|
||
describe('CONFIDENCE_CALIBRATION resolver', () => {
|
||
// CSO owns a distinct evidence rubric; the shared numerical confidence
|
||
// resolver would conflict with that contract and its private startup.
|
||
const CONFIDENCE_SKILLS = ['review', 'ship', 'plan-eng-review'];
|
||
|
||
for (const skill of CONFIDENCE_SKILLS) {
|
||
test(`${skill} generated SKILL.md contains confidence calibration`, () => {
|
||
const content = readSkillUnion(skill); // ship: moved to sections/review-army.md
|
||
expect(content).toContain('Confidence Calibration');
|
||
expect(content).toContain(skill === 'review' ? 'score every finding (1-10)' : 'confidence score');
|
||
});
|
||
}
|
||
|
||
test('confidence calibration includes scoring rubric with all tiers', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('9-10');
|
||
expect(content).toContain('7-8');
|
||
expect(content).toContain('5-6');
|
||
expect(content).toContain('3-4');
|
||
expect(content).toContain('1-2');
|
||
});
|
||
|
||
test('confidence calibration includes display rules', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Show normally');
|
||
expect(content).toContain('Suppress from main report');
|
||
});
|
||
|
||
test('confidence calibration includes finding format example', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('[CRITICAL] (confidence:');
|
||
expect(content).toContain('SQL injection');
|
||
});
|
||
|
||
test('confidence calibration includes calibration learning feedback loop', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
const flat = content.replace(/\s+/g, ' ');
|
||
expect(flat).toContain('Calibration learning');
|
||
expect(flat).toContain('log the corrected pattern as a learning');
|
||
});
|
||
|
||
test('skills without confidence calibration do NOT contain it', () => {
|
||
// office-hours and retro do NOT use confidence calibration
|
||
for (const skill of ['office-hours', 'retro']) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('## Confidence Calibration');
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('gen-skill-docs prefix warning (#620/#578)', () => {
|
||
const { execSync } = require('child_process');
|
||
|
||
test('warns about skill_prefix when config has prefix=true', () => {
|
||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-prefix-warn-'));
|
||
try {
|
||
// Create a fake ~/.gstack/config.yaml with skill_prefix: true
|
||
const fakeHome = tmpDir;
|
||
const fakeGstack = path.join(fakeHome, '.gstack');
|
||
fs.mkdirSync(fakeGstack, { recursive: true });
|
||
fs.writeFileSync(path.join(fakeGstack, 'config.yaml'), 'skill_prefix: true\n');
|
||
|
||
// Render into an out-dir under the fixture (the warning fires on any
|
||
// non-dry-run generation) so the live tree is never rewritten.
|
||
const outDir = path.join(tmpDir, 'out');
|
||
const output = execSync(`bun run scripts/gen-skill-docs.ts --out-dir "${outDir}"`, {
|
||
cwd: ROOT,
|
||
env: { ...process.env, HOME: fakeHome },
|
||
encoding: 'utf-8',
|
||
timeout: 30000,
|
||
});
|
||
expect(output).toContain('skill_prefix is true');
|
||
expect(output).toContain('gstack-relink');
|
||
} finally {
|
||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||
}
|
||
});
|
||
|
||
test('no warning when skill_prefix is false or absent', () => {
|
||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-prefix-warn-'));
|
||
try {
|
||
const fakeHome = tmpDir;
|
||
const fakeGstack = path.join(fakeHome, '.gstack');
|
||
fs.mkdirSync(fakeGstack, { recursive: true });
|
||
fs.writeFileSync(path.join(fakeGstack, 'config.yaml'), 'skill_prefix: false\n');
|
||
|
||
const outDir = path.join(tmpDir, 'out');
|
||
const output = execSync(`bun run scripts/gen-skill-docs.ts --out-dir "${outDir}"`, {
|
||
cwd: ROOT,
|
||
env: { ...process.env, HOME: fakeHome },
|
||
encoding: 'utf-8',
|
||
timeout: 30000,
|
||
});
|
||
expect(output).not.toContain('skill_prefix is true');
|
||
} finally {
|
||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('voice-triggers processing', () => {
|
||
const { extractVoiceTriggers, processVoiceTriggers } = require('../scripts/gen-skill-docs') as {
|
||
extractVoiceTriggers: (content: string) => string[];
|
||
processVoiceTriggers: (content: string) => string;
|
||
};
|
||
|
||
test('extractVoiceTriggers parses valid YAML list', () => {
|
||
const content = `---\nname: cso\ndescription: |\n Security audit.\nvoice-triggers:\n - "see-so"\n - "security review"\n---\nBody`;
|
||
const triggers = extractVoiceTriggers(content);
|
||
expect(triggers).toEqual(['see-so', 'security review']);
|
||
});
|
||
|
||
test('extractVoiceTriggers returns [] when no field present', () => {
|
||
const content = `---\nname: qa\ndescription: |\n QA testing.\n---\nBody`;
|
||
expect(extractVoiceTriggers(content)).toEqual([]);
|
||
});
|
||
|
||
test('processVoiceTriggers appends voice triggers to description', () => {
|
||
const content = `---\nname: cso\ndescription: |\n Security audit. (gstack)\nvoice-triggers:\n - "see-so"\n - "security review"\n---\nBody`;
|
||
const result = processVoiceTriggers(content);
|
||
expect(result).toContain('Voice triggers (speech-to-text aliases): "see-so", "security review".');
|
||
});
|
||
|
||
test('processVoiceTriggers strips voice-triggers field from output', () => {
|
||
const content = `---\nname: cso\ndescription: |\n Security audit. (gstack)\nvoice-triggers:\n - "see-so"\n---\nBody`;
|
||
const result = processVoiceTriggers(content);
|
||
expect(result).not.toContain('voice-triggers:');
|
||
});
|
||
|
||
test('processVoiceTriggers returns content unchanged when no voice-triggers', () => {
|
||
const content = `---\nname: qa\ndescription: |\n QA testing.\n---\nBody`;
|
||
expect(processVoiceTriggers(content)).toBe(content);
|
||
});
|
||
|
||
test('generated CSO SKILL.md contains voice triggers in description', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('"see-so"');
|
||
expect(content).toContain('Voice triggers (speech-to-text aliases):');
|
||
});
|
||
|
||
test('generated CSO SKILL.md does NOT contain raw voice-triggers field', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(0, fmEnd);
|
||
expect(frontmatter).not.toContain('voice-triggers:');
|
||
});
|
||
|
||
test('generated Claude CSO skill preauthorizes only the trusted launcher', () => {
|
||
const expected = [
|
||
'Bash(~/.claude/skills/gstack/bin/gstack-cso-launcher *)',
|
||
'Bash(~/.claude/skills/gstack/bin/gstack-cso-launcher.exe *)',
|
||
];
|
||
for (const file of ['cso/SKILL.md.tmpl', 'cso/SKILL.md']) {
|
||
const content = fs.readFileSync(path.join(ROOT, file), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = Bun.YAML.parse(content.slice(4, fmEnd)) as Record<string, unknown>;
|
||
expect(frontmatter['allowed-tools'], file).toEqual(expected);
|
||
}
|
||
});
|
||
|
||
test('generated CSO host variants retain challenge fallback and host-containment disclosure', () => {
|
||
const claude = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
||
const codex = fs.readFileSync(path.join(EXTERNAL_OUT, '.agents', 'skills', 'gstack-cso', 'SKILL.md'), 'utf-8');
|
||
for (const content of [claude, codex]) {
|
||
expect(content).toContain('sequential challenge; independent agent unavailable');
|
||
expectMentions(content, [['does not','containment','sandbox']], 'content');
|
||
expectMentions(content, [['do not','independent','reviewer']], 'content');
|
||
}
|
||
});
|
||
|
||
// Gen-time-only keys: interactive + benefits-from are read from the .tmpl by
|
||
// buildContext; the generated copy has no reader (the host reads name/
|
||
// description/allowed-tools/hooks; gbrain: is runtime-read and NOT stripped).
|
||
// Pin the strip so a stripFields refactor can't silently re-add the always-on
|
||
// frontmatter weight — mirrors the voice-triggers pins above.
|
||
test('generated SKILL.md strips gen-time-only keys the .tmpl still declares', () => {
|
||
const tmpl = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf-8');
|
||
const tmplFm = tmpl.slice(0, tmpl.indexOf('\n---', 4));
|
||
expect(tmplFm).toContain('interactive:');
|
||
expect(tmplFm).toContain('benefits-from:');
|
||
|
||
const generated = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||
const genFm = generated.slice(0, generated.indexOf('\n---', 4));
|
||
expect(genFm).not.toContain('interactive:');
|
||
expect(genFm).not.toContain('benefits-from:');
|
||
|
||
// The runtime-read and host-read keys survive the strip.
|
||
const investigate = fs.readFileSync(path.join(ROOT, 'investigate', 'SKILL.md'), 'utf-8');
|
||
const invFm = investigate.slice(0, investigate.indexOf('\n---', 4));
|
||
expect(invFm).toContain('hooks:');
|
||
expect(invFm).toContain('gbrain:');
|
||
});
|
||
});
|
||
|
||
describe('CEO accepted requirement preservation', () => {
|
||
test('HOLD SCOPE includes implementation work needed to meet accepted requirements', () => {
|
||
const main = fs.readFileSync(path.join(ROOT, 'plan-ceo-review/SKILL.md'), 'utf8');
|
||
const hold = main.slice(main.indexOf('**For HOLD SCOPE**'), main.indexOf('**For SCOPE REDUCTION**'));
|
||
expectMentions(hold, [['acceptance criteria', 'in scope']], 'HOLD SCOPE');
|
||
});
|
||
|
||
test('loaded sections keep requirement conflicts unresolved until an authorized decision', () => {
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-ceo-review/sections/review-sections.md'), 'utf8');
|
||
const policy = section.slice(section.indexOf('**Preserve accepted requirements.**'), section.indexOf('### Section 1:')).replace(/\s+/g, ' ');
|
||
expect(policy).toContain('Report gaps and propose remedies, including omitted mechanisms in HOLD SCOPE');
|
||
expectMentions(policy, [['never','guarantee','violation']], 'policy');
|
||
expectMentions(policy, [['do not','documentation','requirements']], 'policy');
|
||
expect(policy).toContain('Changing a requirement needs explicit authority');
|
||
expectMentions(policy, [['cannot','auto-decide','constraints']], 'policy');
|
||
});
|
||
});
|
||
|
||
describe('plan-mode-info resolver (handshake-replacement)', () => {
|
||
const REVIEW_SKILLS = [
|
||
'plan-ceo-review',
|
||
'plan-eng-review',
|
||
'plan-design-review',
|
||
'plan-devex-review',
|
||
];
|
||
|
||
// Header for the vestigial handshake that was removed. If it ever reappears,
|
||
// someone accidentally re-introduced the resolver.
|
||
const HANDSHAKE_MARKER = '## Plan Mode Handshake';
|
||
// Header for the new plan-mode-info section (previously lived at the tail
|
||
// of completion-status.ts; now hoisted to position 1 of the preamble).
|
||
const PLAN_MODE_INFO_MARKER = '## Skill Invocation During Plan Mode';
|
||
|
||
test('vestigial handshake is absent from all generated Claude SKILL.md files', () => {
|
||
// Scan every generated SKILL.md under ROOT (top-level directory per skill).
|
||
// Using fs.readdirSync + filter instead of a glob so we catch any skill
|
||
// that gets added later without updating this list.
|
||
const entries = fs.readdirSync(ROOT, { withFileTypes: true });
|
||
let checked = 0;
|
||
for (const entry of entries) {
|
||
if (!entry.isDirectory()) continue;
|
||
const skillMd = path.join(ROOT, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
expect(content, `handshake marker in ${entry.name}/SKILL.md`).not.toContain(HANDSHAKE_MARKER);
|
||
checked++;
|
||
}
|
||
expect(checked).toBeGreaterThan(0);
|
||
});
|
||
|
||
test('vestigial handshake is absent from non-Claude host outputs', () => {
|
||
// Non-Claude hosts render to hostSubdirs (.agents/, .openclaw/, etc). The
|
||
// plan-mode-info resolver has no host-scoping — all hosts get the new
|
||
// section, none get the old handshake. Scan every candidate host tree in
|
||
// the module-level out-dir render (--host all), which is always present —
|
||
// so the check can no longer silently degrade to a console warning.
|
||
const hostDirs = ['.agents', '.openclaw', '.opencode', '.factory', '.hermes', '.kiro', '.cursor', '.slate'];
|
||
let checked = 0;
|
||
for (const host of hostDirs) {
|
||
const skillsRoot = path.join(EXTERNAL_OUT, host, 'skills');
|
||
if (!fs.existsSync(skillsRoot)) continue;
|
||
const entries = fs.readdirSync(skillsRoot, { withFileTypes: true });
|
||
for (const entry of entries) {
|
||
if (!entry.isDirectory()) continue;
|
||
const skillMd = path.join(skillsRoot, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
expect(content, `handshake marker in ${host}/skills/${entry.name}/SKILL.md`).not.toContain(HANDSHAKE_MARKER);
|
||
checked++;
|
||
}
|
||
}
|
||
expect(checked).toBeGreaterThan(0);
|
||
});
|
||
|
||
test.each(REVIEW_SKILLS)(
|
||
'%s/SKILL.md contains the new plan-mode-info section near the top',
|
||
(skill) => {
|
||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
const idx = content.indexOf(PLAN_MODE_INFO_MARKER);
|
||
expect(idx).toBeGreaterThan(0);
|
||
// Position 1 in preamble composition = within the first ~300 lines.
|
||
// Roughly translates to first ~15KB of text.
|
||
expect(idx).toBeLessThan(15_000);
|
||
},
|
||
);
|
||
|
||
test('plan-mode-info is wired BEFORE generateUpgradeCheck in preamble', () => {
|
||
// Token-reduction Phase 2: generateUpgradeCheck's render output is now
|
||
// ONLY the steady-state PROACTIVE-false + SKILL_PREFIX rules (the
|
||
// UPGRADE_AVAILABLE prose emits from bin/gstack-skill-start at runtime),
|
||
// so those rules are the resolver's order marker.
|
||
const content = fs.readFileSync(
|
||
path.join(ROOT, 'plan-ceo-review', 'SKILL.md'),
|
||
'utf-8',
|
||
);
|
||
const planModeIdx = content.indexOf(PLAN_MODE_INFO_MARKER);
|
||
const upgradeIdx = content.search(/If `PROACTIVE` is `"?false"?`/i);
|
||
expect(planModeIdx).toBeGreaterThan(0);
|
||
expect(upgradeIdx).toBeGreaterThan(0);
|
||
expect(planModeIdx).toBeLessThan(upgradeIdx);
|
||
});
|
||
|
||
test('0D authority and fresh-approval paths precede mode selection', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8').replace(/\s+/g, ' ');
|
||
const approachIdx = content.indexOf('### 0D.');
|
||
const constructIdx = content.indexOf('Build one `currentDecision`', approachIdx);
|
||
const saveIdx = content.indexOf('**Pre-question checkpoint:**', constructIdx);
|
||
const presentIdx = content.indexOf('Ask one row per call', saveIdx);
|
||
const stopOffset = content.slice(presentIdx).search(/stop for the actual answer/i);
|
||
const stopIdx = stopOffset < 0 ? -1 : presentIdx + stopOffset;
|
||
const modeIdx = content.indexOf('### 0E. Mode Selection');
|
||
const preludeIdx = content.indexOf('### 0F');
|
||
const positions = [approachIdx, constructIdx, saveIdx, presentIdx, stopIdx, modeIdx, preludeIdx];
|
||
expect(positions.every(position => position > 0)).toBe(true);
|
||
expect(positions).toEqual([...positions].sort((a, b) => a - b));
|
||
const approach = content.slice(approachIdx, modeIdx);
|
||
expect(approach).toMatch(/Reuse (?:an )?exact approvals?/);
|
||
expect(content).toMatch(/(?:the actual instruction\/answer|Cite actual instructions\/answers) and exact scope/);
|
||
const reopenRule = approach.match(/Reuse exact approvals\. Reopen only for contradictions, changed assumptions or user instructions, never speculation or reviewer agreement/)?.[0];
|
||
expect(reopenRule).toBeDefined();
|
||
expect(reopenRule!).toMatch(/reopen only|requires reopening it/i);
|
||
expect(content.indexOf(reopenRule!)).toBeGreaterThan(approachIdx);
|
||
expect(content.indexOf(reopenRule!)).toBeLessThan(presentIdx);
|
||
const gate = content.slice(stopIdx, modeIdx);
|
||
const startup = content.slice(content.indexOf('### 0C.'), approachIdx);
|
||
expect(startup).toContain('Before 0E, call 0D for unresolved approaches');
|
||
expect(startup).toContain('A) current/requested plan, B) smallest scoped alternative');
|
||
expectMentions(startup, [['no','required','choices']], 'startup');
|
||
expectMentions(approach, [['not','selection','returns']], 'approach');
|
||
expectTokens(approach, ['**Mode change**'], 'approach');
|
||
const modeChange = content.slice(content.indexOf('**Mode change:**'), preludeIdx);
|
||
expectMentions(modeChange, [['ask','four-mode','answered']], 'modeChange');
|
||
expectMentions(gate, [['do not','calling','return']], 'gate');
|
||
expect(gate).toContain('even for a lone option');
|
||
expect(approach).toContain('A recommendation is not approval');
|
||
expectMentions(approach, [['without','recomposing','unchanged']], 'approach');
|
||
expect(gate).toMatch(/(?:Record its reference|Save the answer reference) and scope in Exact approval and scope/);
|
||
expect(gate).toMatch(/update Status,? and amend only (?:what it authorizes|authorized work)/);
|
||
expectMentions(gate, [['before','storage','another']], 'gate');
|
||
});
|
||
});
|
||
|
||
// Every skill that renders {{PLAN_FILE_REVIEW_REPORT}} / {{EXIT_PLAN_MODE_GATE}},
|
||
// on every host: the union covers each skill-specific branch of both resolvers.
|
||
function renderPlanReportAndGate(): string {
|
||
const { generatePlanFileReviewReport } = require('../scripts/resolvers/review-dashboard');
|
||
const { generateExitPlanModeGate } = require('../scripts/resolvers/plan-gates');
|
||
const { HOST_PATHS } = require('../scripts/resolvers/types');
|
||
const skills = ['codex', 'devex-review', 'plan-ceo-review', 'plan-design-review', 'plan-devex-review', 'plan-eng-review'];
|
||
return ALL_HOST_CONFIGS.flatMap((host) => skills.map((skillName) => {
|
||
const ctx = { skillName, tmplPath: `${skillName}/SKILL.md.tmpl`, host: host.name, paths: HOST_PATHS[host.name] };
|
||
return `${generatePlanFileReviewReport(ctx)}\n${generateExitPlanModeGate(ctx)}`;
|
||
})).join('\n');
|
||
}
|
||
|
||
// GSTACK REVIEW REPORT report-at-bottom contract — verifies the prompt-text
|
||
// fix in scripts/resolvers/review-dashboard.ts (the load-bearing change for the
|
||
// "report not at bottom of plan in plan mode" bug). The bug is in the
|
||
// prompt's contradictory write-flow instructions, not in observable
|
||
// runtime behavior we can cheaply gate in CI. Verifying the prompt text
|
||
// directly is the deterministic equivalent of the regression test the
|
||
// PTY harness can't reliably drive (autoplan needs auto-progression of
|
||
// AskUserQuestions to reach the report-write step, which the harness
|
||
// doesn't support today).
|
||
describe('GSTACK REVIEW REPORT delete-then-append flow', () => {
|
||
const PLAN_REVIEW_SKILLS = [
|
||
'plan-ceo-review',
|
||
'plan-design-review',
|
||
'plan-devex-review',
|
||
'plan-eng-review',
|
||
];
|
||
|
||
for (const skill of PLAN_REVIEW_SKILLS) {
|
||
test(`${skill}/SKILL.md prescribes delete-then-append, not in-place replace`, () => {
|
||
// Carved skills (v2 plan Phase B) relocate the review-report prose into
|
||
// sections/*.md; readSkillUnion follows the content wherever the carve put it.
|
||
const content = readSkillUnion(skill);
|
||
|
||
// The new (correct) instruction must be present.
|
||
expect(content).toContain('delete-then-append flow');
|
||
|
||
// The old contradictory bullets must be gone. The signature phrase
|
||
// from the buggy prompt was 'replace it entirely using the Edit tool'
|
||
// which is what allowed mid-file reports to stay mid-file.
|
||
expect(content).not.toContain('replace it** entirely using the Edit tool');
|
||
expect(content).not.toContain('If it was found mid-file, move it');
|
||
});
|
||
}
|
||
|
||
test('plan-file review report resolver renders the rewritten flow', () => {
|
||
const src = renderPlanReportAndGate();
|
||
expect(src).toContain('delete-then-append flow');
|
||
// Old contradictory bullets are gone from the source resolver.
|
||
expect(src).not.toContain('replace it** entirely using the Edit tool');
|
||
expect(src).not.toContain('If it was found mid-file, move it');
|
||
});
|
||
});
|
||
|
||
describe('LEARNINGS_SEARCH resolver: query parameter', () => {
|
||
// Lazy-load resolver and types after describe block to keep test file self-contained.
|
||
const { generateLearningsSearch } = require('../scripts/resolvers/learnings');
|
||
const { HOST_PATHS } = require('../scripts/resolvers/types');
|
||
|
||
const claudeCtx = {
|
||
skillName: 'test',
|
||
tmplPath: 'test/SKILL.md.tmpl',
|
||
host: 'claude',
|
||
paths: HOST_PATHS.claude,
|
||
};
|
||
const codexCtx = { ...claudeCtx, host: 'codex', paths: HOST_PATHS.codex };
|
||
|
||
test('no args → bash does not contain --query (backwards-compat)', () => {
|
||
const out = generateLearningsSearch(claudeCtx);
|
||
expect(out).not.toContain('--query');
|
||
});
|
||
|
||
test('claude host + query=foo bar → both cross-project and project-scoped branches contain --query', () => {
|
||
const out = generateLearningsSearch(claudeCtx, ['query=foo bar']);
|
||
// Both branches of the if/else must carry the flag.
|
||
const lines = out.split('\n').filter(l => l.includes('gstack-learnings-search'));
|
||
expect(lines.length).toBeGreaterThanOrEqual(2);
|
||
for (const line of lines) {
|
||
expect(line).toContain('--query "foo bar"');
|
||
}
|
||
});
|
||
|
||
test('codex host + query=foo bar → codex bash variant contains --query', () => {
|
||
const out = generateLearningsSearch(codexCtx, ['query=foo bar']);
|
||
expect(out).toContain('--query "foo bar"');
|
||
expect(out).toContain('$GSTACK_BIN/gstack-learnings-search');
|
||
});
|
||
|
||
test('empty value query= → bash does not contain --query (locked semantics: falls through)', () => {
|
||
const claudeOut = generateLearningsSearch(claudeCtx, ['query=']);
|
||
expect(claudeOut).not.toContain('--query');
|
||
const codexOut = generateLearningsSearch(codexCtx, ['query=']);
|
||
expect(codexOut).not.toContain('--query');
|
||
});
|
||
|
||
test('shell-injection chars in query= → throws at gen-time (defense in depth)', () => {
|
||
for (const bad of ['$(whoami)', '`cmd`', 'a;b', 'a&b', 'a"b', 'a\\b', 'foo$x']) {
|
||
expect(() => generateLearningsSearch(claudeCtx, [`query=${bad}`])).toThrow(/alphanumeric/);
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('EXIT PLAN MODE GATE placement', () => {
|
||
// Fresh skill list — do NOT reuse REVIEW_SKILLS upstream (3 entries, missing plan-devex).
|
||
const planSkills = ['plan-eng-review', 'plan-ceo-review', 'plan-design-review', 'plan-devex-review'];
|
||
|
||
// Strip fenced code blocks before matching headings — PLAN_FILE_REVIEW_REPORT
|
||
// already contains `## GSTACK REVIEW REPORT` inside a markdown example fence,
|
||
// and the gate text itself shows `## GSTACK REVIEW REPORT` inside a fence too.
|
||
const stripFences = (md: string) => md.replace(/```[\s\S]*?```/g, '');
|
||
|
||
test('gate follows review work, with only the declared cache hook after CEO and Eng verification', () => {
|
||
for (const skill of planSkills) {
|
||
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
const stripped = stripFences(md);
|
||
const headings = [...stripped.matchAll(/^## .+$/gm)].map(m => m[0]);
|
||
const lastH2 = headings.at(-1);
|
||
if (['plan-ceo-review', 'plan-eng-review'].includes(skill)) {
|
||
const gate = headings.indexOf('## EXIT PLAN MODE GATE (BLOCKING)');
|
||
expect(gate).toBeGreaterThan(headings.indexOf('## Section self-check (before you finish)'));
|
||
expect(headings.slice(gate)).toEqual(['## EXIT PLAN MODE GATE (BLOCKING)', '## Brain Cache Background Refresh']);
|
||
const tail = md.slice(md.indexOf('## EXIT PLAN MODE GATE (BLOCKING)'));
|
||
const telemetry = tail.indexOf('**Telemetry (run last)**');
|
||
expect(telemetry).toBeGreaterThan(0);
|
||
const cache = tail.indexOf('## Brain Cache Background Refresh');
|
||
if (skill === 'plan-ceo-review') expect(telemetry).toBeGreaterThan(cache);
|
||
else expect(telemetry).toBeLessThan(cache);
|
||
expect(tail).toContain(skill === 'plan-ceo-review' ? 'Passed with a verified persisted report' : 'After the gate passes');
|
||
const finalHandoff = tail.slice(tail.indexOf('## Brain Cache Background Refresh'));
|
||
expect(finalHandoff).toContain(skill === 'plan-ceo-review' ? 'Call ExitPlanMode where required or return to the caller' : 'After success telemetry and cache dispatch, call ExitPlanMode');
|
||
if (skill === 'plan-ceo-review') {
|
||
expect(finalHandoff.indexOf('Call ExitPlanMode')).toBeGreaterThan(finalHandoff.indexOf('**Telemetry (run last)**'));
|
||
expect(finalHandoff).toContain('handoff starts a separate workflow');
|
||
}
|
||
if (skill === 'plan-eng-review') {
|
||
expectMentions(tail.replace(/\s+/g, ' '), [['cannot','persistence','unrecovered']], 'tail.replace(/\s+/g, )');
|
||
expectMentions(finalHandoff, [['only', 'host', 'plan']], 'finalHandoff');
|
||
expectMentions(finalHandoff, [['do not','conversation','exitplanmode']], 'finalHandoff');
|
||
} else {
|
||
expect(tail.replace(/\s+/g, ' ')).toContain('**Pass with log-only gaps:**');
|
||
expectTokens(tail.replace(/\s+/g, ' '), ['**not persisted**'], 'tail.replace(/\s+/g, )');
|
||
expectMentions(tail.replace(/\s+/g, ' '), [['do not', 'unwritten', 'artifacts']], 'tail.replace(/\s+/g, )');
|
||
expectMentions(tail.replace(/\s+/g, ' '), [['without','exitplanmode','telemetry']], 'tail.replace(/\s+/g, )');
|
||
}
|
||
} else {
|
||
expect(lastH2, `${skill}/SKILL.md last ## heading (fences stripped)`).toBe('## EXIT PLAN MODE GATE (BLOCKING)');
|
||
}
|
||
if (skill === 'plan-eng-review') {
|
||
const gate = extractMarkdownSection(md, '## EXIT PLAN MODE GATE (BLOCKING)').replace(/\s+/g, ' ');
|
||
expectTokens(gate, ['**Blocked outcome**'], 'gate');
|
||
expectMentions(gate, [['without', 'exitplanmode', 'telemetry']], 'gate');
|
||
} else if (skill === 'plan-ceo-review') {
|
||
expect(md, `${skill}/SKILL.md gate body`).toContain('Failed checks use **Gate outcome: Blocked**');
|
||
} else {
|
||
expect(extractMarkdownSection(md, '## EXIT PLAN MODE GATE (BLOCKING)'), `${skill}/SKILL.md gate body`).toMatch(/do not call ExitPlanMode/i);
|
||
}
|
||
}
|
||
});
|
||
|
||
test('codex/SKILL.md contains gate (mid-file per D5; Step 2B/2C follow)', () => {
|
||
const codex = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
||
expect(codex).toContain('## EXIT PLAN MODE GATE (BLOCKING)');
|
||
expect(extractMarkdownSection(codex, '## EXIT PLAN MODE GATE (BLOCKING)')).toMatch(/do not call ExitPlanMode/i);
|
||
});
|
||
});
|
||
|
||
describe('scope-gate exceptions drift-guard', () => {
|
||
// The plan-mode auto-select-B exceptions block is hand-duplicated in the
|
||
// plan-eng-review and plan-design-review templates (matching the gate
|
||
// around it, which predates this block). The two copies must stay
|
||
// identical in their target-selection policy. Eng resolves that policy
|
||
// before announcing its target and has a separate bootstrap transport;
|
||
// Design retains its own startup sequence and page vocabulary.
|
||
// A future edit to one copy that silently misses the other fails here
|
||
// instead of drifting. The real fix (shared {{SCOPE_GATE}} resolver) is a
|
||
// filed TODO — this guard is the stopgap that makes the duplication safe.
|
||
const START_MARKER = '**Exceptions — check in this order, BEFORE asking:**';
|
||
const END_MARKER = 'When no exception above applied:';
|
||
|
||
function extractExceptionsBlock(skill: string): string {
|
||
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
const start = md.indexOf(START_MARKER);
|
||
expect(start, `${skill}/SKILL.md: exceptions block start marker present`).toBeGreaterThan(-1);
|
||
const end = md.indexOf(END_MARKER, start);
|
||
expect(end, `${skill}/SKILL.md: exceptions block end marker present`).toBeGreaterThan(start);
|
||
return md.slice(start, end + END_MARKER.length);
|
||
}
|
||
|
||
const normalizeTargetPolicy = (block: string) => block
|
||
.split('\n').filter(line => /^[12]\. /.test(line)).join('\n')
|
||
.replace(/ Announce (?:it|an auto-selected plan) in one line so the user can interrupt: "Scope gate: plan mode — auto-selected B \(reviewing <target>\)\."/g, '')
|
||
.replace(/ Then run the pre-review audit, .*? against that plan\./, '')
|
||
.replace('a path, a page, a doc they pasted,', 'a path, a doc they pasted,');
|
||
|
||
test('eng and design retain the same target-selection policy across distinct startup sequences', () => {
|
||
const engRaw = extractExceptionsBlock('plan-eng-review');
|
||
const eng = normalizeTargetPolicy(engRaw);
|
||
const design = normalizeTargetPolicy(extractExceptionsBlock('plan-design-review'));
|
||
expect(eng).toBe(design);
|
||
expect(eng).toContain('their choice wins — use it instead');
|
||
expect(eng).toContain('still ambiguous — ask');
|
||
expectMentions(eng, [['ask','default','doubt']], 'eng');
|
||
expect(engRaw.indexOf('their choice wins')).toBeLessThan(engRaw.indexOf('Announce an auto-selected plan'));
|
||
const engMd = fs.readFileSync(path.join(ROOT, 'plan-eng-review/SKILL.md'), 'utf8');
|
||
const transport = engMd.indexOf('Choose listed, enabled MCP AskUserQuestion, otherwise listed native');
|
||
expect(transport).toBeGreaterThan(engMd.indexOf(END_MARKER));
|
||
expect(transport).toBeLessThan(engMd.indexOf('## Preamble (after scope gate)'));
|
||
});
|
||
|
||
test('exceptions block carries the announcement string the PTY detectors pin', () => {
|
||
for (const skill of ['plan-eng-review', 'plan-design-review']) {
|
||
const block = extractExceptionsBlock(skill);
|
||
expect(block, `${skill}: verbatim announcement`).toContain(
|
||
'Scope gate: plan mode — auto-selected B (reviewing <target>).',
|
||
);
|
||
}
|
||
});
|
||
|
||
test('gate menu carries the question strings the PTY question detector pins', () => {
|
||
// isScopeGateQuestionVisible (claude-pty-runner.ts) anchors on the
|
||
// question text + option A's body. If the menu is reworded without
|
||
// updating the detector, the paid smokes' must-stay-false assertions go
|
||
// vacuous — this free pin fails first.
|
||
for (const skill of ['plan-eng-review', 'plan-design-review']) {
|
||
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
expect(md, `${skill}: gate question text`).toContain('What should I review?');
|
||
expect(md, `${skill}: option A body text`).toContain('The current branch diff');
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('GSTACK REVIEW REPORT mandatory unresolved-decisions status', () => {
|
||
// Report text rides in PLAN_FILE_REVIEW_REPORT → every report consumer gets it.
|
||
// devex-review is a report consumer but NOT a gate consumer, so the two target
|
||
// sets differ (CP5/CX5). Regression guard: a future token-cut that drops the
|
||
// unresolved-status line again fails here. See plan-flag-unresolved-issues.
|
||
const REPORT_CONSUMERS = [
|
||
'plan-ceo-review',
|
||
'plan-eng-review',
|
||
'plan-design-review',
|
||
'plan-devex-review',
|
||
'codex',
|
||
'devex-review',
|
||
];
|
||
// Gate text rides in EXIT_PLAN_MODE_GATE (lives in SKILL.md, not sections).
|
||
const GATE_SKILLS = [
|
||
'plan-ceo-review',
|
||
'plan-eng-review',
|
||
'plan-design-review',
|
||
'plan-devex-review',
|
||
'codex',
|
||
];
|
||
|
||
for (const skill of REPORT_CONSUMERS) {
|
||
test(`${skill}: report mandates the unresolved-decisions status as final content`, () => {
|
||
const content = readSkillUnion(skill);
|
||
expect(content).toContain('NO UNRESOLVED DECISIONS');
|
||
// \s+ tolerates prose line-wraps within "final non-whitespace line".
|
||
expect(content).toMatch(/final\s+non-whitespace\s+line/i);
|
||
});
|
||
}
|
||
|
||
for (const skill of GATE_SKILLS) {
|
||
test(`${skill}: exit gate blocks unless the unresolved status is the final line`, () => {
|
||
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
// Gate check #4 — present, sentinel named, and explicitly blocking (no escape).
|
||
expect(md).toContain('NO UNRESOLVED DECISIONS');
|
||
if (['plan-ceo-review', 'plan-eng-review'].includes(skill)) {
|
||
const gate = md.split('## EXIT PLAN MODE GATE (BLOCKING)')[1]!.replace(/\s+/g, ' ');
|
||
expect(gate).toMatch(/final\s+non-whitespace\s+line is the exact unbolded `NO UNRESOLVED DECISIONS`/i);
|
||
expect(gate).toContain('`**UNRESOLVED DECISIONS:**`');
|
||
expect(gate).toMatch(/missing\s+status[^.]*fails/i);
|
||
expect(gate).toContain(skill === 'plan-eng-review'
|
||
? 'If any check fails, follow **Blocked outcome** without success telemetry or ExitPlanMode'
|
||
: 'Failed checks use **Gate outcome: Blocked**');
|
||
} else {
|
||
const gate = extractMarkdownSection(md, '## EXIT PLAN MODE GATE (BLOCKING)');
|
||
expect(gate).toMatch(/final\s+non-whitespace\s+line/i);
|
||
expect(gate).toContain('`**UNRESOLVED DECISIONS:**`');
|
||
expect(gate).toMatch(/missing\s+status[^.]*fails/i);
|
||
}
|
||
});
|
||
}
|
||
|
||
test('plan-file review report and exit gate resolvers render the mandatory block + blocking gate', () => {
|
||
const src = renderPlanReportAndGate();
|
||
expect(src).toContain('NO UNRESOLVED DECISIONS');
|
||
expect(src).toMatch(/avoids double-counting/i);
|
||
expect(src).toMatch(/final\s+non-whitespace\s+line/i);
|
||
expect(src).toMatch(/missing\s+status[^.]*fails/i);
|
||
// The old soft wording must be gone from the gate.
|
||
expect(src).not.toContain('absorbs CODEX / CROSS-MODEL / UNRESOLVED lines if applicable');
|
||
});
|
||
});
|
||
|
||
// ─── {{PREAMBLE}} requires an explicit preamble-tier ────────
|
||
|
||
describe('PREAMBLE resolution requires declared preamble-tier', () => {
|
||
test('resolving {{PREAMBLE}} without preamble-tier throws with the template path', async () => {
|
||
const { generatePreamble } = await import('../scripts/resolvers/preamble');
|
||
const { HOST_PATHS } = await import('../scripts/resolvers/types');
|
||
const ctx = {
|
||
skillName: 'tierless-skill',
|
||
tmplPath: 'tierless-skill/SKILL.md.tmpl',
|
||
host: 'claude' as const,
|
||
paths: HOST_PATHS.claude,
|
||
// preambleTier deliberately absent — the generator must refuse to default it.
|
||
};
|
||
expect(() => generatePreamble(ctx)).toThrow(/tierless-skill\/SKILL\.md\.tmpl/);
|
||
expect(() => generatePreamble(ctx)).toThrow(/preamble-tier/);
|
||
});
|
||
|
||
test('every template that resolves {{PREAMBLE}} declares preamble-tier in frontmatter', () => {
|
||
const entries = fs.readdirSync(ROOT, { withFileTypes: true });
|
||
const offenders: string[] = [];
|
||
const checkTmpl = (tmplPath: string) => {
|
||
const tmpl = fs.readFileSync(tmplPath, 'utf-8');
|
||
if (tmpl.includes('{{PREAMBLE}}') && !/^preamble-tier:\s*\d+$/m.test(tmpl)) {
|
||
offenders.push(path.relative(ROOT, tmplPath));
|
||
}
|
||
};
|
||
checkTmpl(path.join(ROOT, 'SKILL.md.tmpl'));
|
||
for (const e of entries) {
|
||
if (!e.isDirectory() || e.name.startsWith('.') || e.name === 'node_modules') continue;
|
||
const tmplPath = path.join(ROOT, e.name, 'SKILL.md.tmpl');
|
||
if (fs.existsSync(tmplPath)) checkTmpl(tmplPath);
|
||
}
|
||
expect(offenders).toEqual([]);
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// #2499: gbrain MCP detection must read BOTH ~/.claude.json scopes.
|
||
// Claude Code registers MCP servers at user scope (.mcpServers) and project
|
||
// scope (.projects["/abs/path"].mcpServers — what `claude mcp add` without
|
||
// --scope user writes). The rendered brain-sync block previously read only
|
||
// user scope, so a correctly configured project-scoped brain was invisible.
|
||
// ---------------------------------------------------------------------------
|
||
describe('brain-sync block reads project-scoped MCP registrations (#2499)', () => {
|
||
// Phase 1: the artifacts-sync bash (including the MCP-scope jq probe) moved
|
||
// from the rendered SKILL.md into bin/gstack-skill-start. Pin the LIVE
|
||
// script bytes — same assertions, new home. The render carries only the
|
||
// ARTIFACTS_SYNC interpretation prose.
|
||
const rendered = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
|
||
|
||
test('rendered _GBRAIN_MCP_ENTRY jq resolves project scope with nearest-ancestor cwd match', () => {
|
||
const line = rendered.split('\n').find((l) => l.includes('_GBRAIN_MCP_ENTRY=$('));
|
||
expect(line).toBeDefined();
|
||
// Project-scope read present, driven by $PWD.
|
||
expect(line!).toContain('--arg cwd "$PWD"');
|
||
expect(line!).toContain('.projects');
|
||
// User scope still resolved first.
|
||
expect(line!).toContain('.mcpServers.gbrain');
|
||
// The old user-scope-only filter is gone from the rendered output.
|
||
expect(rendered).not.toContain('.mcpServers.gbrain.type // .mcpServers.gbrain.transport');
|
||
expect(rendered).not.toContain(".mcpServers.gbrain.url // empty");
|
||
});
|
||
|
||
test('rendered _GBRAIN_MCP_TYPE and _GBRAIN_HOST extract from the resolved entry', () => {
|
||
const typeLine = rendered.split('\n').find((l) => l.includes('_GBRAIN_MCP_TYPE=$('));
|
||
const hostLine = rendered.split('\n').find((l) => l.includes('_GBRAIN_HOST=$('));
|
||
expect(typeLine).toBeDefined();
|
||
expect(hostLine).toBeDefined();
|
||
expect(typeLine!).toContain('_GBRAIN_MCP_ENTRY');
|
||
expect(hostLine!).toContain('_GBRAIN_MCP_ENTRY');
|
||
});
|
||
|
||
test('rendered jq lines FUNCTION: project-scoped registration resolves for a cwd inside the project', () => {
|
||
// Execute the exact rendered bytes, not a re-derivation: extract the
|
||
// _GBRAIN_MCP_ENTRY + _GBRAIN_MCP_TYPE lines from the generated SKILL.md
|
||
// and run them in bash against a fixture ~/.claude.json that carries ONLY
|
||
// a project-scoped gbrain registration.
|
||
const lines = rendered.split('\n');
|
||
const entryLine = lines.find((l) => l.includes('_GBRAIN_MCP_ENTRY=$('));
|
||
const typeLine = lines.find((l) => l.includes('_GBRAIN_MCP_TYPE=$('));
|
||
expect(entryLine).toBeDefined();
|
||
expect(typeLine).toBeDefined();
|
||
|
||
const tmpHome = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-2499-home-'));
|
||
const projectDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-2499-proj-'));
|
||
const nestedCwd = path.join(projectDir, 'src', 'deep');
|
||
fs.mkdirSync(nestedCwd, { recursive: true });
|
||
try {
|
||
fs.writeFileSync(
|
||
path.join(tmpHome, '.claude.json'),
|
||
JSON.stringify({
|
||
projects: {
|
||
[projectDir]: {
|
||
mcpServers: { gbrain: { type: 'http', url: 'https://brain.example.com/mcp' } },
|
||
},
|
||
},
|
||
}),
|
||
);
|
||
const script = `cd "$1" || exit 1\n${entryLine!.trim()}\n${typeLine!.trim()}\necho "RESOLVED:$_GBRAIN_MCP_TYPE"`;
|
||
const r = spawnSync('bash', ['-c', script, 'bash', nestedCwd], {
|
||
encoding: 'utf-8',
|
||
env: { ...process.env, HOME: tmpHome },
|
||
timeout: 10_000,
|
||
});
|
||
expect(r.stdout).toContain('RESOLVED:http');
|
||
|
||
// Discriminator: a cwd OUTSIDE the project must NOT resolve it.
|
||
const outside = spawnSync('bash', ['-c', script, 'bash', os.tmpdir()], {
|
||
encoding: 'utf-8',
|
||
env: { ...process.env, HOME: tmpHome },
|
||
timeout: 10_000,
|
||
});
|
||
expect(outside.stdout).toContain('RESOLVED:\n');
|
||
} finally {
|
||
fs.rmSync(tmpHome, { recursive: true, force: true });
|
||
fs.rmSync(projectDir, { recursive: true, force: true });
|
||
}
|
||
});
|
||
});
|
||
|
||
// #2896: Claude Code replaces `$<digit>` tokens in a SKILL.md body with the
|
||
// invocation's argument words before the model reads it (2.1.271+:
|
||
// /\$(\d+)(?!\w)/g), so shell/awk fields arrive rewritten yet still parse.
|
||
// Write awk fields as $(N) and shell positionals as ${N}. Section files are
|
||
// read later with Read and are not substituted.
|
||
describe('generated Claude skill bodies carry no $N token Claude Code rewrites (#2896)', () => {
|
||
const CLAUDE_ARG_TOKEN = /\$\d+(?!\w)/g;
|
||
test('tripwire', () => {
|
||
const offenders: string[] = [];
|
||
let checked = 0;
|
||
for (const file of ['SKILL.md', ...fs.readdirSync(ROOT).map(d => path.join(d, 'SKILL.md'))]) {
|
||
const full = path.join(ROOT, file);
|
||
if (!fs.existsSync(full)) continue;
|
||
const lines = fs.readFileSync(full, 'utf-8').split('\n');
|
||
const bodyStart = lines[0] === '---' ? lines.indexOf('---', 1) + 1 : 0;
|
||
checked++;
|
||
lines.slice(bodyStart).forEach((line, i) => {
|
||
for (const m of line.matchAll(CLAUDE_ARG_TOKEN)) offenders.push(`${file}:${bodyStart + i + 1}: ${m[0]} in ${line.trim().slice(0, 120)}`);
|
||
});
|
||
}
|
||
expect(checked).toBeGreaterThan(40);
|
||
expect(offenders).toEqual([]);
|
||
});
|
||
|
||
test('the tripwire pattern matches what Claude Code substitutes', () => {
|
||
expect('awk \'{print $1}\' "$0" ~$0.05 $12'.match(CLAUDE_ARG_TOKEN)).toEqual(['$1', '$0', '$0', '$12']);
|
||
expect('awk \'{print $(1)}\' "${1}" $ARGUMENTS $1a $_x'.match(CLAUDE_ARG_TOKEN)).toBeNull();
|
||
});
|
||
});
|