mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-11 19:20:26 +01:00
* docs: approved reliability follow-ups plan (Oct 2026)
* D1: score overlay wrong answers as comparison results (contract v4)
A completed answer that fails the fixture's output check is now a valid
measurement with taskCorrect false and answerError set, in either arm.
Only execution, scope and metric failures invalidate a trial. ON
correctness still gates the case, so an OFF wrong answer stays in the
comparison. OVERLAY_CONTRACT moves to v4; v3 verdicts are not
reinterpreted. Replays the 37198445662 sonnet trials (OFF-0 wrong).
* C4: score auq-matrix substance with a 3-sample judge panel
Each skill keeps one native capture in the same execution; the
recommendation-substance grader now runs as a JUDGE_PANEL_SAMPLES panel on
that capture and gates on the mean against the unchanged minimum of 4. A
failed sample fails the panel without resampling. Approved at the gate
2026-10-04 for censuses after this change.
* E1: one Bun version source; setup refuses below the 1.3.3 floor, updater checks before advancing
bin/gstack-bun-version.sh holds the security floor (1.3.3) and the tested
version (1.4.0, the CI pin and engines.bun). setup refuses Bun below the
floor before writing anything, warns between floor and tested, warns and
continues on an unparseable version, and prints a 1.4.0 install hint.
Prereleases sort below their release (semver).
gstack-session-update now fetches, reads the incoming release's floor and
checks the bun setup will run before fast-forwarding to that exact
revision; below it the update is held with reason bun-too-old and a
resume command, leaving the revision and installs untouched.
Fixture copies of setup (team-mode, claude-code migration) now include the
sourced file; the slow-git lock test sleeps on fetch, the updater's
network step.
* E3: record the owned Chromium's PID over CDP so the #2709 reap runs on Playwright 1.62
Playwright 1.62's Browser has no process(), so chromiumProcInfo was always
null and browse stop never reaped a surviving headless Chromium. The owned
headless launch now reads the browser PID from CDP SystemInfo.getProcessInfo
(bounded at 2s; a hung or failed call logs once and records nothing). The
reaper's start-time and cmdline identity checks are unchanged, so a recycled
PID or an externally owned browser is never signalled.
* A1: ship docs gate records post-child hashes with gstack-docs-candidate, saves each record once
Stored transcripts (37198445662 gate-census-2) show the parent's reads of
release files take ~1 s; the parent's own model time dominates (completion
268 s of 392 s, store ~270 s of 387 s), and 45-78 s of it is retyping hash
maps, child output and records into several artifacts after the child
returns. The child contract is unchanged.
* A2: /plan-ceo-review sends the mode handoff line before any other tool call
plan-ceo-mode-routing's HOLD SCOPE session spends its 240 s post-selection
window on model work: 37176837432 skipped the handoff helper and paraphrased
the mode, 37198445662 ran it, then wrote the plan, loaded sections and probed
Codex for 114 s before the posture chat. The handoff line is the posture.
* B3: accept wrapped unchanged rows in the cropped Edit preview
PR run 37176835584 timed out with the floor's owned Edit menu on screen:
unchanged diff rows soft-wrap under their gutter with a blank marker
column, and a trimmed empty unchanged row keeps only its number. The
parser accepted wrapped +/- rows only, so it never verified the owned
edit. Both forms are now parsed and still compared with the exact source
lines; the captured viewport is replayed with negative controls.
* A3: record per-phase headroom diagnosis for docsync, plan-mode-no-op, mode-routing and gate slice 11
* E4: /review Step 1 keeps going when the base fetch fails and reports stale base coverage
Under Codex's read-only .git, 'git fetch && merge-base && diff --stat'
stopped at the fetch, so the empty output read as 'Nothing to review'.
The fetch is now separate, prints BASE_REFRESH fresh or stale with the
local base revision, and the review states stale coverage.
* D2: reword the format-approach question prompt
Both plan-ceo-review-format-approach refusals in census 37195203538
(Fable 5.1 model_refusal_no_fallback, category reasoning_extraction:
"duplicating model outputs") were anchored on the session prompt, whose
capture instruction asked for the "verbatim" question and "the exact
prose you would have shown" in a "format-capture test". The approach
case now asks for the same question in the same file as the step's
output, without that wording. The other three plan-format cases keep
their unchanged prompt (no refusals in 72 trials).
* C2: /qa defines its outcome filename fields; QA report templates name the measured timing fields
Low judge samples in failing panels named both: qa workflow (37186854666)
'an undefined {user} in the outcome filename', and qa-only workflow
(37198445662) 'template asks for Duration / durationMs totals while
reporting.md mandates Probe budget and Guarded command time'. The qa
union stays under its 1.11 size ratio (93292 of 93294 bytes).
* 0.1: session ledger written by the claude -p, Agent SDK, PTY and Codex runners
Each armed model session appends one best-effort line to
$GSTACK_EVAL_DIR/session-ledger.jsonl: a stable session key, the timeout it
armed, monotonic elapsed time, its end reason (completed, session_timeout,
observer_timeout, api_error, refusal, aborted, error), one sanitized evidence
line and, for stream-JSON runners, a compact liveness summary (longest
in-request silence with nothing outstanding, last event kind, tool open at
end). Partial messages now stream for every claude -p session and for SDK
sessions without a custom query provider (runSharedInteractive opts in); their
rows feed only the summary, so transcripts keep their size unless
publicStreamDiagnostics is set. Structured refusal and API-error signals win
over phrases, and tool results are never read for them.
* 0.2: failure_cause beside failure_class, from runner facts and the session ledger
One pure failureCauseOf() beside failureClassOf() sets failure_cause
(contract > pre_turn_infra > api_error > refusal > provider_stall >
session_timeout > observer_timeout > assertion > unknown; no machine-set
'detector') with one bounded evidence line, and failureDetailOf() keeps a
Bun matcher's Expected/Received or a judge's failing dimensions in
failure_detail. Both record builders (classifyTrialShard, junitCensus) call
them and attach the trial's ledger sessions. failure_class, error and
TRIAL_ERROR_MAX are computed exactly as before; the new fields are optional,
bounded and validated only when present, and a frozen copy of the v1 reader
accepts every new record. A provider stall needs partial messages and
STALL_WINDOW_MS (120 s, eval-budgets.ts beside HEADROOM_WARN/FAIL) of
in-request silence with nothing outstanding; it stays a failure under
EVAL_POLICY v1.
* E4: fingerprint under a read-only .git via a private object dir owned by review-log
Codex's sandbox mounts .git read-only, so gstack-wtree's git add -A could not
write blobs and /review lost its start capture. gstack-review-log now creates
a private 0700 object directory under TMPDIR, passes it to gstack-wtree (which
writes new objects there with the common-dir object store, resolved before
exporting, as an alternate, so linked worktrees work), exposes it to the
bindReview/checkSharedLibsReuse consumers that read the tree and blobs, and
removes it on EXIT/INT/TERM after they finish. gstack-wtree prints one stderr
line and exits 1 on a failed write or unwritable TMPDIR instead of printing a
partial hash; review-log no longer swallows that line. Standalone gstack-wtree
keeps writing to the repository store so git diff between saved snapshots
(/ship) still works.
* 0.3: red lines name their values, evidence and an after-a-repair command; one published-text sanitizer
Each red line in the census report keeps its stored error and adds the
machine cause with its evidence, the failure_detail values (Expected/Received,
or each failing judge dimension with its mean, sample count and the sample
sentence naming it), the slice artifact and shard directory that hold the
transcript, a fetch command (eval:pass-rates --run <id> --case <id>), and
'after a repair:' with a command that selects only that case (E2E ids via
--case, judges via their selection and an exact test-name filter). A
session that completed and failed a check says so instead of 'assertion
(success)'. Report text that reaches a PR comment, issue or committed file
goes through scripts/lib/published-text.ts: control and bidi characters
stripped, credential-scan patterns redacted, mentions neutralized, HTML and
link brackets escaped inline, blocks in a fence longer than any backtick run,
length capped. Verdicts and collector verdict objects are unchanged.
* E2: one make-pdf resolver for readiness and every $P block on env-var hosts
make-pdf exported $P from its readiness block, which Codex and the other
env-var hosts lose between blocks. binaryAssignment gains 'make-pdf': the
MAKE-PDF SETUP readiness check and the runtime prelude both use it, so every
block that runs $P derives the same make-pdf/dist/pdf binary, with
MAKE_PDF_BIN (dev builds) winning when it is executable. Claude output keeps
its repo-local-first probe; only the false 'subsequent blocks' comment
changed. Root plus P=... fits the 400-byte prelude budget; the fixed
MAKE_PDF_OVERRIDE segment is the one exemption, named in runtime-root.ts and
the conformance test.
* 0.1: eval:pass-rates --headroom, --reds and --run report views; 85% headroom alarm in --gate
A new pure module, scripts/lib/eval-history.ts, reports per case the slowest
armed session against the timeout it armed (censored when it timed out,
insufficient below 3 samples, unknown without a session ledger, with the
case wall shown separately as an upper bound), the census critical path from
the slice jobs' timings, the verdict red ledger per census by failure class
and machine cause with shrunk per-case rates, and the all-green probability
(an approximation: the product of 1 - p_i, with an interval). --run <id>
lists one census's reds with their values and history and downloads only the
slice artifacts those reds name, reporting any over 64 MB instead of skipping
them. eval-flake-rank.ts only routes flags: --help works offline, an unknown
flag or case id exits 2 before any fetch, --case narrows the JSON and the
alarms, every view prints its run/branch scope, and --gate also alarms on a
session above HEADROOM_FAIL (85%) of its budget.
* E5: GSTACK_DESIGN_IMAGE_MODEL override; the live model check always tests the defaults
The image_generation tool model gains an env override validated against a
conservative gpt-image-<version> pattern; a bad value fails before any
request and the error names the variable. Pairing and rejection hints name
the tool model actually requested. design/scripts/live-model-check.ts now
ignores GSTACK_DESIGN_MODEL and GSTACK_DESIGN_IMAGE_MODEL (and says so), so
an override cannot mask a retirement of a default, and its exit 2 (no key)
says nothing was checked.
* C3: /deslop-shared-libs names a concrete risk for each recommendation, never 'none'
Both shared-libs-opportunity-judgment reds (37193478719 t3, 37195203538 t1)
failed adoption_and_risk because the top recommendation said 'Main risk:
none found inside the repo' / 'none in behavior'. The skill asked to 'name
the main risk or uncertainty' without saying what counts.
* B4: --case selects loop-registered cases; an unknown id exits before any process
caseSelection() resolves the one paid file that registers a case and how a
trial selects it: by its literal Bun test name, by the id a registration loop
names its test at runtime (coverage audits, TPA, journeys), or by running the
whole file when it registers no other case (plan-mode-no-op, whose test names
are not the id, so the old name pattern ran nothing). A whole-file case runs
one file shard per trial and is judged by shard status, as CI judges it.
--list prints the selection. Every E2E_TIERS id now resolves except
carve-section-loading, which fifteen files register (CI runs each as its own
shard) and which still fails with the owning files named.
* 0.3: after-a-repair runs the shard file when --case cannot resolve a case
A case that several files register (carve-section-loading) cannot be selected
by --case; its red line now prints the bun test command for the shard file
that went red, so every printed command selects exactly one case.
* C1: pre-register workflow-judge control bundles for qa, qa-only, review and setup-deploy
Positive (production bundle), negative (headings only) and one targeted
execution-blocking defect per judge, one 3-sample panel each, built with the
eval's own reader, prompt builder, options and floors.
* F: cost_known: a trial whose harness captured no billing says so; totals never hide it
cost_usd stays a finite number. Trial and eval records gain an optional
cost_known boolean (absent means known, so old records still parse; the
validator accepts only a boolean). Session-ledger rows record whether the
session's terminal result carried total_cost_usd; PTY and Codex sessions
never do, and SDK captures without a terminal result do not. A trial is
cost-known only when it has eval records and every record and session
captured billing, so JUnit-only PTY cases (125 of 298 trials in run
37198445662 recorded 0) now read cost unknown. The census headline prints the
known sum plus the unknown count, never a bare total over unknown costs, and
no total when every cost is unknown; the history import keeps the flag.
PTY transcripts are not priced: test/helpers/pricing.ts estimates exclude
cache writes, so they cannot give the billed cost.
* B1: replay corpora for the five repeat-offender cases
shared-libs-plan-callers, plan-ceo-section-loading, auto-decide-preserved,
plan-design-review-plan-mode and ship-docsync-late-result each get an
inventory of every wave census in scope (all artifacts still available)
and minimal, redacted detector or actor inputs replayed through the real
detector in the free suite: census passes, every distinct census red, and
derived negative controls labelled as derived. Fixed false reds stay as
expected passes with their history; the two genuine auto-decide product
failures (37176837432, 37179171083) stay refused. A contract test bounds
each entry at 64 KB, scans it for HIGH credentials, requires a pass and a
refusal per case, and reports entries whose series identity is stale.
Building the auto-decide corpus exposed a latent native detector misread:
a bold 'Mode application and rationale.' heading was parsed as a
withdrawn Mode field (37186854666). A field status now ends at its
sentence.
* B2: read approved stale-fill decisions from more valid ledger records
The structured ledger reader is authoritative before the prose fallback,
but it missed two valid approved records in passing censuses
(37186854666: a camelCase writeProfile subject; 37195203538: a
slash-joined 'D1 / STALE-FILL' row id), which then rested on the prose
detector that caused both historical reds. Both forms are accepted; an
unresolved row, an unrelated decision, the copied fixture plan and
non-write camelCase words stay refused.
* C1: record workflow-judge control results (negatives separated; qa and review miss a removed step)
* C2: record every low workflow-judge sample in the wave censuses with its disposition
* 0.3: published lines cannot break a fixed fence
evals.yml wraps collector-outcomes failures in a fixed three-backtick fence
for the PR comment, so those lines (and eval:pass-rates output meant for the
tracking issue) go through sanitizeFixedFenceLine: cleaned like every
published line, with backticks turned into quotes. Inline published text
also neutralizes backticks.
* A1: fit the post-child record wording under ship's 1.428 parity size cap
* E4: inline the Step 1 base refresh to fit review's 1.199 parity size cap
* B2: inventory of gate and periodic phrase-only detector checks
docs/evals/detector-inventory.md lists every pass/fail decision in gate
and periodic E2E cases that rests on phrases in model free text, with
its structured source (if any), whether the contract is user-visible,
and a disposition: already structured-first, convert candidate, phrase
witness required, or no structured source. Convert candidates are
ordered by census red history.
* 0.2: failureCauseOf takes the stall window instead of importing it; the PTY ledger never throws
eval-store.ts imported STALL_WINDOW_MS from eval-budgets.ts, which free tests
replace with mock.module stubs (review-finalization-budget, office-hours,
qa-fix-loop, qa-only), so loading eval-store failed there. failureCauseOf is
now dependency-free: the record builders pass STALL_WINDOW_MS. A PTY session
whose handle cannot show its text (test doubles) records no panel instead of
throwing from runPtySession's cleanup. The 37186854666 replay also runs
through classifyTrialShard.
* A2/auto-decide: the CEO handoff chat repeats the helper's printed line verbatim
Claude Code 2.1.284 collapses the helper's Bash output ('Ran 1 shell
command'), so the 'Auto-decided review mode → …' line reaches the user only
through chat. In lane D's auto-decide-preserved corpus every passing capture's
chat starts with that line, and the reds paraphrased it ('HOLD SCOPE was
auto-selected…', 'Mode decision done…', "I'm defaulting to HOLD SCOPE…").
The template now says verbatim and why.
* C2: record the judge panels rerun on the final ship and review wording
* integrate: touchfiles and module-size ratchet entries for the wave's new modules
GLOBAL_TOUCHFILES gains session-ledger.ts, published-text.ts and
auq-substance-panel.ts; bin/gstack-bun-version.sh joins every entry that
lists setup or the updater; auq-matrix stays a rule case with its C4
evidence beside E2E_KINDS; eval-history, published-text and session-ledger
are registered new modules.
* E1: BROWSE SETUP install hint pins the tested Bun (1.4.0)
* E2: quote "$P" in make-pdf commands so install paths with spaces work
* E1: /gstack-upgrade checks the incoming Bun floor before pulling
gstack_bun_incoming_hold (bin/gstack-bun-version.sh) is now the one
pre-advance check: the auto-updater calls it before its fast-forward and
/gstack-upgrade's git path calls it after fetching. Below the incoming
release's floor the upgrade stops with BUN_TOO_OLD and the checkout,
VERSION and installs stay untouched. Vendored installs were already safe:
setup refuses before writing and the swap restores the previous install.
* 0.3: only main posts or closes tracking issues; reports carry alarms, red ledger, headroom and all-green
A branch-dispatched periodic or marathon run writes its report to the step
summary and a report artifact; the upsert and close steps run only on
refs/heads/main. The periodic body now keeps every pass-rate alarm line (the
old grep kept only the header), adds the red ledger and session headroom read
with an explicit --branch, the all-green estimate and the triage guide link,
and routes report text through scripts/lib/published-text.ts. The marathon
lane gains the matching close-on-green step.
Migrated: evals-workflow-wiring pinned report-summary.md in the upsert step;
the body is now written by 'Write the census report'.
* C3: prosons neutral-posture detector ignores a negated "taste call"
Both plan-review-prosons-neutral-neg reds (37179171083 t3, 37193478719 t3)
recommended A and closed with "a coverage call, not a taste call"; the
old /taste call/i matched the negation. The detector moves to a helper so
the two census captures replay free as negatives, with positive controls.
* B2: credit the CEO mode handoff only when the chat shows its line
Claude Code 2.1.284 collapses the gstack-ceo-mode-handoff Bash result, so
the printed AUTO_DECIDE line is user-visible only when a later chat message
of the session begins with it. The native detector now requires that
witness; the helper's tool result locates it but no longer replaces it, and
a generic mode statement cannot stand in once a handoff ran. Census
37182865432 is relabelled a genuine miss (expected: refused): the chat said
"I'm defaulting to HOLD SCOPE…" and "Mode decided: HOLD SCOPE". Corpus
replay: 5 credited, 3 refused. ceo-mode-handoff tests migrated to the
witness rule (deliberate behavior change).
* test: baseline the wave's approved detector and harness replay tests
B1 corpora, B3, C3, C4 and D2 replay captures through paid-case detectors,
actors or fixtures (test code by design); each entry carries its reason.
* E6: live Codex multi-block periodic case in the host-run Codex job
test/codex-e2e-multiblock-live.test.ts runs ./setup --host codex once from the
job's checkout into a fresh HOME and CODEX_HOME (auth.json copied from the CI
login home), asserts codex debug prompt-input lists the gstack router as a
real file, seeds one learning, and runs /learn's Show recent block through
codex exec: that later block must root itself through its prelude, run the
installed gstack-learnings-search and print the seeded learning; no block may
report a missing install or an unresolved gstack helper path.
The file joins CODEX_CI_FILES so the planner gives it to the host-run Codex
job. paid-test-durations.json carries a 180 s packing estimate (refreshed by
--write-durations after the first census) so the plan keeps every slice
within max-parallel. Oracles live in test/helpers/codex-multiblock.ts with
free replays in test/codex-multiblock.test.ts. Registration (E2E_TOUCHFILES,
E2E_TIERS periodic, E2E_KINDS rule) is the integrator's.
* E5: design live-model check as a periodic case
test/skill-e2e-design-model-smoke.test.ts runs design/scripts/live-model-check.ts
(default models only) under a temp HOME. Without OPENAI_API_KEY the case is a
Bun skip that the census lists as SKIPPED with no coverage credit; with a key,
exit 0 passes and any other exit fails with the script output (exit 2 under a
present key fails as 'nothing was checked'). paid-test-durations.json carries
a 90 s packing estimate. Registration (E2E_TOUCHFILES, E2E_TIERS periodic,
E2E_KINDS rule) is the integrator's.
* A1: docsync parent points children at a saved interface file
The /ship docsync fixture saved its observation interface only in the
parent prompt and told the parent to copy it into every child prompt,
about 30-40 s of parent generation per session (lane S, 37198445662).
docsSessionOptions now writes it to <home>/fixture-interface.md (0600) and
the parent tells each child to Read that file first. Fault-actor prompts,
which prepare already writes to a file, are unchanged. Fixture-only.
* A3: a JUnit case's recorded wall is capped by its shard's wall
junitCensus summed a case's JUnit test times as duration_ms; for
plan-mode-no-op's concurrent PTY tests that is 346 s inside a 176 s shard
(the "432-613 s" history). The case wall headroom shows is now
min(sum, shard elapsed), a true upper bound. Session headroom already
used the ledger's per-session clocks.
* A3: refresh paid-test durations from census 37198445662
Recorded 2026-09-29 estimates under-packed gate slice 11 (17 files
estimated 1078 s ran 1353 s; plan-review-report 76 s -> 198 s,
plan-ceo-plan-mode 37 s -> 112 s, safety-codex-consult 83 s -> 150 s).
Refreshed both tiers with --report --write-durations from the latest
census's slice walls. Old plan re-costed with observed walls: worst slice
~677 s (slice 11); refreshed plan: every slice estimated <= 539 s.
* C2: Browser Quick gets a feasible budget; QA loop names its directory and stop path
Judge rationales on the qa-only panels (2 of the last 3) cited the 30 s
Browser Quick budget: homepage + 5 targets, each a checkpoint write and a
guarded command, left ~5 s per probe. Quick is now 180 s (method text: 3
minutes); the qa-only-no-fix paid check expects the same native deadline.
PROBE_DIR points at its source line, a stopped loop goes to the final
report steps (annotations, materialize, Markdown), and /review and /ship
say that after smoke expiry only plan checks rerun in revalidation.
* C2: name the flows judges found implicit in review resolvers
Structured outside review: MISSING COVERAGE asks no fix question and is
never reported clean (was "preserve the existing user decision flow").
/review's HIGH-impact plan gate: spawned or non-interactive runs report
REQUIREMENTS MISSING and continue, writing no TODOs. Value cards: the byte
clamp applies to card lines; JSON records you write keep full values.
* C1: a free test catches a removed workflow step, which the judges missed
C1 controls showed the qa workflow judge passing /qa without Phase 10 and
the review judge passing /review without Step 3. The new test pins every
step/phase heading of qa, qa-only, review, ship, plan-ceo-review,
plan-eng-review and setup-deploy (derived from the templates' step
headings) and names any that goes missing or out of order; new step
headings must be pinned. The judge rubric is unchanged (EVAL_POLICY).
* size: fit the wave's wording under ship, review and qa parity caps
Measured after all lane merges: ship -79, review -210, qa -68 bytes over.
Trimmed the added wording first (post-expiry rule, MISSING COVERAGE,
plan-gate default, value-card JSON, stop line) and folded /review's
redundant INFORMATIONAL sentence and findings bullet. ship 268,036 (room
8), review 130,118 (room 1). qa needed 11 more bytes: cap 1.11 -> 1.1102
with the measured 93,305 / 84,049 = 1.11013 and its reason beside it.
* docs: troubleshooting sections for the wave's new messages
Bun floor (setup, auto-update, /gstack-upgrade), /review stale base and
fingerprint temp-dir errors, the design image model override, the
unrecorded Chromium PID, and a contributor section for census report
lines (pass-rates flags, failure causes, headroom, cost unknown, overlay
v4, stale corpora, auq substance). A free test checks every
troubleshooting.md#anchor printed by setup or bin scripts exists.
* TODOS: reliability follow-ups deferrals; close the Bun floor, make-pdf $P and #2709 reap
* 0.4: 'A census went red' guide; docs for causes, session ledger, headroom and new messages
docs/evals/census-red.md walks one red line to a diagnosis: prerequisites, the
report fields, eval:pass-rates --run/--reds/--headroom with expected output,
what unknown/insufficient/unrecorded data means, free replay, and the single
'after a repair' paid run allowed by TESTING_INTERNALS. Linked from
CONTRIBUTING, TESTING_INTERNALS and both census reports.
TESTING_INTERNALS documents the failure_cause glossary (one row per
FAILURE_CAUSES value, diagnostic only under EVAL_POLICY v1), failure_detail,
sessions[], cost_known, the session ledger, the pass-rates views and the 85%
headroom alarm, main-only tracking issues, overlay contract v4, auq-matrix's
3-sample substance panel and the E5/E6 periodic cases. troubleshooting.md
gains #bun-too-old and an anchor per new user-facing message from lanes P, D
and F. README names GSTACK_DESIGN_IMAGE_MODEL. test/census-red-docs.test.ts
pins the glossary to FAILURE_CAUSES, parses every guide command and checks the
links.
* v1.91.20.0: CHANGELOG and version for the reliability follow-ups
* regen: SKILL.md and sections for all hosts, digest, context-budget fixture
bun run gen:skill-docs --host all; goldens and llms.txt re-rendered
unchanged; bun test/helpers/capture-context-budget.ts.
* integrate: register codex-multiblock-live and design-model-smoke (periodic, rule)
* 0.4: census report and eval:pass-rates views link the census-red guide
* test: cookie approval fixture reverses the Bun install-hint pin (E1)
The historical approved cookie-workflow prompt carried BUN_VERSION=1.3.10;
the regenerated install hint now pins 1.4.0, so the fixture's reversal
list gains that edit like the earlier waves' entries.
* test: release floor counts the two new periodic cases (E5, E6)
* fix: /freeze, /guard, /unfreeze and /investigate work on Codex and other env-var hosts
Templates quoted "$HOME/.claude/skills/gstack/...", which the env-var hosts'
rewrites turned into "$HOME/.agents/skills/gstack/..." on Codex (a path a
global install never has; same class on the other hosts), and setup's
runtime roots had no freeze/bin. Quoted runtime paths now use
{{RUNTIME_ROOT}}: $GSTACK_ROOT on env-var hosts (the fence gets the
runtime prelude), the unchanged $HOME path on Claude. _link_runtime_dists
links freeze/bin and the careful/bin helper it sources into every
env-var host's root. Also fixes the same guess in /benchmark-models,
/retro and /diagram. test/env-host-runtime-assets.test.ts fails on the
previous code: no fence guesses a $HOME/<dir>/skills/gstack path, every
$GSTACK_ROOT path a fence uses is in the host's runtime root, and the
installed Codex /freeze and /unfreeze fences run from a fresh shell.
Template-reading tests now read the Claude render or expect the
placeholder (deliberate change).
* CHANGELOG/TODOS: lane W's workflow, cases and guide; the env-var host freeze fix
* E6: a block that reads a SKILL.md is not a missing install
The first live codex-multiblock-live run passed its real check (the /learn
block found the install and printed the seeded learning) but the oracle
flagged a sed of the learn SKILL.md, whose prelude source quotes the
'gstack: no install found (tried $_r)' text. Only an emitted line that
starts with the message and names a real root counts now; free replay
negative added.
* test: qa-only fixture keeps its caller-set 30 s deadline
The qa-only-no-fix fixture tells the model to start the guard with 30 s
(a caller limit), so its deadline assertion's default stays 30 000 ms; the
product's Browser Quick default (180 s) does not apply to it. Reverts the
helper and its test to their base.
* C2: keep PROBE_DIR 'owned' (qa-functional-prompt pins it); qa measured 93,311
* test: paid file count and periodic floor include the two new periodic cases
* test: ship docs freshness pin checks meaning after A1's wording trim
The snapshot still hashes selected release paths (--select narrows them),
generated outputs and docs; the pin now checks that instead of the
pre-trim sentence.
* E3: reap the recorded Chromium only after the daemon has exited
With the PID now recorded (E3), browse stop reaped Chromium while the
daemon was still shutting down; the daemon read that as a crash, exited 1
and left its state file (chromium-profile-isolation's headless stop test
failed 3/3 locally, passes on base). The CLI waits up to 5 s for the daemon
to exit before reaping a survivor.
* regen: qa and qa-only exploratory sections (PROBE_DIR wording)
* B1: batching counter reads a target declared mid-line; replay corpus
GSTA-23's census 37228573062 failed plan-eng-multi-finding-batching with
reviewCount=0 after 975 s although D1-D10 were asked one per finding,
each with its saved decision-ledger record. The report wrote 'Report
destination requested by the user. Review target (fixed): `PLAN.md` ...'
and the counter read target fields only at a line start, so no brief
inherited the plan identity. A target field may now open any sentence of
its line; a mid-sentence mention or another file still gives no identity.
Floor (3), budget and kind unchanged. Corpus: 9 earlier passing censuses,
the red (now credited at D5), and a derived batching refusal; 975 s noted
in the headroom diagnosis.
* B1: shared-libs corpus labels its git path and covers the supported shape
Every census entry was recorded in CI's eval image (git 2.43), where
gstack-safe-git falls back because --no-lazy-fetch is rejected; each entry
now says so (git_path: fallback). Two synthetic supported-path entries
(HEAD and a history citation in the brief, as git >= 2.44 produces) keep
the 37193478719 pass and the 37195203538 refusal, and a test asserts both
shapes are covered and the actor reads no git-path-specific text.
* v1.91.21.0: move off v1.91.20.0 (claimed by #3031)
* test: settle nav-guard commands before asserting; make two new tests Windows-safe
navigation-guard: Bun's expect(promise).rejects slows each Playwright round-trip
to ~1s while it waits (50 page.evaluate: 43ms plain vs 46.6s inside the matcher),
so the click's 5s timeout expired before the guard saw the link-local request
(CI: 'click: Timeout 5000ms exceeded ... waiting for scheduled navigations').
Settle the command first, then assert on the refusal; a source check keeps it so.
env-host-runtime-assets: PROBE_FILES now matches either path separator.
detector-corpus ship-docsync-late-result: JSON-escape the <ROOT> substitution so a
Windows temp path's backslashes do not break JSON.parse.
* fix: CI run/job ids are not phone numbers; context-recovery fixture gets the full runtime bin
pii.phone.e164 redacted the run id in the published eval report's fetch command
(--run 37235771700), breaking the command it prints. Digit-only spans in a run/job
flag, gh run argument, actions/runs/ or /job/ URL segment, or run_id/job_id key are
now exempt; the same digits elsewhere still report.
context-recovery-artifacts: the fixture's ./bin/ lacked gstack-skill-start and
gstack-context-recovery (the preamble's runtime scripts since v1.71), so the agent
spent its 10 turns hunting for them and hit error_max_turns with no final message
(run 37235771700, 11 turns). Link the whole runtime bin; diagnostic rerun passed
in 6 turns.
* test: replay run 37237194905's late-result capture through the docs authority detector
The only failure was call 20's write to a mistyped shard root (3GanyY for 3GanyU),
which the model disclosed; the detector is correct and the A1 post-snapshot
--out under the fixture home stays permitted. Named red recorded in TODOS.
* test: keep docsync-authority in the Windows profile
The replay test's literal helper path tripped the Windows curator's shebang-spawn
pattern, dropping the whole file from the Windows profile (test-free-shards
invariant). The snapshot --out case is already covered by docsync-command-grammar,
so it is removed here, and the Linux-path replay skips on win32.
* fix: gstack-docs-candidate --help works; docs gate red lines name the denying tool failure
PR run 37239016858 ship-docsync-completion: the parent ran
'gstack-docs-candidate --help' (the helper's usage error points at --help, which
failed). That undeclared command made docsToolFailures non-empty, which denied
the atomic-replacement proof of the child's legitimate native Edit, so the red
line blamed the Edit's temp file. A replay of the capture proves the atomic
replace once --help is declared.
- the helper prints its usage for --help/-h and exits 0
- the fixture interface and command grammar admit the helper's bare --help
- the atomic-proof denial names its check and the tool failures behind it
(deny labels renumbered to their lines)
- replay with negative controls (undeclared command, temp left behind, renamed
elsewhere); diagnostic rerun of ship-docsync-completion passed
* test: retro-base-branch fixture uses main and a shallow report path
PR run 37241493377: the agent wrote retro-output.md one directory above the
requested <tmp>/retro-base/retro-output.md (it dropped the retro-base segment),
so the report check found nothing. The fixture also ran plain git init, which
makes master on CI's git while the prompt says the repo is on main; the agent
logged that mismatch as a learning. Init on main and use the temp dir itself as
the repo. Diagnostic rerun passed (18 turns, 203s).
* test: qa-quick sets a caller probe limit and drains past its session budget
C2 raised Browser Quick's native budget to 180 s. qa-quick is the only gate case
on the native budget, and its wall went 194 s (main periodic, pre-C2) -> 254 s and
274 s (this PR's runs) -> 300 s session timeout (run 37242691913): the agent now
probes every navigation target with a checkpoint, then needs ~80 s to report.
Like qa-only-no-fix, the eval now sets a caller probe limit (30 s), scopes the
fixture, and gives bun's outer timeout the session drain margin instead of racing
the 300 s session. The product's 180 s budget and the session budget are unchanged.
Diagnostic rerun passed in 195 s.
* review/ship: use main's post-expiry smoke sentence; v1.91.24.0
GSTA-23's measurement on the review workflow judge: main's 'Post-expiry smoke
rechecks are not-run.' passed 10/10; the C2 wording passed 6/10 and the
'..., even in 4c.' variant failed in PR run 37248650879. qa-probe-gates pins
main's sentence before step 4c.
* test: record the plan-design corpus test in the test-of-test baseline
#3031's ratchet now counts tests that slice a paid test file's source; the B1
plan-design-review-plan-mode replay corpus reads test/skill-e2e-design.test.ts
like its sibling corpora already listed.
* fix: --case no longer deadlocks importing the running CLI
runCaseDiagnosis loaded runPaidShards with a dynamic import of
scripts/test-paid-shards.ts; when that file is the entry it is still in its
top-level await, so the import never resolved and `--case` hung after its
first line (on main since #3031, found by the per-case measurement runs). The
CLI now passes its runner in; library callers keep the lazy import.
* test: name the in-flight file and the native error when bun dies before its summary (#3037)
* Bun 1.4.2 everywhere CI and setup pin the tested version
Bun 1.4.0 on Windows aborts about 3% of free-suite shards with native
bad-handle errors (main too; #3037); the same head on 1.4.2 had none in 180
shard runs. Every current 1.4.0 pin moves together, as bun-version-drift
requires: setup-bun in every workflow (CSO orchestration included), the eval
image (sha256 from bun-v1.4.2's SHASUMS256.txt, re-hashed locally), GitLab,
the Ubicloud free-suite setup, gstack-bun-version.sh's tested version,
engines.bun and @types/bun, the install hint in generated skills, the Dia and
cookie native-qualification pins, and docs and tests that state the current
pin. Recorded 1.4.0 observations, fixtures, CHANGELOG history and the CSO
images' oven/bun:1.3.10 base stay.
* fix: --case runs a single-case file whole
A session option naming the case (testName: 'office-hours-section-loading')
made caseSelection pick test-name mode, whose filter matched no Bun test, so
every trial would run hollow. A file that registers only this case now runs
whole before any name match is tried.
* fix(office-hours): reviewer returns a hash-bound receipt instead of echoing its verdict
The spec reviewer now Writes its verdict once, seals it with
gstack-office-hours-review seal (validates the saved JSON and prints
OFFICE_HOURS_VERDICT round=N sha256=<hash> path=<verdict>), and returns
only that line. check --receipt recomputes the hash and fails closed on a
missing, malformed, stale-round, wrong-path or mismatched receipt. The
saved file stays the sole findings inventory; validators bind the receipt
to the saved bytes and the judge reads those bytes.
* fix(office-hours): only blocking findings send a design back for another review round
Review findings carry severity blocking|minor (verdict schema v2). PASS when
no blocking findings remain; minor findings are recorded in Reviewer
Concerns and the Spec Review table, never a reason for another round.
CONVERGENCE needs a persisting blocking prior. A persisting or unverified
blocking prior cannot be relabeled minor. The 3-round cap is unchanged.
* fix(office-hours): rounds 2 and 3 are delta re-reviews of the captured design diff
prepare saves each reviewed design as round-N.design.md and gives later
reviewers the exact line diff since the previous round. A new later-round
blocking finding must cite (changed_text) an excerpt of a changed line;
seal, check and the validators verify it against the snapshots and fail
closed when a citation or snapshot is missing. Other new observations are
minor. Severity, anti-downgrade and CONVERGENCE rules are unchanged.
* test(office-hours): preservation judge accepts COMPLETED with recorded minor findings
* fix(office-hours): parent fixes only blocking findings; callers may set --max-rounds
Between rounds the parent edits for blocking findings only; minor findings
stay recorded. gstack-office-hours-review takes --max-rounds 1|2|3 (default
3) on every command; reaching it stops at MAX_ITERATIONS. The marathon
fixture sets 2 rounds so round 1 (full) and round 2 (delta) run live.
* docs(changelog): office-hours review loop changes in v1.91.28.0
* fix(cso): helper JSON keeps run URLs only in the validated workflow slot
The shared redactor now treats GitHub Actions run ids as CI metadata
(c245d1bc2), so main's #3041 invariant that the same run digits stay
redacted outside qualification.workflow held only through the phone
false positive. Make it explicit in the CSO sanitizer.
* test(plan-ceo): split-overflow progress detector accepts the skill's candidate wording
Collection never ended when the model wrote 'E1 Slack:' / 'E1-SLACK:' or
labels like 'Include in this quarter' / 'Defer with flip trigger', so the
session ran the whole review (930-1464 s) and failed on its extra findings
or the case deadline (6 of 21 runs). The candidate check now accepts a
platform-bound E-id and the action is the bucket's leading verb, as the
skill names them (Include/Defer/Cut/Hold). The semantic judge, call floor
and include/defer/cut validation are unchanged. Replays the captured CI
37374526437, t2, t9 and t15 menus with negative controls.
* test(plan-ceo): split-overflow candidate identity is the one E1-E5 token
The skill fixes no header or question wording, and shape-matching kept
missing new forms (header 'D1.1 Slack' with 'SCOPE-E1:', 'D1.1 E1 Slack',
'E1 Slack DM bot ...:', a misspelled 'Mattermst'). Identity is now exactly
one E1-E5 token across the header and the question's first line, with no
other candidate's platform named. Options rules, judge, call floor and
include/defer/cut validation are unchanged. Replays all 31 measured runs;
the quoted-row controls now quote a second ledger id.
* v1.91.30.0: move past #3044's claimed 1.91.29.0
* test: register the split-overflow wording replay as a free fixture
* fix(skills): resolve judge-cited ambiguities in /qa and /plan-eng-review
- tasks JSONL keeps the raw branch as GIT_BRANCH instead of overwriting
gstack-slug's sanitized BRANCH used for artifact names
- the Codex outside-voice block runs for ready or unverified, matching
its routing rule
- plan-eng startup names the preamble's own setup prompts; the
cross-project question waits for Step 0
- /qa 8c: a fix that applies but leaves the finding reproducing gets one
revised fix, else it is reverted; browser verification names its before
screenshot; the exploratory loop drops its duplicate 3-step summary
* fix(plan-eng): keep the startup steps in their pinned order
* fix(plan-eng): explicit spawned sessions follow the spawned rule at the scope gate; one announcement rule
A missing tool still sends the prose menu whatever the session type, but a
session the caller or host explicitly marks spawned selects the menu's
Recommendation without prose or BLOCKED. The pre-load announcement
sentence now states Step 0's single check.
* eval policy v3: judge panels gate on the per-dimension median of 3 samples
A judge dimension now passes when at least 2 of its 3 samples meet the
unchanged threshold (judgePanelMedian); any other sample count or a
non-numeric sample fails closed. Thresholds, sample count, judge model and
prompts are unchanged; panel logs show every sample and the mean, and name
the median as the gate. Applies to the skill-llm-eval panels, the
workflow-judge cache, the auq-matrix substance panel and the calibration
replicas. EVAL_POLICY 2 -> 3 and HARNESS_VERSION 1 -> 2 (verdicts change),
so every case starts a new pass-rate series. Replays the captured panels
from jobs 112032744416 and 112049856586 as replayed evidence.
4884 lines
240 KiB
TypeScript
4884 lines
240 KiB
TypeScript
import { describe, test, expect, afterAll } from 'bun:test';
|
||
import { assertSinglePreamble } from '../scripts/gen-skill-docs';
|
||
import { validateSkillFrontmatter } from '../scripts/skill-check';
|
||
import { externalHostPathLeaks } from './helpers/skill-parser';
|
||
import { COMMAND_DESCRIPTIONS } from '../browse/src/commands';
|
||
import { SNAPSHOT_FLAGS } from '../browse/src/snapshot';
|
||
import * as fs from 'fs';
|
||
import * as path from 'path';
|
||
import * as os from 'os';
|
||
import { spawnSync } from 'child_process';
|
||
import { runCapturedCommand } from './helpers/sync-command-capture';
|
||
import { expectMentions, expectTokens } from './helpers/prompt-structure';
|
||
|
||
const ROOT = path.resolve(import.meta.dir, '..');
|
||
const MAX_SKILL_DESCRIPTION_LENGTH = 1024;
|
||
|
||
// Carved-skill aware (v2 plan T9): ship is now a skeleton SKILL.md + sections/*.md.
|
||
// Read the union so assertions about content that MOVED into a section still pass.
|
||
// The skeleton is a subset of the union, so skeleton-only assertions also hold,
|
||
// and negative assertions stay safe (the absent phrases live in neither file).
|
||
function readSkillUnion(skill: string): string {
|
||
let t = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
const secDir = path.join(ROOT, skill, 'sections');
|
||
if (fs.existsSync(secDir)) {
|
||
for (const f of fs.readdirSync(secDir).sort()) {
|
||
if (f.endsWith('.md')) t += '\n' + fs.readFileSync(path.join(secDir, f), 'utf-8');
|
||
}
|
||
}
|
||
return t;
|
||
}
|
||
function readShipUnion(): string {
|
||
return readSkillUnion('ship');
|
||
}
|
||
|
||
function ordered(text: string, anchors: string[]): void {
|
||
const positions = anchors.map(anchor => text.indexOf(anchor));
|
||
expect(positions.every(position => position >= 0), `missing one of ${JSON.stringify(anchors)}`).toBe(true);
|
||
expect(positions).toEqual([...positions].sort((a, b) => a - b));
|
||
}
|
||
|
||
function readExternalSkillUnion(output: string, hostSubdir: string, skill: string): string {
|
||
const dir = path.join(output, hostSubdir, 'skills', `gstack-${skill}`);
|
||
const sections = path.join(dir, 'sections');
|
||
return fs.readFileSync(path.join(dir, 'SKILL.md'), 'utf-8')
|
||
+ (fs.existsSync(sections) ? fs.readdirSync(sections).sort()
|
||
.filter(file => file.endsWith('.md'))
|
||
.map(file => '\n' + fs.readFileSync(path.join(sections, file), 'utf-8')).join('') : '');
|
||
}
|
||
|
||
|
||
// Token-reduction Phase 1: the preamble's inline bash (session bookkeeping,
|
||
// config echoes, telemetry producers, artifacts sync) moved into
|
||
// bin/gstack-skill-start / bin/gstack-skill-end. The render carries a one-line
|
||
// invocation fence + interpretation prose. Assertions that pinned inline-bash
|
||
// internals now pin the scripts (the new home); render-side assertions pin the
|
||
// fence + prose. Script behavior is pinned by test/gstack-skill-start.test.ts.
|
||
const SKILL_START_SCRIPT = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
|
||
const SKILL_END_SCRIPT = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-end'), 'utf-8');
|
||
|
||
function extractDescription(content: string): string {
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
expect(fmEnd).toBeGreaterThan(0);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
const lines = frontmatter.split('\n');
|
||
let description = '';
|
||
let inDescription = false;
|
||
const descLines: string[] = [];
|
||
|
||
for (const line of lines) {
|
||
if (line.match(/^description:\s*\|?\s*$/)) {
|
||
inDescription = true;
|
||
continue;
|
||
}
|
||
if (line.match(/^description:\s*\S/)) {
|
||
return line.replace(/^description:\s*/, '').trim();
|
||
}
|
||
if (inDescription) {
|
||
if (line === '' || line.match(/^\s/)) {
|
||
descLines.push(line.replace(/^ /, ''));
|
||
} else {
|
||
break;
|
||
}
|
||
}
|
||
}
|
||
|
||
if (descLines.length > 0) {
|
||
description = descLines.join('\n').trim();
|
||
}
|
||
return description;
|
||
}
|
||
|
||
function extractMarkdownSection(content: string, heading: string): string {
|
||
const escaped = heading.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||
const startMatch = content.match(new RegExp(`^${escaped}.*$`, 'm'));
|
||
expect(startMatch?.index).toBeDefined();
|
||
const start = startMatch!.index!;
|
||
const afterHeading = start + startMatch![0].length;
|
||
const nextSection = content.slice(afterHeading).match(/\n## /);
|
||
const end = nextSection?.index === undefined
|
||
? content.length
|
||
: afterHeading + nextSection.index;
|
||
return content.slice(start, end).trim();
|
||
}
|
||
|
||
function extractPreambleBeforeWorkflow(content: string, workflowMarkers: string[]): string {
|
||
const markerIndexes = workflowMarkers
|
||
.map(marker => content.indexOf(marker))
|
||
.filter(index => index >= 0);
|
||
expect(markerIndexes.length).toBeGreaterThan(0);
|
||
return content.slice(0, Math.min(...markerIndexes));
|
||
}
|
||
|
||
function isRepoRootSymlink(candidateDir: string): boolean {
|
||
try {
|
||
return fs.realpathSync(candidateDir) === fs.realpathSync(ROOT);
|
||
} catch {
|
||
return false;
|
||
}
|
||
}
|
||
|
||
// Dynamic template discovery — matches the generator's findTemplates() behavior.
|
||
// New skills automatically get test coverage without updating a static list.
|
||
const ALL_SKILLS = (() => {
|
||
const skills: Array<{ dir: string; name: string }> = [];
|
||
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
||
skills.push({ dir: '.', name: 'root gstack' });
|
||
}
|
||
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
||
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||
if (fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) {
|
||
skills.push({ dir: entry.name, name: entry.name });
|
||
}
|
||
}
|
||
return skills;
|
||
})();
|
||
|
||
// The claude host deliberately skips some skills (skipSkills — e.g. the
|
||
// /claude-code outside-voice skill exists only for non-Claude hosts), so those
|
||
// dirs have a SKILL.md.tmpl but no generated claude-host SKILL.md on a fresh
|
||
// checkout. Every generated-file assertion must exclude them or it is red on
|
||
// every clean clone (it was, invisibly, until the free suite ran in CI).
|
||
import { getHostConfig as __getHostConfig } from '../hosts/index';
|
||
const CLAUDE_SKIPPED = new Set(__getHostConfig('claude').generation.skipSkills ?? []);
|
||
const CLAUDE_GENERATED_SKILLS = ALL_SKILLS.filter(s => !CLAUDE_SKIPPED.has(s.dir));
|
||
|
||
// ─── Out-dir render isolation ────────────────────────────────
|
||
// Every generator invocation in this file that used to regenerate the live
|
||
// tree (the gitignored .agents/.factory/... host dirs included) now renders
|
||
// into this module-level out-dir: ONE `--host all` render covers the claude
|
||
// host plus every external host, and all golden-artifact reads plus the
|
||
// per-host `--dry-run` determinism checks point here. The tracked tree is
|
||
// only ever READ (the `generated files are fresh` dry-run deliberately
|
||
// compares against the committed files — that is a read, not a write).
|
||
// Out-dir renders of external hosts are byte-identical to in-place renders
|
||
// (pinned by test/gen-skill-docs-out-dir.test.ts).
|
||
const EXTERNAL_OUT = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-gen-docs-out-'));
|
||
{
|
||
const render = Bun.spawnSync(
|
||
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--out-dir', EXTERNAL_OUT],
|
||
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 },
|
||
);
|
||
if (render.exitCode !== 0) {
|
||
throw new Error(
|
||
`gen-skill-docs --host all --out-dir failed (exit ${render.exitCode}):\n${render.stderr.toString()}`,
|
||
);
|
||
}
|
||
}
|
||
afterAll(() => {
|
||
fs.rmSync(EXTERNAL_OUT, { recursive: true, force: true });
|
||
});
|
||
|
||
describe('gen-skill-docs', () => {
|
||
// Browse carve (token-reduction Phase 4): the command reference + snapshot
|
||
// flags render into browse/sections/command-list.md now — read the
|
||
// skeleton+sections union so these pins hold across the carve.
|
||
test('generated SKILL.md contains all command categories', () => {
|
||
const content = readSkillUnion('browse');
|
||
const categories = new Set(Object.values(COMMAND_DESCRIPTIONS).map(d => d.category));
|
||
for (const cat of categories) {
|
||
expect(content).toContain(`### ${cat}`);
|
||
}
|
||
});
|
||
|
||
test('generated SKILL.md contains all commands', () => {
|
||
const content = readSkillUnion('browse');
|
||
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
||
const display = meta.usage || cmd;
|
||
expect(content).toContain(display);
|
||
}
|
||
});
|
||
|
||
test('command table is sorted alphabetically within categories', () => {
|
||
const content = readSkillUnion('browse');
|
||
// Extract command names from the Navigation section as a test
|
||
const navSection = content.match(/### Navigation\n\|.*\n\|.*\n([\s\S]*?)(?=\n###|\n## )/);
|
||
expect(navSection).not.toBeNull();
|
||
const rows = navSection![1].trim().split('\n');
|
||
const commands = rows.map(r => {
|
||
const match = r.match(/\| `(\w+)/);
|
||
return match ? match[1] : '';
|
||
}).filter(Boolean);
|
||
const sorted = [...commands].sort();
|
||
expect(commands).toEqual(sorted);
|
||
});
|
||
|
||
test('snapshot flags section contains all flags', () => {
|
||
const content = readSkillUnion('browse');
|
||
for (const flag of SNAPSHOT_FLAGS) {
|
||
expect(content).toContain(flag.short);
|
||
expect(content).toContain(flag.description);
|
||
}
|
||
});
|
||
|
||
test('every skill has a SKILL.md.tmpl template', () => {
|
||
for (const skill of ALL_SKILLS) {
|
||
const tmplPath = path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
||
expect(fs.existsSync(tmplPath)).toBe(true);
|
||
}
|
||
});
|
||
|
||
test('every skill has a generated SKILL.md with auto-generated header', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const mdPath = path.join(ROOT, skill.dir, 'SKILL.md');
|
||
expect(fs.existsSync(mdPath)).toBe(true);
|
||
const content = fs.readFileSync(mdPath, 'utf-8');
|
||
expect(content).toContain('AUTO-GENERATED from SKILL.md.tmpl');
|
||
expect(content).toContain('Regenerate: bun run gen:skill-docs');
|
||
}
|
||
});
|
||
|
||
// #1778: strict YAML parsers (Codex/OpenAI skill loading) reject frontmatter
|
||
// whose plain `description:` scalar contains an interior ": " (read as a nested
|
||
// mapping). Parse EVERY generated frontmatter block with a strict YAML parser,
|
||
// not just string-check that name:/description: exist.
|
||
test('every generated SKILL.md frontmatter parses as strict YAML', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
expect(validateSkillFrontmatter(content),
|
||
`frontmatter for ${skill.dir} must be valid YAML with string name/description`).toEqual([]);
|
||
}
|
||
});
|
||
|
||
test('every generated Codex (.agents/skills) frontmatter parses as strict YAML', () => {
|
||
// Reads the module-level out-dir render (guaranteed present — the render
|
||
// throws at module load if it fails), never the live gitignored tree.
|
||
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
||
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
||
if (!entry.isDirectory()) continue;
|
||
const mdPath = path.join(agentsDir, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(mdPath)) continue;
|
||
expect(validateSkillFrontmatter(fs.readFileSync(mdPath, 'utf-8')),
|
||
`Codex frontmatter for ${entry.name} must be valid YAML with string name/description`).toEqual([]);
|
||
}
|
||
});
|
||
|
||
test(`every generated SKILL.md description stays within ${MAX_SKILL_DESCRIPTION_LENGTH} chars`, () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
const description = extractDescription(content);
|
||
expect(description.length).toBeLessThanOrEqual(MAX_SKILL_DESCRIPTION_LENGTH);
|
||
}
|
||
});
|
||
|
||
test(`every Codex SKILL.md description stays within ${MAX_SKILL_DESCRIPTION_LENGTH} chars`, () => {
|
||
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
||
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
||
if (!entry.isDirectory()) continue;
|
||
const skillMd = path.join(agentsDir, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
const description = extractDescription(content);
|
||
expect(description.length).toBeLessThanOrEqual(MAX_SKILL_DESCRIPTION_LENGTH);
|
||
}
|
||
});
|
||
|
||
test('every Codex SKILL.md description stays under 900-char warning threshold', () => {
|
||
const WARN_THRESHOLD = 900;
|
||
const agentsDir = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
||
const violations: string[] = [];
|
||
for (const entry of fs.readdirSync(agentsDir, { withFileTypes: true })) {
|
||
if (!entry.isDirectory()) continue;
|
||
const skillMd = path.join(agentsDir, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
const description = extractDescription(content);
|
||
if (description.length > WARN_THRESHOLD) {
|
||
violations.push(`${entry.name}: ${description.length} chars (limit ${MAX_SKILL_DESCRIPTION_LENGTH}, ${MAX_SKILL_DESCRIPTION_LENGTH - description.length} remaining)`);
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
|
||
test('package.json version matches VERSION file (npm-valid translation)', () => {
|
||
// Decision 11 (v1.67 wave): VERSION stays the 4-digit source of truth;
|
||
// package.json carries the npm-valid 3-digit translation (npm rejects a
|
||
// fourth component). The pre-v1.67 1:1 four-digit mirror is also accepted
|
||
// (grandfathered until the next write), matching gstack-version-bump's
|
||
// own drift contract.
|
||
const pkg = JSON.parse(fs.readFileSync(path.join(ROOT, 'package.json'), 'utf-8'));
|
||
const version = fs.readFileSync(path.join(ROOT, 'VERSION'), 'utf-8').trim();
|
||
const npmTranslation = version.split('.').slice(0, 3).join('.');
|
||
expect([npmTranslation, version]).toContain(pkg.version);
|
||
});
|
||
|
||
test('generated files are fresh (match --dry-run)', () => {
|
||
// Deliberately compares against the LIVE TRACKED SKILL.md files (no
|
||
// --out-dir): this is the freshness gate for the committed tree. Dry-run
|
||
// writes nothing — it is a read.
|
||
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--dry-run'], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
// Every skill should be FRESH
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const file = skill.dir === '.' ? 'SKILL.md' : `${skill.dir}/SKILL.md`;
|
||
expect(output).toContain(`FRESH: ${file}`);
|
||
}
|
||
expect(output).not.toContain('STALE');
|
||
});
|
||
|
||
test('no generated SKILL.md contains unresolved placeholders', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
const unresolved = content.match(/\{\{\w+\}\}/g);
|
||
expect(unresolved).toBeNull();
|
||
}
|
||
});
|
||
|
||
test('templates contain placeholders', () => {
|
||
// P2 (v1.2.0): the root template is a pure router — only {{PREAMBLE}}.
|
||
// The browse command/snapshot placeholders live in browse/SKILL.md.tmpl now.
|
||
const rootTmpl = fs.readFileSync(path.join(ROOT, 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(rootTmpl).toContain('{{PREAMBLE}}');
|
||
expect(rootTmpl).not.toContain('{{COMMAND_REFERENCE}}');
|
||
expect(rootTmpl).not.toContain('{{SNAPSHOT_FLAGS}}');
|
||
|
||
// Browse carve: the reference resolvers moved into the on-demand section
|
||
// template (so gen-skill-docs keeps them fresh from browse/src); the
|
||
// skeleton points at the section instead of inlining the reference.
|
||
const browseTmpl = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(browseTmpl).not.toContain('{{COMMAND_REFERENCE}}');
|
||
expect(browseTmpl).not.toContain('{{SNAPSHOT_FLAGS}}');
|
||
expect(browseTmpl).toContain('{{SECTION:command-list}}');
|
||
expect(browseTmpl).toContain('{{PREAMBLE}}');
|
||
|
||
const browseSectionTmpl = fs.readFileSync(
|
||
path.join(ROOT, 'browse', 'sections', 'command-list.md.tmpl'), 'utf-8');
|
||
expect(browseSectionTmpl).toContain('{{COMMAND_REFERENCE}}');
|
||
expect(browseSectionTmpl).toContain('{{SNAPSHOT_FLAGS}}');
|
||
|
||
// Aside is the primary browser: every browsing skill renders the Aside
|
||
// contract ({{ASIDE_SETUP}}); the browse binary is its fallback.
|
||
const qaTmpl = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(qaTmpl).not.toContain('{{ASIDE_SETUP}}');
|
||
expect(qaTmpl).toContain('{{SECTION:browser-setup}}');
|
||
expect(fs.readFileSync(path.join(ROOT, 'qa/sections/browser-setup.md.tmpl'), 'utf8'))
|
||
.toContain('{{ASIDE_SETUP}}');
|
||
expect(browseTmpl).toContain('{{ASIDE_SETUP}}');
|
||
});
|
||
|
||
test('generated SKILL.md contains operational self-improvement (replaced contributor mode)', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('Contributor Mode');
|
||
expect(content).not.toContain('gstack_contributor');
|
||
expect(content).not.toContain('contributor-logs');
|
||
expect(content).toContain('Operational Self-Improvement');
|
||
expect(content).toContain('gstack-learnings-log');
|
||
// The learnings-resurface call moved from the inline preamble bash into
|
||
// the skill-start script (Phase 1) — same command, new home.
|
||
expect(SKILL_START_SCRIPT).toContain('gstack-learnings-search" --limit 3');
|
||
});
|
||
|
||
test('generated SKILL.md with LEARNINGS_LOG contains operational type', () => {
|
||
// Check a skill that has LEARNINGS_LOG (e.g., review)
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('operational');
|
||
});
|
||
|
||
test('session awareness lives in gstack-skill-start (registry touch + stale cleanup)', () => {
|
||
// The sessions registry moved from inline preamble bash into the script:
|
||
// it records the harness pid (--parent-pid identity) and expires entries
|
||
// older than 120 minutes.
|
||
expect(SKILL_START_SCRIPT).toContain('sessions/$PARENT_PID');
|
||
expect(SKILL_START_SCRIPT).toContain('-mmin +120');
|
||
// The render keeps the completion-status protocol the sessions feed into.
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('RECOMMENDATION');
|
||
});
|
||
|
||
test('branch detection lives in gstack-skill-start and is echoed as BRANCH', () => {
|
||
expect(SKILL_START_SCRIPT).toContain('_BRANCH=$(git branch --show-current');
|
||
expect(SKILL_START_SCRIPT).toContain('echo "BRANCH: $_BRANCH"');
|
||
});
|
||
|
||
// #2001: update_check: false silences the binary but the upgrade-handling
|
||
// instruction prose used to ship unconditionally. Token-reduction Phase 2
|
||
// made the gate STRUCTURAL: the prose left the renders entirely (absence is
|
||
// pinned by test/onboarding-moved-literals.test.ts) and now emits from
|
||
// gstack-skill-start's instruction layer ONLY when the update-check binary
|
||
// produced output — and that binary silences itself on update_check=false.
|
||
// Opted-out installs can never see the prose, by construction.
|
||
test('update_check opt-out gates the update binary and upgrade-flow emission (issue #2001)', () => {
|
||
// The config-echo cluster lives in gstack-skill-start: the flag is still
|
||
// read and echoed as a STATUS line for the model.
|
||
expect(SKILL_START_SCRIPT, 'script must read update_check config').toContain('_UPDATE_CHECK=$(');
|
||
expect(SKILL_START_SCRIPT, 'script must echo UPDATE_CHECK').toContain('echo "UPDATE_CHECK: $_UPDATE_CHECK"');
|
||
// Gate half 1: the update-check binary exits silently when opted out.
|
||
const updateCheck = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-update-check'), 'utf-8');
|
||
expect(updateCheck, 'binary must read update_check config').toContain('get update_check');
|
||
expect(updateCheck, 'binary must exit silently on update_check=false')
|
||
.toMatch(/if \[ "\$_UC" = "false" \]; then\n\s*exit 0/);
|
||
// Gate half 2: the upgrade-flow instruction block emits only when the
|
||
// binary emitted something (empty when opted out, cached, or up to date).
|
||
expect(SKILL_START_SCRIPT, 'upgrade-flow must be gated on update-check output')
|
||
.toMatch(/if \[ -n "\$_UPD" \]; then\n\s*_emit_block upgrade-flow/);
|
||
});
|
||
|
||
test('tier 2+ skills contain ELI10 simplification rules (AskUserQuestion format)', () => {
|
||
// Root SKILL.md is tier 1 and CSO intentionally uses a private startup with
|
||
// no shared PREAMBLE. Check a regular tier 2+ PREAMBLE consumer instead.
|
||
// v1.7.0.0 Pros/Cons format uses "ELI10 (ALWAYS)" rather than "Simplify (ELI10".
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('ELI10');
|
||
expect(content).toContain('plain English');
|
||
expect(content).toContain('not function names');
|
||
});
|
||
|
||
test('tier 1 skills do NOT contain AskUserQuestion format', () => {
|
||
// Use benchmark (tier 1) instead of root — root SKILL.md gets overwritten by Codex test setup
|
||
const content = fs.readFileSync(path.join(ROOT, 'benchmark', 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('## AskUserQuestion Format');
|
||
expect(content).not.toContain('## Completeness Principle');
|
||
});
|
||
|
||
test('telemetry producer lives in the scripts; render documents the analytics sink', () => {
|
||
// The skill-usage.jsonl producers moved into the scripts (Phase 1).
|
||
expect(SKILL_START_SCRIPT).toContain('analytics/skill-usage.jsonl');
|
||
expect(SKILL_END_SCRIPT).toContain('analytics/skill-usage.jsonl');
|
||
// The render still tells the model where telemetry lands.
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('$GSTACK_STATE_ROOT/analytics/');
|
||
});
|
||
|
||
test('plan-review generated preambles stay under the Option A budget', () => {
|
||
const reviewSkills = [
|
||
{
|
||
path: path.join(ROOT, 'plan-ceo-review', 'SKILL.md'),
|
||
markers: ['# Mega Plan Review Mode', '## Step 0: Detect platform and base branch'],
|
||
},
|
||
{
|
||
path: path.join(ROOT, 'plan-eng-review', 'SKILL.md'),
|
||
markers: ['# Plan Review Mode'],
|
||
},
|
||
];
|
||
|
||
// Plan skills carry the same preamble surface as other tier-≥2 skills
|
||
// (Artifacts Sync, Context Recovery, Routing Injection are load-bearing
|
||
// functionality, not optional). Budget is set to current size + small
|
||
// headroom; ratchet down if a future slim trims real bytes.
|
||
// Ratcheted from 33000 → 35000 when the gbrain context-load block was
|
||
// added (per /sync-gbrain plan §4). Ratcheted 35000 → 36500 in v1.27.0.0
|
||
// when generate-brain-sync-block.ts gained the gbrain_mcp_mode probe +
|
||
// remote-mode ARTIFACTS_SYNC status line (Path 4 of /setup-gbrain).
|
||
// Ratcheted 36500 → 39000 in the contributor wave when #1205 added the
|
||
// \\u-escape CJK rule (rule 12 + self-check item) to the AskUserQuestion
|
||
// preamble.
|
||
// Ratcheted 39000 → 40000 in plan-tune cathedral T14: question-tuning
|
||
// resolver gained the <gstack-qid:...> marker convention + the
|
||
// (recommended) label requirement (D2 + D18 — both load-bearing for
|
||
// hook enforcement). Adds ~700 bytes.
|
||
// Ratcheted 40000 → 60000 in v1.52.0.0 cap audit: ~20K headroom so
|
||
// future preamble adds don't trip the gate on each PR. Real runaway
|
||
// (preamble doubling) still trips; normal scope growth doesn't.
|
||
for (const skill of reviewSkills) {
|
||
const content = fs.readFileSync(skill.path, 'utf-8');
|
||
const preamble = extractPreambleBeforeWorkflow(content, skill.markers);
|
||
expect(Buffer.byteLength(preamble, 'utf-8')).toBeLessThan(60_000);
|
||
}
|
||
});
|
||
|
||
test('voice and writing-style preamble sections stay compact', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
const voice = extractMarkdownSection(content, '## Voice');
|
||
const writingStyle = extractMarkdownSection(content, '## Writing Style');
|
||
|
||
expect(Buffer.byteLength(voice, 'utf-8')).toBeLessThan(3_000);
|
||
expect(Buffer.byteLength(writingStyle, 'utf-8')).toBeLessThan(2_000);
|
||
});
|
||
|
||
test('slim voice section preserves the gstack voice contract', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
const voice = extractMarkdownSection(content, '## Voice');
|
||
|
||
expect(voice).toMatch(/lead with the point|direct/i);
|
||
expect(voice).toMatch(/file|function|line|command|real numbers/i);
|
||
expect(voice).toMatch(/user.*outcome|user.*experience|real user/i);
|
||
expect(voice).toMatch(/corporate|academic|PR|hype/i);
|
||
expect(voice).toMatch(/AI vocabulary|delve|crucial|robust/i);
|
||
expect(voice).toMatch(/user decides|user.*context|sovereignty|recommendation, not a decision/i);
|
||
});
|
||
|
||
test('preamble .pending-* glob is zsh-safe (uses find, not shell glob)', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
if (!content.includes('.pending-')) continue;
|
||
// Must NOT have a bare shell glob ".pending-*" outside of find's -name argument
|
||
expect(content).not.toMatch(/for _PF in [^\n]*\/\.pending-\*/);
|
||
// Must use find to avoid zsh NOMATCH error on glob expansion
|
||
expect(content).toContain("find ~/.gstack/analytics -maxdepth 1 -name '.pending-*'");
|
||
}
|
||
});
|
||
|
||
test('bash blocks with shell globs are zsh-safe (setopt guard or find)', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
const bashBlocks = [...content.matchAll(/```bash\n([\s\S]*?)```/g)].map(m => m[1]);
|
||
|
||
for (const block of bashBlocks) {
|
||
const lines = block.split('\n');
|
||
|
||
for (const line of lines) {
|
||
const trimmed = line.trimStart();
|
||
if (trimmed.startsWith('#')) continue;
|
||
if (!trimmed.includes('*')) continue;
|
||
// Skip lines where * is inside find -name, git pathspecs, or $(find)
|
||
if (/\bfind\b/.test(trimmed)) continue;
|
||
if (/\bgit\b/.test(trimmed)) continue;
|
||
if (/\$\(find\b/.test(trimmed)) continue;
|
||
|
||
// Check 1: "for VAR in <glob>" must use $(find ...) — caught above by the
|
||
// $(find check, so any surviving for-in with a glob pattern is a violation
|
||
if (/\bfor\s+\w+\s+in\b/.test(trimmed) && /\*\./.test(trimmed)) {
|
||
throw new Error(
|
||
`Unsafe for-in glob in ${skill.dir}/SKILL.md: "${trimmed}". ` +
|
||
`Use \`for f in $(find ... -name '*.ext')\` for zsh compatibility.`
|
||
);
|
||
}
|
||
|
||
// Check 2: ls/cat/rm/grep with glob file args must have setopt guard
|
||
const isGlobCmd = /\b(?:ls|cat|rm|grep)\b/.test(trimmed) &&
|
||
/(?:\/\*[a-z.*]|\*\.[a-z])/.test(trimmed);
|
||
if (isGlobCmd) {
|
||
expect(block).toContain('setopt +o nomatch');
|
||
}
|
||
}
|
||
}
|
||
}
|
||
});
|
||
|
||
test('preamble-using skills have correct skill name in telemetry', () => {
|
||
const PREAMBLE_SKILLS = [
|
||
{ dir: '.', name: 'gstack' },
|
||
{ dir: 'ship', name: 'ship' },
|
||
{ dir: 'review', name: 'review' },
|
||
{ dir: 'qa', name: 'qa' },
|
||
{ dir: 'retro', name: 'retro' },
|
||
];
|
||
for (const skill of PREAMBLE_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
// The skill name now travels as --skill into gstack-skill-start (the
|
||
// preamble fence) and gstack-skill-end (the telemetry epilogue) — the
|
||
// scripts write it into the JSONL events.
|
||
expect(content, `${skill.dir} preamble fence must pass its own name`)
|
||
.toMatch(new RegExp(`--skill "${skill.name}" --model`));
|
||
expect(content, `${skill.dir} epilogue must pass its own name`)
|
||
.toContain(`gstack-skill-end --skill "${skill.name}"`);
|
||
}
|
||
});
|
||
|
||
test('qa and qa-only load the shared QA_METHODOLOGY through exploration', () => {
|
||
const qaSkeletonTmpl = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(qaSkeletonTmpl).toContain('{{SECTION:exploratory}}');
|
||
expect(qaSkeletonTmpl).not.toContain('{{QA_METHOD_READS}}');
|
||
expect(qaSkeletonTmpl).not.toContain('{{QA_METHODOLOGY}}');
|
||
const qaSectionTmpl = fs.readFileSync(path.join(ROOT, 'qa', 'sections', 'qa-patterns.md.tmpl'), 'utf-8');
|
||
expect(qaSectionTmpl).toContain('{{QA_METHODOLOGY}}');
|
||
|
||
const qaOnlyTmpl = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(qaOnlyTmpl).not.toContain('{{QA_METHODOLOGY}}');
|
||
expect(qaOnlyTmpl).toContain('{{SECTION:exploratory}}');
|
||
expect(qaOnlyTmpl).not.toContain('{{QA_METHOD_READS}}');
|
||
expect(qaOnlyTmpl).toContain('Load the shared preparation gate now');
|
||
for (const skill of ['qa', 'qa-only']) {
|
||
expect(fs.readFileSync(path.join(ROOT, skill, 'sections/exploratory.md.tmpl'), 'utf8'))
|
||
.toContain('{{QA_EXPLORATORY}}');
|
||
const entry = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf8');
|
||
const explorer = fs.readFileSync(path.join(ROOT, skill, 'sections/exploratory.md'), 'utf8');
|
||
const directoryRead = skill === 'qa'
|
||
? 'Read `sections/scope.md` relative to the installed `qa`/`gstack-qa` SKILL.md directory'
|
||
: "Use this host's installed `qa`/`gstack-qa` SKILL.md directory for these reads";
|
||
expect(entry).toContain('sections/exploratory.md');
|
||
expect(explorer).toContain(directoryRead);
|
||
for (const method of ['system-functional', 'qa-patterns']) {
|
||
const methodRead = `Read \`sections/${method}.md\` in full.`;
|
||
expect(entry).not.toContain(methodRead);
|
||
expect(explorer).toContain(methodRead);
|
||
expect(explorer.split(methodRead)).toHaveLength(2);
|
||
expect(explorer.indexOf(directoryRead)).toBeLessThan(explorer.indexOf(methodRead));
|
||
const directory = path.resolve(ROOT, skill, skill === 'qa-only' ? '../qa' : '.');
|
||
expect(fs.realpathSync(path.join(directory, 'sections', `${method}.md`)))
|
||
.toBe(path.join(ROOT, 'qa', 'sections', `${method}.md`));
|
||
}
|
||
expect(explorer).toContain('**Browser surfaces only:**\nRead `sections/qa-patterns.md` in full.');
|
||
expectMentions(explorer, [['before','complete','charters']], 'explorer');
|
||
expectMentions(explorer, [['do not','invocation','completed']], 'explorer');
|
||
expect(explorer.indexOf('Read `sections/qa-patterns.md`')).toBeLessThan(explorer.indexOf('Write a **charter**'));
|
||
}
|
||
});
|
||
|
||
test('QA_METHODOLOGY is expanded in the shared resource referenced by qa and qa-only', () => {
|
||
const qaContent = readSkillUnion('qa'); // carved: methodology lives in qa/sections/qa-patterns.md
|
||
const qaOnlyUnion = readSkillUnion('qa-only');
|
||
expect(qaOnlyUnion).toContain("Use this host's installed `qa`/`gstack-qa` SKILL.md directory for these reads");
|
||
expect(qaOnlyUnion).toContain('**Browser surfaces only:**\nRead `sections/qa-patterns.md` in full.');
|
||
expect(qaOnlyUnion).not.toContain('Health Score Rubric');
|
||
const qaOnlyContent = qaOnlyUnion + fs.readFileSync(path.join(ROOT, 'qa/sections/qa-patterns.md'), 'utf8');
|
||
|
||
// Both should contain the health score rubric
|
||
expect(qaContent).toContain('Health Score Rubric');
|
||
expect(qaOnlyContent).toContain('Health Score Rubric');
|
||
|
||
// Both should contain framework guidance
|
||
expect(qaContent).toContain('Framework-Specific Guidance');
|
||
expect(qaOnlyContent).toContain('Framework-Specific Guidance');
|
||
|
||
// Both should contain the important rules
|
||
expect(qaContent).toContain('Important Rules');
|
||
expect(qaOnlyContent).toContain('Important Rules');
|
||
|
||
// Both should contain the 6 phases
|
||
expect(qaContent).toContain('Phase 1');
|
||
expect(qaOnlyContent).toContain('Phase 1');
|
||
expect(qaContent).toContain('Phase 6');
|
||
expect(qaOnlyContent).toContain('Phase 6');
|
||
});
|
||
|
||
test('qa-only has no-fix guardrails', () => {
|
||
const qaOnlyContent = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md'), 'utf-8');
|
||
expect(extractMarkdownSection(qaOnlyContent, '## Additional Rules (qa-only specific)')).toMatch(/never fix/i);
|
||
// Should not have Edit, Glob, or Grep in allowed-tools.
|
||
// Scope to frontmatter (between the first two --- lines) — the body can
|
||
// legitimately mention these tool names in prose (e.g., Claude model
|
||
// overlay names the host's dedicated file and search tools).
|
||
const fmMatch = qaOnlyContent.match(/^---\n([\s\S]*?)\n---/);
|
||
expect(fmMatch).not.toBeNull();
|
||
const frontmatter = fmMatch![1];
|
||
expect(frontmatter).toMatch(/allowed-tools:/);
|
||
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Edit/);
|
||
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Glob/);
|
||
expect(frontmatter).not.toMatch(/allowed-tools:[\s\S]*?- Grep/);
|
||
});
|
||
|
||
test('qa has fix-loop tools and phases', () => {
|
||
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
|
||
// Should have Edit, Glob, Grep in allowed-tools
|
||
expect(qaContent).toContain('Edit');
|
||
expect(qaContent).toContain('Glob');
|
||
expect(qaContent).toContain('Grep');
|
||
// Should have fix-loop phases
|
||
expect(qaContent).toContain('Phase 7');
|
||
expect(qaContent).toContain('Phase 8');
|
||
expect(qaContent).toContain('Fix Loop');
|
||
expect(qaContent).toContain('Triage');
|
||
expect(qaContent).toMatch(/every 5 fixes/i);
|
||
});
|
||
});
|
||
|
||
describe('BASE_BRANCH_DETECT resolver', () => {
|
||
// Find a generated SKILL.md that uses the placeholder (ship is guaranteed to)
|
||
const shipContent = readShipUnion();
|
||
|
||
test('resolver output contains PR base detection command', () => {
|
||
expect(shipContent).toContain('gh pr view --json baseRefName');
|
||
});
|
||
|
||
test('resolver output contains repo default branch detection command', () => {
|
||
expect(shipContent).toContain('gh repo view --json defaultBranchRef');
|
||
});
|
||
|
||
test('resolver output contains fallback to main', () => {
|
||
expect(shipContent).toMatch(/fall\s*back\s+to\s+`main`/i);
|
||
});
|
||
|
||
test('resolver output uses "the base branch" phrasing', () => {
|
||
expect(shipContent).toContain('the base branch');
|
||
});
|
||
|
||
test('resolver output contains GitLab CLI commands', () => {
|
||
expect(shipContent).toContain('glab');
|
||
});
|
||
|
||
test('resolver output contains git-native fallback', () => {
|
||
expect(shipContent).toContain('git symbolic-ref');
|
||
});
|
||
|
||
test('resolver output mentions GitLab platform', () => {
|
||
expect(shipContent).toMatch(/gitlab/i);
|
||
});
|
||
});
|
||
|
||
describe('GitLab support in generated skills', () => {
|
||
const retroContent = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
|
||
const shipSkillContent = readShipUnion();
|
||
|
||
test('retro contains GitLab MR number extraction', () => {
|
||
expect(retroContent).toContain('[#!]');
|
||
});
|
||
|
||
test('retro uses BASE_BRANCH_DETECT (contains glab)', () => {
|
||
expect(retroContent).toContain('glab');
|
||
});
|
||
|
||
test('ship contains glab mr create', () => {
|
||
expect(shipSkillContent).toContain('glab mr create');
|
||
});
|
||
|
||
test('ship checks .gitlab-ci.yml', () => {
|
||
expect(shipSkillContent).toContain('.gitlab-ci.yml');
|
||
});
|
||
});
|
||
|
||
/**
|
||
* Quality evals — catch description regressions.
|
||
*
|
||
* These test that generated output is *useful for an AI agent*,
|
||
* not just structurally valid. Each test targets a specific
|
||
* regression we actually shipped and caught in review.
|
||
*/
|
||
describe('description quality evals', () => {
|
||
// Regression: snapshot flags lost value hints (-d <N>, -s <sel>, -o <path>)
|
||
// Browse carve: the flag reference renders into browse/sections/command-list.md.
|
||
test('snapshot flags with values include value hints in output', () => {
|
||
const content = readSkillUnion('browse');
|
||
for (const flag of SNAPSHOT_FLAGS) {
|
||
if (flag.takesValue) {
|
||
expect(flag.valueHint).toBeDefined();
|
||
expect(content).toContain(`${flag.short} ${flag.valueHint}`);
|
||
}
|
||
}
|
||
});
|
||
|
||
// Regression: "is" lost the valid states enum
|
||
test('is command lists valid state values', () => {
|
||
const desc = COMMAND_DESCRIPTIONS['is'].description;
|
||
for (const state of ['visible', 'hidden', 'enabled', 'disabled', 'checked', 'editable', 'focused']) {
|
||
expect(desc).toContain(state);
|
||
}
|
||
});
|
||
|
||
// Regression: "press" lost common key examples
|
||
test('press command lists example keys', () => {
|
||
const desc = COMMAND_DESCRIPTIONS['press'].description;
|
||
expect(desc).toContain('Enter');
|
||
expect(desc).toContain('Tab');
|
||
expect(desc).toContain('Escape');
|
||
});
|
||
|
||
// Regression: "console" lost --errors filter note
|
||
test('console command describes --errors behavior', () => {
|
||
const desc = COMMAND_DESCRIPTIONS['console'].description;
|
||
expect(desc).toContain('--errors');
|
||
});
|
||
|
||
// Regression: snapshot -i lost "@e refs" context
|
||
test('snapshot -i mentions @e refs', () => {
|
||
const flag = SNAPSHOT_FLAGS.find(f => f.short === '-i')!;
|
||
expect(flag.description).toContain('@e');
|
||
});
|
||
|
||
// Regression: snapshot -C lost "@c refs" context
|
||
test('snapshot -C mentions @c refs', () => {
|
||
const flag = SNAPSHOT_FLAGS.find(f => f.short === '-C')!;
|
||
expect(flag.description).toContain('@c');
|
||
});
|
||
|
||
// Guard: every description must be at least 8 chars (catches empty or stub descriptions)
|
||
test('all command descriptions have meaningful length', () => {
|
||
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
||
expect(meta.description.length).toBeGreaterThanOrEqual(8);
|
||
}
|
||
});
|
||
|
||
// Guard: snapshot flag descriptions must be at least 10 chars
|
||
test('all snapshot flag descriptions have meaningful length', () => {
|
||
for (const flag of SNAPSHOT_FLAGS) {
|
||
expect(flag.description.length).toBeGreaterThanOrEqual(10);
|
||
}
|
||
});
|
||
|
||
// Guard: descriptions must not contain pipe (breaks markdown table cells)
|
||
// Usage strings are backtick-wrapped in the table so pipes there are safe.
|
||
test('no command description contains pipe character', () => {
|
||
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
||
expect(meta.description).not.toContain('|');
|
||
}
|
||
});
|
||
|
||
// Guard: generated output uses → not ->
|
||
test('generated SKILL.md uses unicode arrows', () => {
|
||
// P2 (v1.2.0): the browse body moved out of the top-level router into
|
||
// browse/SKILL.md. Guard arrow style on the browse body (sliced from its
|
||
// H1 so the auto-generated `-->` header comments are excluded).
|
||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||
const h1 = content.search(/^# browse: /m);
|
||
expect(h1).toBeGreaterThan(-1);
|
||
const body = content.slice(h1);
|
||
expect(body).toContain('→');
|
||
expect(body).not.toContain('->');
|
||
});
|
||
});
|
||
|
||
describe('REVIEW_DASHBOARD resolver', () => {
|
||
const REVIEW_SKILLS = ['plan-ceo-review', 'plan-eng-review', 'plan-design-review'];
|
||
|
||
for (const skill of REVIEW_SKILLS) {
|
||
test(`review dashboard appears in ${skill} generated file`, () => {
|
||
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
||
expect(content).toContain('gstack-review');
|
||
expect(content).toContain('REVIEW READINESS DASHBOARD');
|
||
});
|
||
}
|
||
|
||
test('review dashboard appears in ship generated file', () => {
|
||
const content = readShipUnion();
|
||
// The dashboard reads the review log through its reader; the literal
|
||
// reviews.jsonl filename used to come only from the Context Recovery fence.
|
||
expect(extractMarkdownSection(content, '## Review Readiness Dashboard')).toContain('bin/gstack-review-read');
|
||
expect(content).toContain('REVIEW READINESS DASHBOARD');
|
||
});
|
||
|
||
test('dashboard treats review as a valid Eng Review source', () => {
|
||
const content = readShipUnion();
|
||
expect(content).toContain('| Eng Review | `review` or `plan-eng-review` | (DIFF) or (PLAN) |');
|
||
expect(content).toContain('**Content-first rule:** For `review`');
|
||
expect(content).toContain('**Plan records** (plan-ceo-review, plan-eng-review');
|
||
expect(content.replace(/\s+/g, ' ')).toContain('CLEARED requires the selected Eng Review to be `clean`, within 7 days and fresh under step 2');
|
||
});
|
||
|
||
test('shared dashboard propagates review source to plan-eng-review', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
expect(content).toContain('| Eng Review | `review` or `plan-eng-review` | (DIFF) or (PLAN) |');
|
||
expect(content).toContain('**Content-first rule:** For `review`');
|
||
});
|
||
|
||
test('resolver output contains key dashboard elements', () => {
|
||
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
||
expect(content).toContain('VERDICT');
|
||
expect(content).toContain('CLEARED');
|
||
expect(content).toContain('Eng Review');
|
||
expect(content).toContain('7 days');
|
||
expect(content).toContain('Design Review');
|
||
expect(content).toContain('skip_eng_review');
|
||
});
|
||
|
||
test('dashboard bash block includes git HEAD for staleness detection', () => {
|
||
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
||
expect(content).toContain('git rev-parse --short HEAD');
|
||
expect(content).toContain('---HEAD---');
|
||
});
|
||
|
||
test('dashboard includes staleness detection prose', () => {
|
||
const content = readSkillUnion('plan-ceo-review'); // carved: dashboard moved to section
|
||
expect(content).toContain('**2. Check freshness before choosing a verdict.**');
|
||
expect(content).toContain('commit');
|
||
});
|
||
|
||
for (const skill of REVIEW_SKILLS) {
|
||
test(`${skill} contains review chaining section`, () => {
|
||
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
||
expect(content).toContain('Review Chaining');
|
||
});
|
||
|
||
test(`${skill} Review Log includes commit field`, () => {
|
||
const content = readSkillUnion(skill); // carved skills: union skeleton + sections
|
||
expect(content).toContain('"commit"');
|
||
});
|
||
}
|
||
|
||
test('plan-ceo-review chaining mentions eng and design reviews', () => {
|
||
// Carved skill: the chaining prose lives in sections/*.md. (It used to
|
||
// pass against the skeleton only because the preamble's routing-injection
|
||
// rules incidentally named these skills — that prose moved into
|
||
// bin/gstack-skill-start in token-reduction Phase 2.)
|
||
const content = readSkillUnion('plan-ceo-review');
|
||
expect(content).toContain('/plan-eng-review');
|
||
expect(content).toContain('/plan-design-review');
|
||
});
|
||
|
||
test('plan-eng-review chaining mentions design and ceo reviews', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
expect(content).toContain('/plan-design-review');
|
||
expect(content).toContain('/plan-ceo-review');
|
||
});
|
||
|
||
test('plan-design-review chaining mentions eng, ceo, and design skills', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('/plan-eng-review');
|
||
expect(content).toContain('/plan-ceo-review');
|
||
expect(content).toContain('/design-shotgun');
|
||
expect(content).toContain('/design-html');
|
||
});
|
||
|
||
test('ship does NOT contain review chaining', () => {
|
||
const content = readShipUnion();
|
||
expect(content).not.toContain('Review Chaining');
|
||
});
|
||
});
|
||
|
||
// ─── Test Coverage Audit Resolver Tests ─────────────────────
|
||
|
||
describe('TEST_COVERAGE_AUDIT placeholders', () => {
|
||
const planSkill = readSkillUnion('plan-eng-review'); // carved
|
||
const shipSkill = readShipUnion();
|
||
const reviewSkill = readSkillUnion('review'); // carved: Review Army moved to sections/review-army.md
|
||
|
||
test('plan and ship modes share codepath tracing methodology', () => {
|
||
// Review mode delegates test coverage to the Testing specialist subagent (Review Army)
|
||
const normalizedPlanSkill = planSkill.replace(/\s+/g, ' ');
|
||
const normalizedShipSkill = shipSkill.replace(/\s+/g, ' ');
|
||
for (const skill of [normalizedPlanSkill, normalizedShipSkill]) {
|
||
ordered(skill, ['concrete source and test files', '**Trace data flow.**', '**Diagram the execution.**', 'Quality scoring rubric']);
|
||
for (const marker of ['★★★', '★★', 'GAP']) expect(skill).toContain(marker);
|
||
}
|
||
// Plan mode traces the plan, not a git diff
|
||
expect(planSkill).toContain('Trace every codepath in the plan');
|
||
expect(planSkill).not.toContain('git diff origin');
|
||
// Ship mode traces the diff
|
||
expect(shipSkill).toContain('Trace every codepath changed');
|
||
});
|
||
|
||
test('review mode uses Review Army for specialist dispatch', () => {
|
||
expect(reviewSkill).toContain('Review Army');
|
||
expect(reviewSkill).toContain('Specialist Dispatch');
|
||
expect(reviewSkill).toContain('testing.md');
|
||
});
|
||
|
||
test('plan and ship modes include E2E decision matrix', () => {
|
||
// Review mode delegates to Testing specialist
|
||
for (const skill of [planSkill, shipSkill]) {
|
||
expect(skill).toContain('E2E Test Decision Matrix');
|
||
expect(skill).toContain('→E2E');
|
||
expect(skill).toContain('→EVAL');
|
||
}
|
||
});
|
||
|
||
test('plan and ship modes include regression rule', () => {
|
||
// Review mode delegates to Testing specialist
|
||
for (const skill of [planSkill, shipSkill]) {
|
||
expect(skill).toMatch(/^#{3,4} REGRESSION RULE/m);
|
||
}
|
||
expectMentions(planSkill, [['without','requirement','regression']], 'planSkill');
|
||
});
|
||
|
||
test('all five plan audit subheadings stay inside Test review while ship keeps its existing depth', () => {
|
||
const tests = planSkill.split('\n### 3. Test review\n')[1]!.split('\n### 4. Performance review\n')[0]!;
|
||
const instructions = tests.replace(/```[\s\S]*?```/g, '');
|
||
const headings = ['Test Framework Detection', 'E2E Test Decision Matrix', 'REGRESSION RULE (mandatory)',
|
||
'LLM/eval scope', 'Test Plan Artifact'];
|
||
expect([...instructions.matchAll(/^#### (.+)$/gm)].map(match => match[1])).toEqual(headings);
|
||
expect(instructions).not.toMatch(/^#{1,3} /m);
|
||
for (const heading of headings.filter(heading => heading !== 'LLM/eval scope')) {
|
||
expect(shipSkill).toContain(`\n### ${heading}\n`);
|
||
expect(shipSkill).not.toContain(`\n#### ${heading}\n`);
|
||
}
|
||
});
|
||
|
||
test('planned regression coverage gets its own approved contract without making coverage optional', () => {
|
||
const regression = planSkill.split('\n#### REGRESSION RULE (mandatory)\n')[1]!.split('**Step 4.')[0]!;
|
||
expect(regression).toContain('critical requirement');
|
||
expectMentions(regression, [['not','whether','cover']], 'regression');
|
||
expectMentions(regression, [['before','approved','contract']], 'regression');
|
||
expect(regression).not.toContain('No AskUserQuestion');
|
||
expect(regression).not.toContain('the diff broke');
|
||
expectMentions(regression, [['not','running','already']], 'regression');
|
||
});
|
||
|
||
test('plan gap additions wait for their test-contract decision', () => {
|
||
const action = planSkill.split('**Step 5. Add missing tests to the plan:**')[1]!.split('\n#### Test Plan Artifact\n')[0]!;
|
||
expectMentions(action, [['until','contracts','optional']], 'action');
|
||
expectMentions(action, [['wait','applying','answer']], 'action');
|
||
expect(action).not.toContain('explain what broke');
|
||
});
|
||
|
||
test('implemented regressions still require an immediate test without a question', () => {
|
||
const regression = shipSkill.split('### REGRESSION RULE (mandatory)')[1]!.split('**4.')[0]!;
|
||
expectMentions(regression, [['no','askuserquestion']], 'regression');
|
||
});
|
||
|
||
test('plan and ship modes include test framework detection', () => {
|
||
// Review mode delegates to Testing specialist
|
||
for (const skill of [planSkill, shipSkill]) {
|
||
expect(skill).toContain('Test Framework Detection');
|
||
expect(skill).toContain('CLAUDE.md');
|
||
}
|
||
});
|
||
|
||
test('plan framework absence keeps requirements without offering test generation', () => {
|
||
const detection = planSkill.split('\n#### Test Framework Detection\n')[1]!.split('**Step 1.')[0]!;
|
||
expect(detection).toContain('State that the framework is unknown');
|
||
expectMentions(detection, [['no','selection','framework']], 'detection');
|
||
expectMentions(detection, [['do not','framework','proposed']], 'detection');
|
||
expect(detection).not.toContain('skip test generation');
|
||
});
|
||
|
||
test('plan mode adds tests to plan + includes test plan artifact', () => {
|
||
expect(planSkill).toContain('Add missing tests to the plan');
|
||
expect(planSkill).toContain('eng-review-test-plan');
|
||
expect(planSkill).toContain('Test Plan Artifact');
|
||
const artifact = planSkill.split('\n#### Test Plan Artifact\n')[1]!;
|
||
// The artifact is written even when Test review choices are still unanswered (periodic run 37151477069).
|
||
const step5 = planSkill.split('**Step 5. Add missing tests to the plan:**')[1]!.split('\n#### Test Plan Artifact\n')[0]!;
|
||
expectMentions(artifact, [['not','implementation','unresolved']], 'artifact');
|
||
expect(artifact).toContain('TEST_PLAN_USER=$(whoami)');
|
||
expect(artifact).toContain('sanitized `BRANCH` from gstack-slug');
|
||
expect(artifact).toContain('without an origin, write `local-only`');
|
||
});
|
||
|
||
test('ship mode auto-generates tests + includes before/after count', () => {
|
||
expect(shipSkill).toContain('Generate tests for uncovered paths');
|
||
expect(shipSkill).toContain('Before/after test count');
|
||
expect(shipSkill).toContain('30 code paths max');
|
||
expect(shipSkill).toContain('ship-test-plan');
|
||
});
|
||
|
||
test('review mode uses Fix-First + Review Army for specialist coverage', () => {
|
||
expect(reviewSkill).toContain('Fix-First');
|
||
expect(reviewSkill).toContain('INFORMATIONAL');
|
||
// Review Army handles test coverage via Testing specialist subagent
|
||
expect(reviewSkill).toContain('Review Army');
|
||
expect(reviewSkill).toContain('Testing');
|
||
});
|
||
|
||
test('plan mode does NOT include ship-specific content', () => {
|
||
expect(planSkill).not.toContain('Before/after test count');
|
||
expect(planSkill).not.toContain('30 code paths max');
|
||
expect(planSkill).not.toContain('ship-test-plan');
|
||
});
|
||
|
||
test('review mode does NOT include test plan artifact', () => {
|
||
expect(reviewSkill).not.toContain('Test Plan Artifact');
|
||
expect(reviewSkill).not.toContain('eng-review-test-plan');
|
||
expect(reviewSkill).not.toContain('ship-test-plan');
|
||
});
|
||
|
||
test('review/specialists/ directory has all expected checklist files', () => {
|
||
const specDir = path.join(ROOT, 'review', 'specialists');
|
||
const expected = [
|
||
'testing.md',
|
||
'maintainability.md',
|
||
'security.md',
|
||
'performance.md',
|
||
'data-migration.md',
|
||
'api-contract.md',
|
||
'simplification.md',
|
||
'red-team.md',
|
||
];
|
||
for (const f of expected) {
|
||
expect(fs.existsSync(path.join(specDir, f))).toBe(true);
|
||
}
|
||
});
|
||
|
||
// Regression pins for the simplification specialist (advisory carve-out edits
|
||
// the pre-existing quality_score instruction, so the rendered contract is
|
||
// pinned statically — the carve-out and the early-out line must both survive
|
||
// regeneration verbatim).
|
||
test('simplification advisory carve-out and early-out render into review docs', () => {
|
||
const reviewArmySection = fs.readFileSync(
|
||
path.join(ROOT, 'review', 'sections', 'review-army.md'),
|
||
'utf-8',
|
||
);
|
||
expect(reviewArmySection).toContain('"advisory": true');
|
||
expectTokens(reviewArmySection, ['`quality_score`'], 'reviewArmySection');
|
||
expectMentions(reviewArmySection, [['do not', 'specialist', 'findings']], 'reviewArmySection');
|
||
expect(reviewArmySection).toContain('Use the merged NON-advisory specialist findings for both counts and score');
|
||
expect(reviewArmySection).toContain('net: -N lines possible');
|
||
expect(reviewArmySection).toContain('--simplification');
|
||
// The specialist itself must never carry a verdict-shaped zero-findings line.
|
||
const spec = fs.readFileSync(
|
||
path.join(ROOT, 'review', 'specialists', 'simplification.md'),
|
||
'utf-8',
|
||
);
|
||
expect(spec).toContain('NO FINDINGS');
|
||
expect(spec).not.toContain('Lean already. Ship.');
|
||
// Closed tag vocabulary: the disavowed yagni: frame must not appear.
|
||
expect(spec).toContain('speculative');
|
||
expect(spec.toLowerCase()).not.toContain('"yagni"');
|
||
});
|
||
|
||
test('each specialist file has standard header with scope and output format', () => {
|
||
const specDir = path.join(ROOT, 'review', 'specialists');
|
||
const files = fs.readdirSync(specDir).filter(f => f.endsWith('.md'));
|
||
for (const f of files) {
|
||
const content = fs.readFileSync(path.join(specDir, f), 'utf-8');
|
||
// All specialist files must have Scope and Output/JSON in header
|
||
expect(content).toContain('Scope:');
|
||
expect(content.toLowerCase()).toMatch(/output|json/);
|
||
// Must define NO FINDINGS behavior
|
||
expect(content).toContain('NO FINDINGS');
|
||
}
|
||
});
|
||
|
||
// Regression guard: ship output contains key phrases from before the refactor
|
||
test('ship SKILL.md regression guard — key phrases preserved', () => {
|
||
const regressionPhrases = [
|
||
'Coverage goal: every changed behavior is protected by a test that would catch a real regression.',
|
||
'ASCII coverage diagram',
|
||
'processPayment',
|
||
'refundPayment',
|
||
'billing.test.ts',
|
||
'checkout.e2e.ts',
|
||
'COVERAGE:',
|
||
'QUALITY:',
|
||
'GAPS:',
|
||
'Code paths:',
|
||
'User flows:',
|
||
];
|
||
for (const phrase of regressionPhrases) {
|
||
expect(shipSkill).toContain(phrase);
|
||
}
|
||
});
|
||
|
||
test('ship SKILL.md contains review army specialist dispatch', () => {
|
||
expect(shipSkill).toContain('Specialist Dispatch');
|
||
expect(shipSkill).toContain('Step 9.1');
|
||
expect(shipSkill).toContain('Step 9.2');
|
||
});
|
||
|
||
test('ship SKILL.md contains cross-review finding dedup', () => {
|
||
expect(shipSkill).toContain('Cross-review finding dedup');
|
||
expect(shipSkill).toContain('Step 9.3');
|
||
});
|
||
|
||
test('ship SKILL.md contains re-run idempotency behavior', () => {
|
||
expect(shipSkill).toContain('**Route:**');
|
||
expect(shipSkill.replace(/\s+/g, ' ')).toContain('integrate (1–3) → test and review (4–11.5) → prepare the release (12–15) → verify frozen content (16) → push and publish (17–21)');
|
||
expectMentions(shipSkill.replace(/\s+/g, ' '), [['never','verification','duplicate']], 'shipSkill.replace(/\s+/g, )');
|
||
});
|
||
});
|
||
|
||
// --- {{TEST_FAILURE_TRIAGE}} resolver tests ---
|
||
|
||
describe('TEST_FAILURE_TRIAGE resolver', () => {
|
||
const shipSkill = readShipUnion();
|
||
|
||
test('contains all 4 triage steps', () => {
|
||
expect(shipSkill).toContain('Step T1: Classify each failure');
|
||
expect(shipSkill).toContain('Step T2: Handle in-branch failures');
|
||
expect(shipSkill).toContain('Step T3: Handle pre-existing failures');
|
||
expect(shipSkill).toContain('Step T4: Execute the chosen action');
|
||
});
|
||
|
||
test('T1 includes classification criteria (in-branch vs pre-existing)', () => {
|
||
expect(shipSkill).toContain('In-branch');
|
||
expect(shipSkill).toContain('Likely pre-existing');
|
||
expect(shipSkill).toContain('git diff origin/');
|
||
});
|
||
|
||
test('T3 branches on REPO_MODE (solo vs collaborative)', () => {
|
||
expect(shipSkill).toContain('REPO_MODE');
|
||
expect(shipSkill).toContain('solo');
|
||
expect(shipSkill).toContain('collaborative');
|
||
});
|
||
|
||
test('solo mode offers fix-now, TODO, and skip options', () => {
|
||
expect(shipSkill).toContain('Investigate and fix now');
|
||
expect(shipSkill).toContain('Add as P0 TODO');
|
||
expect(shipSkill).toContain('Skip');
|
||
});
|
||
|
||
test('collaborative mode offers blame + assign option', () => {
|
||
expect(shipSkill).toContain('Blame + assign GitHub issue');
|
||
expect(shipSkill).toContain('gh issue create');
|
||
});
|
||
|
||
test('defaults ambiguous failures to in-branch (safety)', () => {
|
||
expect(shipSkill).toContain('When ambiguous, default to in-branch');
|
||
});
|
||
});
|
||
|
||
// --- {{PLAN_FILE_REVIEW_REPORT}} resolver tests ---
|
||
|
||
describe('PLAN_FILE_REVIEW_REPORT resolver', () => {
|
||
const REVIEW_SKILLS = ['plan-ceo-review', 'plan-eng-review', 'plan-design-review', 'codex'];
|
||
|
||
for (const [label, skill] of [['Eng', 'plan-eng-review'], ['CEO', 'plan-ceo-review']]) {
|
||
test(`${label} report example is fenced Markdown, and the original escaped fence is rejected`, async () => {
|
||
const { marked } = await import('marked');
|
||
const content = readSkillUnion(skill!);
|
||
const tokens = marked.lexer(content);
|
||
const headings = tokens.filter(token => token.type === 'heading' && token.depth === 2);
|
||
expect(headings.map(token => token.text)).not.toContain('GSTACK REVIEW REPORT');
|
||
const example = tokens.find(token => token.type === 'code'
|
||
&& token.lang === 'markdown' && token.text.startsWith('## GSTACK REVIEW REPORT\n'));
|
||
expect(example).toBeDefined();
|
||
if (example?.type !== 'code') throw new Error('Missing fenced report example');
|
||
expect(example.text).toContain('| Review | Trigger | Why | Runs | Status | Findings |');
|
||
expect(example.text).toContain('`/' + skill + '`');
|
||
// Reproduce the observed literal-backslash delimiters without changing
|
||
// the parser or preprocessing malformed Markdown into valid fences.
|
||
const broken = content.replace(example.raw, example.raw.replaceAll('`', '\\`'));
|
||
expect(broken).not.toBe(content);
|
||
const brokenTokens = marked.lexer(broken);
|
||
expect(brokenTokens.some(token => token.type === 'code' && token.lang === 'markdown'
|
||
&& token.text.startsWith('## GSTACK REVIEW REPORT\n'))).toBe(false);
|
||
expect(brokenTokens.filter(token => token.type === 'heading' && token.depth === 2)
|
||
.map(token => token.text)).toContain('GSTACK REVIEW REPORT');
|
||
});
|
||
}
|
||
|
||
test('Eng renderers use real code delimiters without changing confidence rules or report fields', async () => {
|
||
const {generateConfidenceCalibration} = await import('../scripts/resolvers/confidence');
|
||
const {generateReviewDashboard, generatePlanFileReviewReport} = await import('../scripts/resolvers/review-dashboard');
|
||
const {generateCodexPlanReview} = await import('../scripts/resolvers/outside-voice-steps');
|
||
const {HOST_PATHS} = await import('../scripts/resolvers/types');
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const ctx = {skillName: 'plan-eng-review', tmplPath: 'plan-eng-review/SKILL.md.tmpl',
|
||
host: host.name, paths: HOST_PATHS[host.name]!};
|
||
const confidence = generateConfidenceCalibration(ctx);
|
||
const dashboard = generateReviewDashboard(ctx);
|
||
const report = generatePlanFileReviewReport(ctx);
|
||
const outside = generateCodexPlanReview(ctx);
|
||
// These four resolver fragments contain no intentional escaped-backtick
|
||
// example; the preamble does, and is deliberately outside this check.
|
||
for (const output of [confidence, dashboard, report, outside]) expect(output).not.toContain('\\`');
|
||
expect(confidence).toBe(generateConfidenceCalibration({...ctx, skillName: 'plan-ceo-review'}).replaceAll('\\`', '`'));
|
||
const ceoDashboard = generateReviewDashboard({...ctx, skillName: 'plan-ceo-review'}).replaceAll('\\`', '`');
|
||
const ceoVoiceSource = 'Below the dashboard, group `autoplan-voices` and `design-outside-voices` by workflow run and phase';
|
||
expect(dashboard.replace(/\s+/g, ' ')).toContain(ceoVoiceSource);
|
||
expect(ceoDashboard.replace(/\s+/g, ' ')).toContain(ceoVoiceSource);
|
||
expect(ceoDashboard).toBe(dashboard);
|
||
for (const field of ['status', 'unresolved', 'critical_gaps', 'issues_found', 'mode', 'commit']) {
|
||
expect(report).toContain('`' + field + '`');
|
||
}
|
||
expect(outside).toContain(host.name === 'codex' ? 'claude auth login' : 'codex login');
|
||
expectMentions(outside, [['never','supplies','coverage']], 'outside');
|
||
}
|
||
});
|
||
|
||
for (const skill of REVIEW_SKILLS) {
|
||
test(`plan file review report appears in ${skill} generated file`, () => {
|
||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('GSTACK REVIEW REPORT');
|
||
});
|
||
}
|
||
|
||
test('resolver output contains key report elements', () => {
|
||
const content = readSkillUnion('plan-ceo-review'); // carved: report writer moved to section
|
||
expect(content).toContain('Trigger');
|
||
expect(content).toContain('Findings');
|
||
expect(content).toContain('VERDICT');
|
||
expect(content).toContain('/plan-ceo-review');
|
||
expect(content).toContain('/plan-eng-review');
|
||
expect(content).toContain('/plan-design-review');
|
||
expect(content).toContain('/codex review');
|
||
});
|
||
|
||
test('Eng task output follows the same write policy as its report', () => {
|
||
const content = readSkillUnion('plan-eng-review');
|
||
// The task example itself has an H2; bound by the following real section.
|
||
const tasks = content.slice(content.indexOf('## Implementation Tasks'), content.indexOf('## Plan File Review Report'));
|
||
expectMentions(tasks, [['only','permits','review']], 'tasks');
|
||
expectMentions(tasks, [['not','persisted','complete']], 'tasks');
|
||
expect(tasks).not.toContain('always write');
|
||
});
|
||
});
|
||
|
||
// --- {{PLAN_COMPLETION_AUDIT}} resolver tests ---
|
||
|
||
describe('PLAN_COMPLETION_AUDIT placeholders', () => {
|
||
const shipSkill = readShipUnion();
|
||
const reviewSkill = readSkillUnion('review'); // carved: plan-completion audit moved to sections/plan-completion.md
|
||
|
||
test('ship SKILL.md contains plan completion audit step', () => {
|
||
expect(shipSkill).toContain('Plan Completion Audit');
|
||
expect(shipSkill).toContain('Step 8');
|
||
});
|
||
|
||
test('review SKILL.md contains plan completion in scope drift', () => {
|
||
expect(reviewSkill).toContain('Plan File Discovery');
|
||
expect(reviewSkill).toContain('Actionable Item Extraction');
|
||
expect(reviewSkill).toContain('Integration with Scope Drift Detection');
|
||
});
|
||
|
||
test('both modes share plan file discovery methodology', () => {
|
||
expect(shipSkill).toContain('Plan File Discovery');
|
||
expect(reviewSkill).toContain('Plan File Discovery');
|
||
// Both should have conversation context first
|
||
expect(shipSkill).toContain('Conversation context (primary)');
|
||
expect(reviewSkill).toContain('Conversation context (primary)');
|
||
// Both should have grep fallback
|
||
expect(shipSkill).toContain('Content-based search (fallback)');
|
||
expect(reviewSkill).toContain('Content-based search (fallback)');
|
||
});
|
||
|
||
test('ship mode has gate logic for NOT DONE items', () => {
|
||
expect(shipSkill).toContain('NOT DONE');
|
||
expect(shipSkill).toContain('Stop — implement the missing items');
|
||
expect(shipSkill).toContain('Ship anyway — defer');
|
||
expect(shipSkill).toContain('intentionally dropped');
|
||
});
|
||
|
||
test('review mode is INFORMATIONAL only', () => {
|
||
expect(reviewSkill).toContain('INFORMATIONAL');
|
||
expect(reviewSkill).toContain('MISSING REQUIREMENTS');
|
||
expect(reviewSkill).toContain('SCOPE CREEP');
|
||
});
|
||
|
||
test('item extraction has 50-item cap', () => {
|
||
expect(shipSkill).toContain('at most 50 items');
|
||
});
|
||
|
||
test('uses file-level traceability (not commit-level)', () => {
|
||
expect(shipSkill).toContain('Cite the specific file');
|
||
expect(shipSkill).not.toContain('commit-level traceability');
|
||
});
|
||
});
|
||
|
||
// --- {{PLAN_VERIFICATION_EXEC}} resolver tests ---
|
||
|
||
describe('PLAN_VERIFICATION_EXEC placeholder', () => {
|
||
const shipSkill = readShipUnion();
|
||
|
||
test('ship SKILL.md contains plan verification step', () => {
|
||
expect(shipSkill).toContain('Step 8.1');
|
||
expect(shipSkill).toContain('Plan Verification');
|
||
});
|
||
|
||
test('references the shared explorer without invoking an entire QA workflow', () => {
|
||
const resource = "From the installed /ship SKILL.md's directory, Read `../qa/sections/exploratory.md` in full";
|
||
expect(shipSkill).toContain(resource);
|
||
const load = shipSkill.indexOf(resource);
|
||
const preflight = shipSkill.indexOf('Run the shared preflight;');
|
||
const probes = shipSkill.indexOf('**3. Run smoke and plan checks.**');
|
||
expect(load).toBeGreaterThan(-1);
|
||
expect(preflight).toBeGreaterThan(load);
|
||
expect(probes).toBeGreaterThan(preflight);
|
||
const shared = fs.readFileSync(path.join(ROOT, 'qa/sections/exploratory.md'), 'utf8');
|
||
const selection = shared.indexOf('in full and select the surfaces');
|
||
const methods = shared.indexOf('Read `sections/system-functional.md`');
|
||
expect(selection).toBeGreaterThan(shared.indexOf('Read `sections/scope.md`'));
|
||
expect(methods).toBeGreaterThan(selection);
|
||
expect(shared.indexOf('Write a **charter**')).toBeGreaterThan(methods);
|
||
expect(shipSkill.slice(preflight, probes)).toContain("For browsers, Read QA's `sections/browser-setup.md`");
|
||
expectMentions(shipSkill, [['do not','invoke','entire']], 'shipSkill');
|
||
});
|
||
|
||
test('keeps declared browser URLs separate from native functional probes', () => {
|
||
expectMentions(shipSkill, [['without','discovering','functional']], 'shipSkill');
|
||
expectMentions(shipSkill.replace(/\s+/g, ' '), [['not', 'automatically', 'page']], 'shipSkill.replace(/\s+/g, )');
|
||
});
|
||
|
||
test('retains automatic exploration when there is no plan or verification section', () => {
|
||
expectMentions(shipSkill, [['no','verification','section']], 'shipSkill');
|
||
expect(shipSkill).toContain('Automatic diff-scoped QA still runs');
|
||
});
|
||
|
||
test('blocks unavailable required checks instead of silently skipping them', () => {
|
||
const flat = shipSkill.replace(/\s+/g, ' ');
|
||
expect(flat).toContain('Noninteractive runs return blocked');
|
||
expectMentions(flat, [['never','required-probe','silently']], 'flat');
|
||
expect(flat).toContain('Missing/unreadable assets block required QA');
|
||
expect(flat).toContain('Keep actual outcomes and incomplete flags; VERIFY_RESULT stays fail');
|
||
});
|
||
});
|
||
|
||
// --- Coverage gate tests ---
|
||
|
||
describe('Coverage gate in ship', () => {
|
||
const shipSkill = readShipUnion();
|
||
const reviewSkill = readSkillUnion('review'); // carved: testing.md specialist ref lives in sections/review-army.md
|
||
|
||
test('ship SKILL.md contains coverage gate with thresholds', () => {
|
||
expect(shipSkill).toContain('Coverage gate');
|
||
expect(shipSkill).toContain('>= target');
|
||
expect(shipSkill).toContain('< minimum');
|
||
});
|
||
|
||
test('ship SKILL.md supports configurable thresholds via CLAUDE.md', () => {
|
||
expect(shipSkill).toContain('## Test Coverage');
|
||
expect(shipSkill).toContain('Minimum:');
|
||
expect(shipSkill).toContain('Target:');
|
||
});
|
||
|
||
test('coverage gate skips on parse failure (not block)', () => {
|
||
expect(shipSkill).toContain('could not determine percentage — skipping');
|
||
});
|
||
|
||
test('review SKILL.md delegates coverage to Testing specialist', () => {
|
||
// Coverage audit moved to Testing specialist subagent in Review Army
|
||
expect(reviewSkill).toContain('testing.md');
|
||
expect(reviewSkill).toContain('INFORMATIONAL');
|
||
});
|
||
});
|
||
|
||
// --- Ship metrics logging ---
|
||
|
||
describe('Ship metrics logging', () => {
|
||
const shipSkill = readShipUnion();
|
||
|
||
test('ship SKILL.md contains metrics persistence step', () => {
|
||
expect(shipSkill).toContain('Step 20');
|
||
expect(shipSkill).toContain('coverage_pct');
|
||
expect(shipSkill).toContain('plan_items_total');
|
||
expect(shipSkill).toContain('plan_items_done');
|
||
expect(shipSkill).toContain('verification_result');
|
||
});
|
||
});
|
||
|
||
// --- Plan file discovery shared helper ---
|
||
|
||
describe('Plan file discovery shared helper', () => {
|
||
// The shared helper should appear in ship (via PLAN_COMPLETION_AUDIT_SHIP)
|
||
// and in review (via PLAN_COMPLETION_AUDIT_REVIEW)
|
||
const shipSkill = readShipUnion();
|
||
const reviewSkill = readSkillUnion('review'); // carved: plan-completion audit moved to sections/plan-completion.md
|
||
|
||
test('plan file discovery appears in both ship and review', () => {
|
||
expect(shipSkill).toContain('Plan File Discovery');
|
||
expect(reviewSkill).toContain('Plan File Discovery');
|
||
});
|
||
|
||
test('both include conversation context first', () => {
|
||
expect(shipSkill).toContain('Conversation context (primary)');
|
||
expect(reviewSkill).toContain('Conversation context (primary)');
|
||
});
|
||
|
||
test('both include content-based fallback', () => {
|
||
expect(shipSkill).toContain('Content-based search (fallback)');
|
||
expect(reviewSkill).toContain('Content-based search (fallback)');
|
||
});
|
||
});
|
||
|
||
// --- Retro plan completion ---
|
||
|
||
describe('Retro plan completion section', () => {
|
||
// Carved: the narrative report format (incl. Plan Completion) lives in
|
||
// retro/sections/report-format.md — read the skeleton+sections union.
|
||
const retroSkill = readSkillUnion('retro');
|
||
|
||
test('retro SKILL.md contains plan completion section', () => {
|
||
expect(retroSkill).toContain('### Plan Completion');
|
||
expect(retroSkill).toContain('plan_items_total');
|
||
expect(retroSkill).toContain('Plan Completion This Period');
|
||
});
|
||
});
|
||
|
||
// --- Plan status footer in preamble ---
|
||
|
||
describe('Plan status footer in preamble', () => {
|
||
test('preamble contains plan status footer as neutral forward reference to EXIT PLAN MODE GATE', () => {
|
||
// Read any skill that uses PREAMBLE
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
expect(content).toContain('Plan Status Footer');
|
||
expect(content).toContain('GSTACK REVIEW REPORT');
|
||
expect(content).toContain('ExitPlanMode');
|
||
expect(content).toContain('EXIT PLAN MODE GATE');
|
||
// The preamble must NOT impose review-report rules on operational skills
|
||
// that have no review report. It's a forward reference, not enforcement.
|
||
expect(content).not.toContain('NO REVIEWS YET');
|
||
});
|
||
});
|
||
|
||
// --- make-pdf setup ordering ---
|
||
|
||
describe('make-pdf setup ordering', () => {
|
||
test('MAKE-PDF SETUP appears before generic preamble footer sections', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'make-pdf', 'SKILL.md'), 'utf-8');
|
||
const preambleIdx = content.indexOf('## Preamble (run first)');
|
||
const setupIdx = content.indexOf('## MAKE-PDF SETUP');
|
||
const planModeIdx = content.indexOf('## Plan Mode Safe Operations');
|
||
const telemetryIdx = content.indexOf('## Telemetry (run last)');
|
||
const workflowIdx = content.indexOf('# make-pdf: publication-quality PDFs from markdown');
|
||
|
||
expect(preambleIdx).toBeGreaterThanOrEqual(0);
|
||
expect(setupIdx).toBeGreaterThan(preambleIdx);
|
||
expect(setupIdx).toBeLessThan(planModeIdx);
|
||
expect(setupIdx).toBeLessThan(telemetryIdx);
|
||
expect(setupIdx).toBeLessThan(workflowIdx);
|
||
expect(content.match(/^## MAKE-PDF SETUP/gm)?.length ?? 0).toBe(1);
|
||
});
|
||
});
|
||
|
||
// --- Skill invocation during plan mode in preamble ---
|
||
|
||
describe('Skill invocation during plan mode in preamble', () => {
|
||
test('preamble contains skill invocation plan mode section', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
expect(content).toContain('Skill Invocation During Plan Mode');
|
||
expect(content).toContain('Do not continue the workflow');
|
||
expect(content).toContain('cancel the skill or leave plan mode');
|
||
});
|
||
|
||
// #2851: a skill cannot grant permissions or redefine the host's mode; the
|
||
// host's restrictions and the user's scope win, in every generated skill.
|
||
test('no generated skill claims precedence over host plan-mode restrictions (#2851)', () => {
|
||
let checked = 0;
|
||
for (const file of ['SKILL.md', ...fs.readdirSync(ROOT).map(d => path.join(d, 'SKILL.md'))]) {
|
||
if (!fs.existsSync(path.join(ROOT, file))) continue;
|
||
const content = fs.readFileSync(path.join(ROOT, file), 'utf-8');
|
||
if (!content.includes('## Skill Invocation During Plan Mode')) continue;
|
||
checked++;
|
||
expect(content, file).not.toContain('takes precedence over generic plan mode');
|
||
expect(content, file).not.toContain('overrides generic plan mode');
|
||
expect(content, file).not.toContain('allowed because they inform the plan');
|
||
expect(content, file).not.toContain('"PLAN MODE EXCEPTION — ALWAYS RUN" execute.');
|
||
expect(content, file).not.toContain('Execute "PLAN MODE EXCEPTION — ALWAYS RUN" commands');
|
||
expectMentions(content, [['only','restrictions','precedence']], 'content');
|
||
expect(content, file).toMatch(/"PLAN MODE EXCEPTION — ALWAYS RUN" (?:commands )?(?:run )?only where the host permits/);
|
||
}
|
||
expect(checked).toBeGreaterThan(40);
|
||
});
|
||
});
|
||
|
||
// --- {{SPEC_REVIEW_LOOP}} resolver tests ---
|
||
|
||
describe('SPEC_REVIEW_LOOP resolver', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
const { generateSpecReviewLoop } = require('../scripts/resolvers/spec-review');
|
||
const { HOST_PATHS } = require('../scripts/resolvers/types');
|
||
const render = (skillName: string, host = 'claude') => generateSpecReviewLoop({
|
||
skillName,
|
||
tmplPath: `${skillName}/SKILL.md.tmpl`,
|
||
host,
|
||
paths: HOST_PATHS[host],
|
||
});
|
||
|
||
test('office-hours prepares a complete reviewer prompt on every host', () => {
|
||
const { renderOfficeHoursReviewerPrompt } = require('../lib/office-hours-review');
|
||
for (const host of Object.keys(HOST_PATHS)) {
|
||
const output = render('office-hours', host);
|
||
expect(output).toContain('gstack-office-hours-review prepare --design');
|
||
expect(output).toContain('--out-dir "<review-directory>"');
|
||
ordered(output, ['mktemp -d "<design-path>.review.XXXXXX"', 'prepare --design', '`round-N.prompt.md`', '`dispatch`']);
|
||
}
|
||
const prompt = renderOfficeHoursReviewerPrompt({ document: '/tmp/design.md', verdictPath: '/tmp/round-1.json' });
|
||
expect(prompt).toContain('Document: /tmp/design.md');
|
||
expect(prompt).toContain('Verdict: /tmp/round-1.json');
|
||
expect(prompt).toContain('"document": "/tmp/design.md"');
|
||
expect(prompt).toMatch(/design and coaching document/i);
|
||
for (const section of ["'The Assignment'", "'What I noticed about how you think'"]) expect(prompt).toContain(section);
|
||
expect(prompt).toMatch(/never invent customer answers/i);
|
||
expect(prompt).toMatch(/including minor findings/i);
|
||
expect(prompt).toContain('`OFFICE_HOURS_VERDICT round=1 sha256=<hash> path=<verdict path>`');
|
||
expect(prompt).toMatch(/run the `Seal:` command/);
|
||
expect(prompt).not.toMatch(/identical JSON/i);
|
||
expect(prompt).toMatch(/classify every preceding finding/i);
|
||
ordered(prompt, ['1. **Completeness**', '2. **Consistency**', '3. **Clarity**', '4. **Scope**', '5. **Feasibility**']);
|
||
expect(prompt).toMatch(/distinguished from committed behavior/i);
|
||
expect(prompt).toContain('"version": 2');
|
||
expect(prompt).toContain('"prior": []');
|
||
});
|
||
|
||
test('office-hours checks stop conditions before edits and preserves concerns mechanically', () => {
|
||
const output = render('office-hours');
|
||
const step2 = output.slice(output.indexOf('**Step 2:'), output.indexOf('**Step 3:'));
|
||
expectMentions(step2, [['before','dispatching','findings']], 'step2');
|
||
ordered(step2, ['gstack-office-hours-review check --receipt "<reviewer response>"', 'mismatched receipt fails the check', '- PASS:', '- CONVERGENCE:', '- MAX_ITERATIONS: round 3', '- CONTINUE:', 'gstack-office-hours-review finalize --design']);
|
||
expectMentions(step2, [['fix only the blocking findings'], ['do not edit for minor findings']], 'step2');
|
||
expectMentions(step2, [['do not','re-dispatch','again']], 'step2');
|
||
expectMentions(step2, [['do not','generated','section']], 'step2');
|
||
expect(step2.slice(step2.indexOf('finalize --design'))).toMatch(/existing user approval/i);
|
||
});
|
||
|
||
test('office-hours finalizes the report after writing it and labels derivable metrics', () => {
|
||
const output = render('office-hours');
|
||
expect(output).toContain('--report "<report-path>"');
|
||
expect(output).toMatch(/after the report exists/i);
|
||
expectMentions(output, [['do not','summarize','replace']], 'output');
|
||
expectMentions(output, [['approval','assignment','preserve']], 'output');
|
||
expect(output).toMatch(/explicit later reviewer evidence/i);
|
||
});
|
||
|
||
test('office-hours errors stay explicit and preserve preceding valid evidence', () => {
|
||
const output = render('office-hours');
|
||
expect(output).toMatch(/--unreviewed "<actual failure cause>"`\s+and only the\s+preceding valid round files/);
|
||
expect(output).toMatch(/do not fabricate JSON/i);
|
||
expect(output).toMatch(/not an approval gate/i);
|
||
});
|
||
|
||
test('CEO keeps its loop limits, scoring, failure handling, and reporting', () => {
|
||
const ceo = render('plan-ceo-review');
|
||
const report = ceo.split('**Step 3:')[1]!.replace(/\s+/g, ' ');
|
||
expect(report).toMatch(/use JSON `null`/i);
|
||
expect(ceo).toMatch(/implementable without follow-up questions/i);
|
||
expect(ceo).not.toContain('design and coaching document');
|
||
expect(ceo).not.toContain('gstack-office-hours-review');
|
||
const dispatch = ceo.slice(ceo.indexOf('**Step 1:'), ceo.indexOf('**Step 2:')).replace(/\s+/g, ' ');
|
||
expect(dispatch).toContain('`run_in_background: false`');
|
||
expectMentions(dispatch, [['wait','tool']], 'dispatch');
|
||
expectMentions(dispatch, [['do not','waiting','advance']], 'dispatch');
|
||
const fixes = ceo.slice(ceo.indexOf('**Step 2:'), ceo.indexOf('**Step 3:')).replace(/\s+/g, ' ');
|
||
ordered(fixes, ['use 0D', 'amend the working plan', 're-dispatch']);
|
||
expectMentions(fixes, [['stop','review','after']], 'fixes');
|
||
expect(fixes).toMatch(/stop the loop/i);
|
||
expectMentions(fixes, [['not','successful','reviewer']], 'fixes');
|
||
expectMentions(fixes, [['does not','missing','require']], 'fixes');
|
||
expect(report).toContain('"## Reviewer Concerns"');
|
||
expect(report).toMatch(/SCORE is the latest attempt's reported 1–10 grade/i);
|
||
expect(report).toContain('"prior review score"');
|
||
expect(report).toMatch(/as not persisted/i);
|
||
expect(report).toContain('**0H spec-review metrics**');
|
||
expectMentions(report, [['stop','completion','required']], 'report');
|
||
expect(report).toContain('GSTACK_STATE_ROOT=$(~/.claude/skills/gstack/bin/gstack-paths --get GSTACK_STATE_ROOT); : "${GSTACK_STATE_ROOT:?gstack-paths failed; reinstall with ./setup or /gstack-upgrade}"');
|
||
expect(report).toContain('mkdir -p "$GSTACK_STATE_ROOT/analytics" || exit 1');
|
||
expect(report).toContain('>> "$GSTACK_STATE_ROOT/analytics/spec-review.jsonl" || exit 1');
|
||
expect(report).not.toContain('Your doc survived');
|
||
});
|
||
|
||
test('CEO spec report separates findings, confirmed fixes, and unresolved concerns', () => {
|
||
const report = render('plan-ceo-review').split('**Step 3: Report and persist metrics**')[1]!.replace(/\s+/g, ' ');
|
||
expect(report).toMatch(/ITERATIONS counts actual reviewer launches/i);
|
||
ordered(report, ['FOUND, FIXED and REMAINING', 'reviewer-confirmed fixes']);
|
||
expect(report).toMatch(/actual counts, never estimates/i);
|
||
expect(report).not.toContain('M issues caught and fixed');
|
||
});
|
||
|
||
test('CEO spec review receives the full source plan as well as the scope artifact on every host', () => {
|
||
for (const host of Object.keys(HOST_PATHS)) {
|
||
const output = render('plan-ceo-review', host).replace(/\s+/g, ' ');
|
||
expect(output).toMatch(/all five dimensions/i);
|
||
expect(output).toMatch(/unsupported accepted expansions/i);
|
||
}
|
||
const source = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf8');
|
||
expect(source).toContain('## Plan under review\n{working plan path, or');
|
||
const template = source.replace(/\s+/g, ' ');
|
||
expectMentions(template, [['cannot','summary','serve']], 'template');
|
||
});
|
||
|
||
test('CEO shares both inputs after spec review and owns unresolved concerns in its scope document', () => {
|
||
const template = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf8');
|
||
const persist = (template.split('### 0H.')[1]?.split('### 0I.')[0] ?? '').replace(/\s+/g, ' ');
|
||
const review = persist.indexOf('{{SPEC_REVIEW_LOOP}}');
|
||
const sharing = persist.search(/present both inputs for final scope-document approval/i);
|
||
expect(review).toBeGreaterThanOrEqual(0);
|
||
expect(sharing).toBeGreaterThan(review);
|
||
const output = render('plan-ceo-review');
|
||
const processing = output.split('**Step 2:')[1]!.split('**Step 3:')[0]!.replace(/\s+/g, ' ');
|
||
const report = output.split('**Step 3:')[1] ?? '';
|
||
expect(report).toContain('CEO summary');
|
||
expect(report).toContain('Reviewer Concerns');
|
||
expect(report).toMatch(/owning\s+input/);
|
||
});
|
||
|
||
test('CEO reviewer receives one complete prompt without duplicate scoring instructions', () => {
|
||
const output = render('plan-ceo-review');
|
||
const dispatch = (output.split('**Step 1: Dispatch reviewer subagent**')[1]?.split('**Step 2:')[0] ?? '').replace(/\s+/g, ' ');
|
||
expect(dispatch.match(/Read both inputs in full/gi)).toHaveLength(1);
|
||
expect(dispatch.match(/quality score/gi)).toHaveLength(1);
|
||
expect(dispatch).toMatch(/quality score \(1-10\)/i);
|
||
expect(dispatch).toMatch(/overall PASS only if all dimensions pass/i);
|
||
expect(dispatch).toMatch(/PASS or numbered issues/i);
|
||
expect(dispatch).toMatch(/cite input and requirement/i);
|
||
});
|
||
|
||
test('contains all 5 review dimensions', () => {
|
||
for (const dim of ['Completeness', 'Consistency', 'Clarity', 'Scope', 'Feasibility']) {
|
||
expect(content).toContain(dim);
|
||
}
|
||
});
|
||
|
||
test('references Agent tool for subagent dispatch', () => {
|
||
expect(content).toMatch(/Agent.*tool/i);
|
||
});
|
||
|
||
test('specifies max 3 iterations', () => {
|
||
expect(content).toMatch(/3.*iteration|maximum.*3/i);
|
||
});
|
||
|
||
test('includes quality score', () => {
|
||
expect(content).toContain('quality score');
|
||
});
|
||
|
||
test('includes metrics path', () => {
|
||
expect(content).toContain('spec-review.jsonl');
|
||
});
|
||
|
||
test('includes convergence guard', () => {
|
||
expect(content).toMatch(/[Cc]onvergence/);
|
||
});
|
||
|
||
test('includes graceful failure handling', () => {
|
||
expect(content).toMatch(/skip.*review|unavailable/i);
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_SKETCH}} resolver tests ---
|
||
|
||
describe('DESIGN_SKETCH resolver', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
|
||
test('references DESIGN.md for design system constraints', () => {
|
||
expect(content).toContain('DESIGN.md');
|
||
});
|
||
|
||
test('contains wireframe or sketch terminology', () => {
|
||
expect(content).toMatch(/wireframe|sketch/i);
|
||
});
|
||
|
||
test('wireframes render through gstack-render (Aside first)', () => {
|
||
expect(content).toContain('gstack-render.ts');
|
||
});
|
||
|
||
test('references screenshot capture', () => {
|
||
expect(content).toContain('--screenshot');
|
||
});
|
||
|
||
test('specifies rough aesthetic', () => {
|
||
expect(content).toMatch(/[Rr]ough|hand-drawn/);
|
||
});
|
||
|
||
test('includes skip conditions', () => {
|
||
expect(content).toMatch(/no UI component|skip/i);
|
||
});
|
||
});
|
||
|
||
// --- {{CODEX_SECOND_OPINION}} resolver tests ---
|
||
|
||
describe('CODEX_SECOND_OPINION resolver', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
const codexContent = readExternalSkillUnion(EXTERNAL_OUT, '.agents', 'office-hours');
|
||
|
||
test('Phase 3.5 section appears in office-hours SKILL.md', () => {
|
||
expect(content).toContain('Phase 3.5: Cross-Model Second Opinion');
|
||
});
|
||
|
||
test('contains codex exec invocation', () => {
|
||
expect(content).toContain('codex exec');
|
||
});
|
||
|
||
test('contains cross-model synthesis instructions', () => {
|
||
expect(content).toMatch(/[Ss]ynthesis/);
|
||
});
|
||
|
||
test('contains Claude subagent fallback', () => {
|
||
expect(content).toContain('CODEX_MODE: not_installed');
|
||
expect(content).toContain('If preflight is not ready');
|
||
expect(content).toContain('Agent tool');
|
||
expect(content).toContain('SECOND OPINION (Claude subagent)');
|
||
});
|
||
|
||
test('contains premise revision check', () => {
|
||
expect(content).toContain('Codex challenged premise');
|
||
});
|
||
|
||
test('contains error handling for auth, timeout, and empty', () => {
|
||
expect(content).toMatch(/[Aa]uth.*fail/);
|
||
expect(content).toMatch(/[Tt]imeout/);
|
||
expect(content).toMatch(/[Ee]mpty response/);
|
||
});
|
||
|
||
test('Codex host runs Phase 3.5 through Claude Code', () => {
|
||
expect(codexContent).toContain('Phase 3.5: Cross-Model Second Opinion');
|
||
expect(codexContent).toContain('gstack-claude-code');
|
||
expect(codexContent).toContain('Claude Code');
|
||
expect(codexContent).not.toMatch(/codex\s+(?:exec|review)\s/);
|
||
});
|
||
});
|
||
|
||
// --- Codex filesystem boundary tests ---
|
||
|
||
describe('Codex filesystem boundary', () => {
|
||
// Skills that call codex exec/review and should contain boundary text
|
||
const CODEX_CALLING_SKILLS = [
|
||
'codex', // /codex skill — 3 modes
|
||
'autoplan', // /autoplan — CEO/design/eng voices
|
||
'review', // /review — adversarial step resolver
|
||
'ship', // /ship — adversarial step resolver
|
||
'plan-eng-review', // outside voice resolver
|
||
'plan-ceo-review', // outside voice resolver
|
||
'office-hours', // second opinion resolver
|
||
];
|
||
|
||
const BOUNDARY_MARKER = /do not read or execute any/i;
|
||
|
||
test('boundary instruction appears in all skills that call codex', () => {
|
||
for (const skill of CODEX_CALLING_SKILLS) {
|
||
// Union: ship's codex call lives in sections/adversarial.md after the carve.
|
||
const content = readSkillUnion(skill);
|
||
expect(content).toMatch(BOUNDARY_MARKER);
|
||
}
|
||
});
|
||
|
||
test('codex skill has Filesystem Boundary section', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('## Filesystem Boundary');
|
||
});
|
||
|
||
test('codex skill has rabbit-hole detection rule', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Detect skill-file rabbit holes');
|
||
expect(content).toContain('gstack-update-check');
|
||
expect(content).toContain('Consider retrying');
|
||
});
|
||
|
||
test('outside-voice-steps.ts CODEX_BOUNDARY constant is interpolated into resolver output', () => {
|
||
// The adversarial step resolver should include boundary text in codex exec
|
||
// prompts. Carved: the adversarial step lives in sections/adversarial.md.
|
||
const reviewContent = readSkillUnion('review');
|
||
// Boundary should appear near codex exec invocations
|
||
const boundaryIdx = reviewContent.search(BOUNDARY_MARKER);
|
||
const codexExecIdx = reviewContent.indexOf('codex exec');
|
||
// Both must exist and boundary must come before a codex exec call
|
||
expect(boundaryIdx).toBeGreaterThan(-1);
|
||
expect(codexExecIdx).toBeGreaterThan(-1);
|
||
});
|
||
|
||
test('autoplan boundary text avoids host-specific paths for cross-host compatibility', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'autoplan', 'SKILL.md.tmpl'), 'utf-8');
|
||
// autoplan template uses generic 'skills/gstack' pattern instead of host-specific
|
||
// paths like ~/.claude/ or .agents/skills (which break Codex/Claude output tests)
|
||
const boundaryStart = content.indexOf('Filesystem Boundary');
|
||
const boundaryEnd = content.indexOf('---', boundaryStart + 1);
|
||
const boundarySection = content.slice(boundaryStart, boundaryEnd);
|
||
expect(boundarySection).not.toContain('~/.claude/');
|
||
expect(boundarySection).not.toContain('.agents/skills');
|
||
expect(boundarySection).toContain('skills/gstack');
|
||
expect(boundarySection).toMatch(BOUNDARY_MARKER);
|
||
});
|
||
});
|
||
|
||
// --- Codex outside-voice availability probe ---
|
||
|
||
describe('Codex outside voice is offered behind an executed availability probe', () => {
|
||
test.each(['office-hours', 'plan-ceo-review', 'plan-design-review', 'plan-eng-review'])('%s probes codex and has a not-installed fallback', skill => {
|
||
const content = readSkillUnion(skill);
|
||
expect(content).toContain('command -v codex');
|
||
expect(content).toContain('not_installed');
|
||
});
|
||
});
|
||
|
||
// --- {{BENEFITS_FROM}} resolver tests ---
|
||
|
||
describe('BENEFITS_FROM resolver', () => {
|
||
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||
const engContent = readSkillUnion('plan-eng-review'); // carved
|
||
|
||
test('plan-ceo-review contains prerequisite skill offer', () => {
|
||
expect(ceoContent).toContain('Prerequisite Skill Offer');
|
||
expect(ceoContent).toContain('/office-hours');
|
||
});
|
||
|
||
test('plan-ceo-review offers /office-hours when no design doc exists and detects a lost user mid-session', () => {
|
||
const offer = extractMarkdownSection(ceoContent, '## Prerequisite Skill Offer');
|
||
expect(offer.replace(/\s+/g, ' ')).toMatch(/when the design doc check above prints "No design doc found,?"/i);
|
||
expect(offer).toContain('A) Run /office-hours now');
|
||
expect(offer).toContain('B) Skip — proceed with standard review');
|
||
expectMentions(offer, [['do not','re-offer','session']], 'offer');
|
||
expect(ceoContent).toContain('**Mid-session detection (0A):**');
|
||
const midSession = ceoContent.indexOf('**Mid-session detection (0A):**');
|
||
expect(ceoContent.slice(midSession, midSession + 300)).toMatch(/offer `\/office-hours`/);
|
||
});
|
||
|
||
test('plan-eng-review contains prerequisite skill offer', () => {
|
||
expect(engContent).toContain('Prerequisite Skill Offer');
|
||
expect(engContent).toContain('/office-hours');
|
||
});
|
||
|
||
test('the optional Eng prerequisite ends before mandatory engineering review', () => {
|
||
const offer = extractMarkdownSection(engContent, '## Prerequisite Skill Offer');
|
||
expect(offer).toContain('B) Skip — proceed with standard review');
|
||
expectMentions(offer, [['do not','re-offer','session']], 'offer');
|
||
expect(offer).not.toContain('### Step 0: Scope Challenge');
|
||
const review = extractMarkdownSection(engContent, '## Engineering review');
|
||
expect(review).toContain('### Step 0: Scope Challenge');
|
||
expectMentions(review, [['before','challenge','mandatory']], 'review');
|
||
expect(review).toContain('sections/review-sections.md');
|
||
});
|
||
|
||
test('Eng initial and prerequisite recheck both discover the canonical override and fall back on slug failure', () => {
|
||
const initial = extractMarkdownSection(engContent, '### Design Doc Check').match(/```bash\n([\s\S]*?)\n```/)![1]!;
|
||
const offer = extractMarkdownSection(engContent, '## Prerequisite Skill Offer');
|
||
expectTokens(offer, ['**Design Doc Check**'], 'offer');
|
||
expectMentions(offer, [['block', 'office-hours', 'completes']], 'offer');
|
||
expect(offer).toContain('This is a fresh execution');
|
||
expectMentions(offer, [['do not','prerequisite','preamble']], 'offer');
|
||
expect(offer).not.toContain('_REVIEW_SLUG=');
|
||
const recheck = initial; // Execute the one canonical block again after the prerequisite.
|
||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'eng-design-slug-'));
|
||
try {
|
||
const home = path.join(dir, 'home'), cwd = path.join(dir, 'project');
|
||
const helper = path.join(home, '.claude/skills/gstack/bin/gstack-slug');
|
||
fs.mkdirSync(path.dirname(helper), {recursive: true});
|
||
fs.mkdirSync(cwd);
|
||
fs.copyFileSync(path.join(ROOT, 'bin/gstack-slug'), helper);
|
||
fs.chmodSync(helper, 0o755);
|
||
fs.copyFileSync(path.join(ROOT, 'bin/gstack-state-root.sh'), path.join(path.dirname(helper), 'gstack-state-root.sh'));
|
||
fs.copyFileSync(path.join(ROOT, 'bin/gstack-remote-identity.sh'), path.join(path.dirname(helper), 'gstack-remote-identity.sh'));
|
||
for (const bin of ['gstack-paths', 'gstack-design-doc-find']) {
|
||
fs.copyFileSync(path.join(ROOT, 'bin', bin), path.join(path.dirname(helper), bin));
|
||
fs.chmodSync(path.join(path.dirname(helper), bin), 0o755);
|
||
}
|
||
const expected = path.join(home, '.gstack/projects/canonical-override/session-unknown-design-current.md');
|
||
const wrong = path.join(home, '.gstack/projects/project/session-unknown-design-wrong.md');
|
||
for (const file of [expected, wrong]) {
|
||
fs.mkdirSync(path.dirname(file), {recursive: true});
|
||
fs.writeFileSync(file, '# Design fixture\n');
|
||
}
|
||
const env = {...process.env, HOME: home, GSTACK_HOME: path.join(home, '.gstack'),
|
||
GSTACK_PROJECT_SLUG: 'canonical-override', GIT_CEILING_DIRECTORIES: dir};
|
||
for (const command of [initial, recheck]) {
|
||
// No eval (#2763): worktree-isolated Claude Code sessions refuse it.
|
||
expect(command).toContain('if SLUG=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG); then');
|
||
expect(command).toContain('BRANCH=$(~/.claude/skills/gstack/bin/gstack-slug --get BRANCH)');
|
||
expect(command).not.toContain('eval ');
|
||
expect(command).toContain('echo "No design doc found"');
|
||
expect(command).not.toContain('remote-slug');
|
||
const result = runCapturedCommand('bash', ['-c', command], {cwd, env, timeout: 5000, captureStdout: true});
|
||
expect(result.status, result.stderr).toBe(0);
|
||
expect(result.stdout).toContain(`Design doc found: ${expected}`);
|
||
expect(result.stdout).not.toContain(wrong);
|
||
}
|
||
fs.writeFileSync(helper, '#!/usr/bin/env bash\necho "slug fixture failed" >&2\nexit 23\n', {mode: 0o755});
|
||
for (const command of [initial, recheck]) {
|
||
const result = runCapturedCommand('bash', ['-c', command], {cwd, env, timeout: 5000, captureStdout: true});
|
||
expect(result.status).toBe(0);
|
||
expect(result.stderr).toContain('slug fixture failed');
|
||
expect(result.stdout).not.toContain('Design doc found:');
|
||
expect(result.stdout).toContain('No design doc found');
|
||
}
|
||
} finally { fs.rmSync(dir, {recursive: true, force: true}); }
|
||
});
|
||
|
||
test('offer includes graceful decline', () => {
|
||
expect(ceoContent).toContain('No worries');
|
||
});
|
||
|
||
test('skills without benefits-from do NOT have prerequisite offer', () => {
|
||
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
|
||
expect(qaContent).not.toContain('Prerequisite Skill Offer');
|
||
});
|
||
|
||
test('inline invocation — no "another window" language', () => {
|
||
expect(ceoContent).not.toContain('another window');
|
||
expect(engContent).not.toContain('another window');
|
||
});
|
||
|
||
test('inline invocation — read-and-follow path present', () => {
|
||
expect(ceoContent).toContain('office-hours/SKILL.md');
|
||
expect(engContent).toContain('office-hours/SKILL.md');
|
||
});
|
||
|
||
test('BENEFITS_FROM delegates to INVOKE_SKILL pattern', () => {
|
||
// Should contain the INVOKE_SKILL-style loading prose (not the old manual skip list)
|
||
expect(engContent).toContain('skipping these sections');
|
||
});
|
||
});
|
||
|
||
// --- {{INVOKE_SKILL}} resolver tests ---
|
||
|
||
describe('INVOKE_SKILL resolver', () => {
|
||
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||
|
||
test('plan-ceo-review uses INVOKE_SKILL for mid-session office-hours fallback', () => {
|
||
// The mid-session detection path should use INVOKE_SKILL-generated prose
|
||
expect(ceoContent).toContain('office-hours/SKILL.md');
|
||
});
|
||
|
||
test('INVOKE_SKILL output includes default skip list', () => {
|
||
expect(ceoContent).toContain('Preamble (run first)');
|
||
expect(ceoContent).toContain('Telemetry (run last)');
|
||
expect(ceoContent).toContain('AskUserQuestion Format');
|
||
});
|
||
|
||
test('INVOKE_SKILL output includes error handling', () => {
|
||
expect(ceoContent).toContain('If unreadable');
|
||
expect(ceoContent).toContain('Could not load');
|
||
});
|
||
|
||
test('template uses {{INVOKE_SKILL:office-hours}} placeholder', () => {
|
||
const tmpl = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf-8');
|
||
expect(tmpl).toContain('{{INVOKE_SKILL:office-hours}}');
|
||
});
|
||
});
|
||
|
||
// --- {{CHANGELOG_WORKFLOW}} resolver tests ---
|
||
|
||
describe('CHANGELOG_WORKFLOW resolver', () => {
|
||
const shipContent = readShipUnion();
|
||
|
||
test('ship SKILL.md contains changelog workflow', () => {
|
||
expect(shipContent).toContain('CHANGELOG (auto-generate)');
|
||
expect(shipContent).toContain('git log origin/<base>..HEAD --oneline');
|
||
});
|
||
|
||
test('changelog workflow includes cross-check step', () => {
|
||
expect(shipContent).toContain('Cross-check');
|
||
const crossCheck = shipContent.slice(shipContent.indexOf('**Cross-check:**'), shipContent.indexOf('**Do NOT ask the user to describe changes.**'));
|
||
expect(crossCheck).toContain('commit list from step 2');
|
||
expect(crossCheck).toMatch(/user-facing/i);
|
||
});
|
||
|
||
test('changelog workflow includes voice guidance', () => {
|
||
expect(shipContent).toContain('Lead with what the user can now **do**');
|
||
});
|
||
|
||
test('template uses {{CHANGELOG_WORKFLOW}} placeholder', () => {
|
||
// Post-carve (T9): the skeleton points to the changelog section, which carries
|
||
// the resolver. Neither should inline the old changelog content.
|
||
const skel = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md.tmpl'), 'utf-8');
|
||
const changelogSection = fs.readFileSync(path.join(ROOT, 'ship', 'sections', 'changelog.md.tmpl'), 'utf-8');
|
||
expect(skel).toContain('{{SECTION:changelog}}');
|
||
expect(changelogSection).toContain('{{CHANGELOG_WORKFLOW}}');
|
||
expect(skel + changelogSection).not.toContain('Group commits by theme');
|
||
});
|
||
|
||
test('changelog workflow includes keep-changelog format', () => {
|
||
expect(shipContent).toContain('### Added');
|
||
expect(shipContent).toContain('### Fixed');
|
||
});
|
||
});
|
||
|
||
// --- Duplicate {{PREAMBLE}} guard (#2508/#2362) ---
|
||
|
||
describe('assertSinglePreamble', () => {
|
||
test('one {{PREAMBLE}} passes', () => {
|
||
expect(() => assertSinglePreamble('a\n{{PREAMBLE}}\nb', 'x/SKILL.md.tmpl')).not.toThrow();
|
||
});
|
||
|
||
test('zero {{PREAMBLE}} passes (sections have none)', () => {
|
||
expect(() => assertSinglePreamble('no macro here', 'x/sections/y.md.tmpl')).not.toThrow();
|
||
});
|
||
|
||
test('a second occurrence throws with the template path — even in prose', () => {
|
||
// The original #2508 bug WAS a prose mention: "emitted by {{PREAMBLE}}'s
|
||
// preamble bash". Resolution is context-blind, so the guard must be too.
|
||
const tmpl = '{{PREAMBLE}}\n\n...later: emitted by {{PREAMBLE}}\'s preamble bash';
|
||
expect(() => assertSinglePreamble(tmpl, 'spec/SKILL.md.tmpl')).toThrow(/spec\/SKILL\.md\.tmpl.*2 times/);
|
||
});
|
||
});
|
||
|
||
// --- Parameterized resolver infrastructure tests ---
|
||
|
||
describe('parameterized resolver support', () => {
|
||
test('gen-skill-docs regex handles colon-separated args', () => {
|
||
// Verify the template containing {{INVOKE_SKILL:office-hours}} was processed
|
||
// without leaving unresolved placeholders
|
||
const ceoContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||
expect(ceoContent).not.toMatch(/\{\{INVOKE_SKILL:[^}]+\}\}/);
|
||
});
|
||
|
||
test('templates with parameterized resolvers pass unresolved check', () => {
|
||
// All generated SKILL.md files should have no unresolved {{...}} placeholders
|
||
const skillDirs = fs.readdirSync(ROOT).filter(d =>
|
||
fs.existsSync(path.join(ROOT, d, 'SKILL.md'))
|
||
);
|
||
for (const dir of skillDirs) {
|
||
const content = fs.readFileSync(path.join(ROOT, dir, 'SKILL.md'), 'utf-8');
|
||
const unresolved = content.match(/\{\{[A-Z_]+(?::[^}]*)?\}\}/g);
|
||
if (unresolved) {
|
||
throw new Error(`${dir}/SKILL.md has unresolved placeholders: ${unresolved.join(', ')}`);
|
||
}
|
||
}
|
||
});
|
||
});
|
||
|
||
// --- Preamble routing injection tests ---
|
||
|
||
describe('preamble routing injection (bin/gstack-skill-start emission layer)', () => {
|
||
// Token-reduction Phase 2: the routing-injection prose left the rendered
|
||
// preamble entirely — bin/gstack-skill-start probes, gates, and emits the
|
||
// whole flow as a GSTACK_INSTRUCTION block (with the AUQ, the routing rules
|
||
// to append, and the decline ack all INSIDE the block). Absence from the
|
||
// renders is pinned by test/onboarding-moved-literals.test.ts (tombstone);
|
||
// this suite pins the gate structure and the emitted block's content.
|
||
const routingBlock = (() => {
|
||
const start = SKILL_START_SCRIPT.indexOf('_emit_block routing-injection');
|
||
expect(start).toBeGreaterThan(0);
|
||
return SKILL_START_SCRIPT.slice(start, SKILL_START_SCRIPT.indexOf('\nEOI', start));
|
||
})();
|
||
|
||
test('routing probe checks CLAUDE.md and AGENTS.md (now in gstack-skill-start)', () => {
|
||
// #2500: the probe iterates CLAUDE.md AND AGENTS.md — non-Claude hosts
|
||
// route skills via AGENTS.md, the cross-harness convention file.
|
||
expect(SKILL_START_SCRIPT).toContain('for _RF in CLAUDE.md AGENTS.md');
|
||
expect(SKILL_START_SCRIPT).toContain('grep -q "## Skill routing" "$_RF"');
|
||
expect(SKILL_START_SCRIPT).toContain('echo "HAS_ROUTING: $_HAS_ROUTING"');
|
||
});
|
||
|
||
test('script reads and echoes routing_declined config', () => {
|
||
expect(SKILL_START_SCRIPT).toMatch(/_ROUTING_DECLINED=\$\("\$_BIN\/gstack-config" get routing_declined/);
|
||
expect(SKILL_START_SCRIPT).toContain('echo "ROUTING_DECLINED: $_ROUTING_DECLINED"');
|
||
});
|
||
|
||
test('emitted block carries the routing injection AskUserQuestion', () => {
|
||
expect(routingBlock).toContain('Add routing rules to CLAUDE.md');
|
||
expect(routingBlock).toContain("I'll invoke skills manually");
|
||
});
|
||
|
||
test('routing injection respects prior decline (gate + in-block ack)', () => {
|
||
expect(SKILL_START_SCRIPT).toContain('[ "$_ROUTING_DECLINED" = "false" ]');
|
||
expect(routingBlock).toMatch(/routing_declined.*true/);
|
||
expect(routingBlock).toContain('re-enable with `__BIN__/gstack-config set routing_declined false`');
|
||
});
|
||
|
||
test('routing injection only fires when all conditions met', () => {
|
||
// Must be: HAS_ROUTING=no AND ROUTING_DECLINED=false AND PROACTIVE_PROMPTED=yes
|
||
expect(SKILL_START_SCRIPT).toContain(
|
||
'if [ "$_HAS_ROUTING" = "no" ] && [ "$_ROUTING_DECLINED" = "false" ] && [ "$_PROACTIVE_PROMPTED" = "yes" ]; then',
|
||
);
|
||
});
|
||
|
||
test('routing section content includes key routing rules', () => {
|
||
expect(routingBlock).toContain('invoke /office-hours');
|
||
expect(routingBlock).toContain('invoke /investigate');
|
||
expect(routingBlock).toContain('invoke /ship');
|
||
expect(routingBlock).toContain('invoke /qa');
|
||
});
|
||
|
||
test('routing section uses renamed checkpoint skills (not stale /checkpoint)', () => {
|
||
expect(routingBlock).toContain('invoke /context-save');
|
||
expect(routingBlock).toContain('invoke /context-restore');
|
||
expect(routingBlock).not.toContain('invoke checkpoint');
|
||
});
|
||
|
||
test('routing section uses soft "when in doubt" policy, not hard "ALWAYS invoke"', () => {
|
||
expect(routingBlock).toContain('When in doubt, invoke the skill');
|
||
expect(routingBlock).not.toContain('Do NOT answer directly');
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_OUTSIDE_VOICES}} resolver tests ---
|
||
|
||
describe('DESIGN_OUTSIDE_VOICES resolver', () => {
|
||
test('plan-design-review contains outside voices section', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Design Outside Voices');
|
||
expect(content).toContain('CODEX_MODE: ready');
|
||
expect(content).toContain('LITMUS SCORECARD');
|
||
});
|
||
|
||
test('design-review contains outside voices section', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Design Outside Voices');
|
||
expect(content).toContain('source audit');
|
||
});
|
||
|
||
test('design-consultation contains outside voices section', () => {
|
||
const content = readSkillUnion('design-consultation');
|
||
expect(content).toContain('Design Outside Voices');
|
||
expect(content).toContain('design direction');
|
||
});
|
||
|
||
test('consultation dispatch shares the complete brief without executing product text', () => {
|
||
const content = readSkillUnion('design-consultation');
|
||
const command = [...content.matchAll(/```bash\n([\s\S]*?)```/g)]
|
||
.map(match => match[1]).find(block => block.includes('codex exec'))!;
|
||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'design-brief-contract-'));
|
||
const quote = (value: string) => "'" + value.replaceAll("'", "'\\''") + "'";
|
||
try {
|
||
const brief = path.join(dir, 'product.md');
|
||
const capture = path.join(dir, 'argv.json');
|
||
const marker = path.join(dir, 'must-not-execute');
|
||
const product = `A product with $(touch ${marker}), \`touch ${marker}\`, and "quotes".\nUsers: builders.\n`;
|
||
fs.writeFileSync(brief, product);
|
||
// Fake codex: the free sandbox preflight succeeds; exec writes its final message to -o.
|
||
fs.writeFileSync(path.join(dir, 'codex'), `#!${process.execPath}\nconst fs = require('fs'); const a = process.argv.slice(2); if (a[0] === 'sandbox') process.exit(0);\nfs.writeFileSync(process.env.CAPTURE, JSON.stringify({ args: a, stdin: a[1] === '-' ? fs.readFileSync(0, 'utf8') : '' }));\nconst msg = 'Recommendation: choose a clear hierarchy because builders need to find their work.';\nif (a.includes('-o')) fs.writeFileSync(a[a.indexOf('-o') + 1], msg);\nconsole.log(msg);\n`, { mode: 0o700 });
|
||
const prepare = (file: string) => command.replace("'<prepared-prompt-file>'", quote(file))
|
||
.replaceAll('$HOME/.claude/skills/gstack', ROOT);
|
||
const env = { ...process.env, PATH: dir + path.delimiter + process.env.PATH, CAPTURE: capture,
|
||
CODEX_THREAD_ID: '', CODEX_SANDBOX: '', CLAUDECODE: '1', GSTACK_ACTIVE_HOST: 'claude',
|
||
GSTACK_HOME: path.join(dir, 'state'), GSTACK_STATE_ROOT: '' };
|
||
const result = spawnSync('bash', ['-c', prepare(brief)], { cwd: ROOT, env, encoding: 'utf8', timeout: 5_000 });
|
||
expect(result.status, result.stderr).toBe(0);
|
||
const { args, stdin } = JSON.parse(fs.readFileSync(capture, 'utf8'));
|
||
expect(args[0]).toBe('exec');
|
||
expect(args[1]).toBe('-');
|
||
expect(stdin).toBe(product);
|
||
expect(args).toContain('read-only');
|
||
expect(fs.readFileSync(brief, 'utf8')).toBe(product);
|
||
expect(fs.existsSync(marker)).toBe(false);
|
||
fs.unlinkSync(capture);
|
||
const missing = spawnSync('bash', ['-c', prepare(path.join(dir, 'absent'))], { cwd: ROOT, env, encoding: 'utf8', timeout: 5_000 });
|
||
expect(missing.status).not.toBe(0);
|
||
expect(missing.stderr).toContain('No such file');
|
||
expect(fs.existsSync(capture)).toBe(false);
|
||
expect(content).toContain('await both before synthesis');
|
||
expect(content).toContain('use only the native voice');
|
||
expect(content).toContain('give the native Agent its absolute path');
|
||
expect(content).toContain('Read the complete product brief at [the absolute DESIGN_BRIEF path printed above]');
|
||
expect(content).toContain('Omit faces you cannot verify');
|
||
expect(content).toContain('a face may serve multiple roles');
|
||
expect(content).not.toContain('a single question that covers everything');
|
||
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
|
||
});
|
||
|
||
test('branches correctly per skillName — different prompts', () => {
|
||
const planContent = readSkillUnion('plan-design-review');
|
||
const consultContent = readSkillUnion('design-consultation');
|
||
// plan-design-review uses analytical prompt (high reasoning)
|
||
expect(planContent).toContain('model_reasoning_effort="high"');
|
||
// design-consultation uses creative prompt (medium reasoning)
|
||
expect(consultContent).toContain('model_reasoning_effort="medium"');
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_HARD_RULES}} resolver tests ---
|
||
|
||
describe('DESIGN_HARD_RULES resolver', () => {
|
||
test('plan-design-review Pass 4 contains hard rules', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Design Hard Rules');
|
||
expect(content).toContain('Classifier');
|
||
expect(content).toContain('MARKETING/LANDING PAGE');
|
||
expect(content).toContain('APP UI');
|
||
});
|
||
|
||
test('design-review contains hard rules', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Design Hard Rules');
|
||
});
|
||
|
||
test('includes all 3 rule sets', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Landing page rules');
|
||
expect(content).toContain('App UI rules');
|
||
expect(content).toContain('Universal rules');
|
||
});
|
||
|
||
test('classifier names the four visitor modes and keeps the legacy aliases', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
for (const mode of ['PERSUADE', 'OPERATE', 'READ', 'EXPERIENCE', 'HYBRID']) expect(content).toContain(`**${mode}**`);
|
||
expect(content).toContain('Read rules');
|
||
expect(content).toContain('Experience rules');
|
||
expectMentions(content, [['not', 'classify', 'section']], 'content');
|
||
});
|
||
|
||
test('carries the craft-floor reflexes and the three-looks calibration', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Reflexes no detector catches');
|
||
expect(content).toContain('Browser surfaces carry the design');
|
||
expect(content).toContain('One authored motion moment');
|
||
expect(content).toContain('Depth has an offset');
|
||
expect(content).toContain('Light or dark comes from the use scene');
|
||
expect(content).toContain('Calibration: the three looks');
|
||
});
|
||
|
||
test('slop section lists detector rule ids and judgment tells outside design-review', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('nested-cards: Nested cards');
|
||
expectMentions(content, [['no', 'judgment', 'detector']], 'content');
|
||
// Never a bracketed gstack-only id.
|
||
expect(content).not.toContain('[hero-metrics]');
|
||
});
|
||
|
||
test('design-consultation carries the font procedure, role-scoped lists, color strategies, and catalog bullets', () => {
|
||
const content = readSkillUnion('design-consultation');
|
||
expect(content).toContain('Choosing faces: a procedure, not a menu');
|
||
expect(content).toContain('**Overused as display**');
|
||
expect(content).toContain('**Banned in any role:** Papyrus');
|
||
expect(content).toContain('Restrained (1 accent + neutrals');
|
||
expectMentions(content, [['not', 'light', 'dials']], 'content');
|
||
expect(content).toContain('Calibration: the three looks');
|
||
// Bullets are prose only: never a bracketed rule id in the proposal skill.
|
||
expect(content).toContain('- A card inside a card is always wrong.');
|
||
expect(content).not.toMatch(/^- \[[a-z-]+\] /m);
|
||
// The old menu is gone.
|
||
expect(content).not.toContain('Font recommendations by purpose');
|
||
});
|
||
|
||
test('design-html blacklist lines carry catalog ids', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
||
expect(content).toMatch(/AI slop blacklist/i);
|
||
expect(content).toContain('Purple/blue gradients as default <!-- ai-color-palette -->');
|
||
expect(content).toContain('lib/design-catalog.ts');
|
||
});
|
||
|
||
test('design-review renders the catalog once: Methodology category 9 carries it, Hard Rules points at it', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expect(content.split('### Design Hard Rules').length - 1).toBe(1);
|
||
// Category 9 lists the rule once (ids only); Typography points at the same id from its overused-face item.
|
||
expect(content.split('[overused-font]').length - 1).toBe(2);
|
||
expect(content).toContain('are Methodology category 9');
|
||
expect(content).toContain('**9. AI Slop Detection**');
|
||
expect(content).toContain('Detector rules (ids only;');
|
||
expect(content).toContain('[nested-cards] nested cards');
|
||
expect(content).toContain('Judgment tells (no detector rule');
|
||
// The legacy blacklist is not repeated as a numbered list in design-review.
|
||
expect(content).not.toMatch(/^1\. Purple\/violet\/indigo/m);
|
||
});
|
||
|
||
test('references shared AI slop blacklist items', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('3-column feature grid');
|
||
expect(content).toContain('Purple/violet/indigo');
|
||
});
|
||
|
||
test('includes OpenAI hard rejection criteria', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Generic SaaS card grid');
|
||
expect(content).toContain('Carousel with no narrative purpose');
|
||
});
|
||
|
||
test('includes OpenAI litmus checks', () => {
|
||
const content = readSkillUnion('plan-design-review');
|
||
expect(content).toContain('Brand/product unmistakable');
|
||
});
|
||
});
|
||
|
||
describe('Design approval reconciliation', () => {
|
||
test('token alignment reuses the approved outcome while preserving first decisions and new tradeoffs', () => {
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-design-review/sections/review-sections.md'), 'utf8');
|
||
const pass = extractMarkdownSection(section, '### Pass 5: Design System Alignment');
|
||
const ask = pass.search(/stop\W+AskUserQuestion once per issue/i);
|
||
expect(ask).toBeGreaterThan(0);
|
||
const beforeQuestion = pass.slice(0, ask);
|
||
expect(beforeQuestion).toMatch(/already approved that outcome/i);
|
||
expect(beforeQuestion).toMatch(/not a new issue/i);
|
||
expectMentions(beforeQuestion, [['only','evidence','again']], 'beforeQuestion');
|
||
});
|
||
|
||
test('loaded review section distinguishes individual issue decisions from navigation', () => {
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-design-review/sections/review-sections.md'), 'utf8').replace(/\s+/g, ' ');
|
||
expectMentions(section, [['no','next-step','remedies']], 'section');
|
||
expectMentions(section, [['never','askuserquestion','issue-approval']], 'section');
|
||
expectMentions(section, [['no','unresolved','findings']], 'section');
|
||
});
|
||
|
||
test('Design blocking Exit checklist reconciles approvals and refreshes stale output after a decision', () => {
|
||
const main = fs.readFileSync(path.join(ROOT, 'plan-design-review/SKILL.md'), 'utf8');
|
||
const check = extractMarkdownSection(main, '## Section self-check');
|
||
expect(check).toMatch(/run approval check 0/i);
|
||
const gate = extractMarkdownSection(main, '## EXIT PLAN MODE GATE (BLOCKING)').replace(/\s+/g, ' ');
|
||
ordered(gate, ['0. Approvals:', '1. Read the plan file']);
|
||
const approvals = gate.slice(gate.indexOf('0. Approvals:'), gate.indexOf('1. Read the plan file'));
|
||
expect(approvals).toMatch(/never group distinct issues/i);
|
||
expect(approvals).toMatch(/navigation are not approval/i);
|
||
expect(approvals).toContain('preamble-authorized');
|
||
expect(approvals).toMatch(/deferrals remain unresolved/i);
|
||
expectMentions(approvals, [['wait','pending','drafts']], 'approvals');
|
||
ordered(approvals, ['refresh the plan and report', 'Read-back gate', 'rerun this gate']);
|
||
});
|
||
|
||
test('CEO readiness binds actual approvals before its read-only exit verification', () => {
|
||
const main = fs.readFileSync(path.join(ROOT, 'plan-ceo-review/SKILL.md'), 'utf8');
|
||
const check = extractMarkdownSection(main, '## Section self-check').replace(/\s+/g, ' ');
|
||
const gate = extractMarkdownSection(main, '## EXIT PLAN MODE GATE (BLOCKING)').replace(/\s+/g, ' ');
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-ceo-review/sections/review-sections.md'), 'utf8');
|
||
const readiness = extractMarkdownSection(section, '## Approval readiness').replace(/\s+/g, ' ');
|
||
expect(section.indexOf('## Approval readiness')).toBeLessThan(section.indexOf('## Required Outputs'));
|
||
expectMentions(readiness, [['approval','preamble-authorized','exact']], 'readiness');
|
||
expectMentions(readiness, [['not','navigation','approvals']], 'readiness');
|
||
expectMentions(readiness, [['only','commitments','approves']], 'readiness');
|
||
expectMentions(readiness, [['only','applies','scope']], 'readiness');
|
||
expect(readiness).toMatch(/delivery-scope deferral is settled/i);
|
||
expect(readiness).toMatch(/leaves that choice unresolved/i);
|
||
expect(readiness).toContain('`Approval readiness: PASS`');
|
||
expect(readiness).toMatch(/navigation alone does not/i);
|
||
expect(main.replace(/\s+/g, ' ')).toMatch(/default to implementation-ready/i);
|
||
ordered(gate, ['Read-only verification', 'Verify `Approval readiness: PASS`', '1. Read the plan file']);
|
||
expectMentions(gate, [['only','readiness','return']], 'gate');
|
||
expect(gate).toContain('**Gate outcome: Blocked**');
|
||
expect(main).toContain('**Pass with log-only gaps:**');
|
||
expectMentions(gate, [['without','exitplanmode','telemetry']], 'gate');
|
||
expect(check).toContain('`sections/review-sections.md`');
|
||
expectMentions(check, [['stop','review','read']], 'check');
|
||
const decisions = main.slice(main.indexOf('### 0D.'), main.indexOf('### 0E.')).replace(/\s+/g, ' ');
|
||
ordered(decisions, ["**Choose the question's route first:**", '**Admin question:**', '**Plan decision:**']);
|
||
expect(decisions).toMatch(/reuse exact prior approvals/i);
|
||
expect(decisions).toContain('`D<N> — <ROW-ID>: <one-line question>`');
|
||
const row = decisions.slice(decisions.indexOf('**Pre-question checkpoint:**'), decisions.indexOf('Effort/risk must'));
|
||
for (const field of ['ID', 'owner', 'Current/Proposed', 'Status', 'Exact approval and scope']) expect(row).toContain(field);
|
||
expectMentions(decisions.slice(decisions.indexOf('**4. Ask')), [['stop','actual','answer']], 'section');
|
||
expect(decisions).toMatch(/amend only (?:what it authorizes|authorized work)/i);
|
||
expectMentions(decisions, [['only','contradictions','instructions']], 'decisions');
|
||
expect(decisions).toContain('"No issues, moving on."');
|
||
const outside = extractMarkdownSection(section, '**Cross-model tension:**').replace(/\s+/g, ' ');
|
||
expectMentions(outside, [['only','completed','external']], 'outside');
|
||
expect(outside).toContain('CROSS-MODEL');
|
||
const findings = extractMarkdownSection(section, '**Integrate reviewer findings:**').replace(/\s+/g, ' ');
|
||
expect(findings).toMatch(/bounded native fallback/i);
|
||
expect(findings).toContain('Outside Voice Integration Rule');
|
||
});
|
||
|
||
test('Eng cannot exit with unasked findings listed only in an unresolved-decisions report', () => {
|
||
const main = fs.readFileSync(path.join(ROOT, 'plan-eng-review/SKILL.md'), 'utf8');
|
||
const rawGate = extractMarkdownSection(main, '## EXIT PLAN MODE GATE (BLOCKING)');
|
||
const gate = rawGate.replace(/\s+/g, ' ');
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-eng-review/sections/review-sections.md'), 'utf8');
|
||
const readiness = extractMarkdownSection(section, '## Approval readiness');
|
||
const normalizedReadiness = readiness.replace(/\s+/g, ' ');
|
||
expect(section.indexOf('## Approval readiness')).toBeLessThan(section.indexOf('## Required outputs'));
|
||
expect(normalizedReadiness).toContain('`Approval readiness: PASS`');
|
||
expect(normalizedReadiness).toMatch(/navigation alone does not/i);
|
||
expect(readiness).not.toMatch(/^ {3}\S/m);
|
||
expect(gate).toMatch(/confirm approval readiness passed/i);
|
||
expectMentions(gate, [['not','verification','read-only']], 'gate');
|
||
const approvalParagraph = rawGate.slice(rawGate.indexOf('Confirm Approval readiness'), rawGate.indexOf('Verify all five checks'));
|
||
expect(approvalParagraph).not.toMatch(/^ {3}\S/m);
|
||
expect([...rawGate.matchAll(/^([1-5])\. /gm)].map(match => match[1])).toEqual(['1', '2', '3', '4', '5']);
|
||
ordered(gate, ['Confirm Approval readiness', '1. Read the report file']);
|
||
expect(normalizedReadiness).toMatch(/every accepted remedy/i);
|
||
expectMentions(normalizedReadiness, [['approval','auto-decision','authorized']], 'normalizedReadiness');
|
||
expect(normalizedReadiness).toMatch(/navigation do not count/i);
|
||
expect(normalizedReadiness).toMatch(/deferrals remain unresolved/i);
|
||
expect(normalizedReadiness).toMatch(/one dedicated decision/i);
|
||
expect(normalizedReadiness).toMatch(/mark that draft pending/i);
|
||
const decisionStart = section.indexOf('\n## Decision procedure\n');
|
||
const decisionEnd = section.indexOf('\n## Scope Challenge\n', decisionStart);
|
||
expect(decisionStart).toBeGreaterThan(0);
|
||
expect(decisionEnd).toBeGreaterThan(decisionStart);
|
||
const decisions = section.slice(decisionStart, decisionEnd).replace(/\s+/g, ' ');
|
||
expect(decisions).toContain('AskUserQuestion({ questions: [currentDecision] })');
|
||
const ask = decisions.indexOf('AskUserQuestion({ questions: [currentDecision] })');
|
||
const stop = decisions.search(/stop until the actual answer arrives/i);
|
||
expect(stop).toBeGreaterThan(ask);
|
||
expect(stop).toBeLessThan(decisions.indexOf('### Record the answer'));
|
||
expectMentions(gate, [['stop','verification','report']], 'gate');
|
||
expect(gate).toContain('**Recovery routing → Late change or missing work**');
|
||
const recovery = readSkillUnion('plan-eng-review').split('**Late change or missing work:**')[1]!.split('**Blocked outcome:**')[0]!.replace(/\s+/g, ' ');
|
||
ordered(recovery, ['Repeat Approval readiness', 'Required outputs', 'navigation']);
|
||
expect(gate).toContain('**Blocked outcome**');
|
||
const report = extractMarkdownSection(section, '### Write to the report file');
|
||
expect(report).toContain('**Blocked outcome**');
|
||
expect(report.replace(/\s+/g, ' ')).toMatch(/follow \*\*Blocked outcome\*\* before Review Log or decision logging/i);
|
||
});
|
||
|
||
test('approval entry does not alter other review Exit checklists', () => {
|
||
for (const skill of ['plan-devex-review']) {
|
||
const gate = extractMarkdownSection(readSkillUnion(skill), '## EXIT PLAN MODE GATE (BLOCKING)');
|
||
expect(gate).not.toContain('0. Approvals:');
|
||
expect(gate).toContain('1. Read the plan file');
|
||
expect(gate).toContain('5. If a plan file is in context');
|
||
}
|
||
});
|
||
|
||
test('task and decision reporting require approval while preserving unanswered findings', () => {
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-design-review/sections/review-sections.md'), 'utf8').replace(/\s+/g, ' ');
|
||
const reconcile = section.search(/before synthesizing tasks or the completion summary/i);
|
||
expect(reconcile).toBeGreaterThan(0);
|
||
expect(reconcile).toBeLessThan(section.indexOf('## Implementation Tasks'));
|
||
expectMentions(section, [['only','individually','decisions']], 'section');
|
||
expectMentions(section, [['not','including','finding']], 'section');
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_DETECTOR}} resolver tests ---
|
||
|
||
describe('DESIGN_DETECTOR resolver', () => {
|
||
const designReview = () => fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
const designHtml = () => fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
||
const bashBlocksOf = (content: string) => [...content.matchAll(/```bash\n([\s\S]*?)```/g)].map(m => m[1]);
|
||
|
||
test('design-review carries the probe, Phase 0, the DOM dump, and the run id', () => {
|
||
const c = designReview();
|
||
expect(c).toContain('gstack-design-detect.ts probe --host claude');
|
||
expect(c).toContain('IMPECCABLE_READY');
|
||
// the consent-gated install: offered once, only on the probe's say-so, never in spawned sessions, never via npx
|
||
expect(c).toContain('DESIGN_DETECTOR_INSTALL_OFFER');
|
||
expect(c).toContain('gstack-design-detect.ts install --host claude');
|
||
expect(c).toContain("Install impeccable's design detector engine?");
|
||
expect(c).toContain('gstack-config set design_detector_install_prompted true');
|
||
expect(c).toContain('`SESSION_KIND: spawned` or a headless run, never install and never ask');
|
||
expect(c).toContain('**Phase 0: mechanical scan**');
|
||
expect(c).toContain('scan --changed <base> --format gstack --host claude');
|
||
expect(c).toContain('### DOM dump (DOM mode only');
|
||
expect(c).toContain('data-gstack-dom-css');
|
||
expect(c).toContain(`$B js '('"$_DUMP"')()' --out "$_TMP/{page}.dom.html" --raw`);
|
||
expect(c).toContain('DOM_DUMP_OK');
|
||
expect(c).toContain('DOM_DUMP_REDACTION_BLOCKED');
|
||
expect(c).toContain('DOM_DUMP_TOO_LARGE');
|
||
expect(c).toContain('REPORT_DIR="$GSTACK_STATE_ROOT/projects/$SLUG/designs/design-audit-$(date +%Y%m%d)"');
|
||
expect(c).toContain('RUN_ID="$(date +%H%M%S)-$$"');
|
||
expect(c).toContain('"schemaVersion": 2');
|
||
expect(c).toContain('engine changed X → Y; rule set may differ');
|
||
expect(c).toContain('Detector: N → M');
|
||
expect(c).toContain('/impeccable typeset');
|
||
});
|
||
|
||
test('the DOM-dump script is loaded from lib/dom-dump.js, never inlined in the prose', () => {
|
||
const c = designReview();
|
||
expect(c).not.toMatch(/```js\n/);
|
||
expect(c).not.toContain('document.documentElement.cloneNode');
|
||
expect(c).toContain('_DUMP=$(cat "$HOME/.claude/skills/gstack/lib/dom-dump.js")');
|
||
expect(c).toContain(`const html = await pg.evaluate('"$_DUMP"');`);
|
||
expect(c).toContain('_TMP=$(mktemp -d "${TMPDIR:-/tmp}/gstack-dom.XXXXXX"); _DUMP=$(cat "$HOME/.claude/skills/gstack/lib/dom-dump.js")');
|
||
});
|
||
|
||
test('every rendered Aside script is single-quoted: a page-controlled <url> is never inside a double-quoted bash string', () => {
|
||
const files = [...fs.readdirSync(ROOT).filter(d => fs.existsSync(path.join(ROOT, d, 'SKILL.md'))).map(d => path.join(ROOT, d, 'SKILL.md')),
|
||
...fs.readdirSync(ROOT).flatMap(d => fs.existsSync(path.join(ROOT, d, 'sections')) ? fs.readdirSync(path.join(ROOT, d, 'sections')).filter(f => f.endsWith('.md')).map(f => path.join(ROOT, d, 'sections', f)) : [])];
|
||
expect(files.length).toBeGreaterThan(10);
|
||
for (const f of files) {
|
||
const c = fs.readFileSync(f, 'utf-8');
|
||
expect(c, path.relative(ROOT, f)).not.toMatch(/^aside repl "/m);
|
||
}
|
||
});
|
||
|
||
test('the E2E fixture slice markers exist in the rendered design skills (a template rename fails here, not in paid CI)', () => {
|
||
const dr = designReview();
|
||
const dh = fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
||
for (const [a, b] of [['**Design detector (optional, deterministic):**', '**Create output directories:**'], ['**Phase 0: mechanical scan**', '## Phases 1-6'], ['### DOM dump (DOM mode only', '### Auth Detection']]) {
|
||
expect(sliceBetween(dr, a, b).length, `${a} .. ${b}`).toBeGreaterThan(100);
|
||
}
|
||
for (const [a, b] of [['**Design detector (optional, deterministic):**', '## Step 0: Input Detection'], ['### Slop Gate (bounded, never a loop)', '### Verification Screenshots']]) {
|
||
expect(sliceBetween(dh, a, b).length, `${a} .. ${b}`).toBeGreaterThan(100);
|
||
}
|
||
});
|
||
|
||
test('design-html carries the probe and the bounded slop gate', () => {
|
||
const c = designHtml();
|
||
expect(c).toContain('gstack-design-detect.ts probe --host claude');
|
||
expect(c).toContain('### Slop Gate (bounded, never a loop)');
|
||
expect(c).toContain('One pass, not a loop.');
|
||
expect(c).toContain('impeccable-disable <rule>: <reason>');
|
||
});
|
||
|
||
test('ship and review unions reach the detector through review-lite and the checklist', () => {
|
||
const ship = readSkillUnion('ship');
|
||
expect(ship).toContain('**Mechanical pass first.**');
|
||
expect(ship).toContain('scan --changed <base> --format gstack --host claude');
|
||
expect(ship).toContain('"detector":D');
|
||
expect(ship).toContain('Detector: "clean" | "N findings');
|
||
const review = readSkillUnion('review');
|
||
const checklist = fs.readFileSync(path.join(ROOT, 'review', 'design-checklist.md'), 'utf-8');
|
||
expect(checklist).toContain('**0. Mechanical pass first.**');
|
||
expect(checklist).toContain('IMPECCABLE_READY');
|
||
});
|
||
|
||
test('every rendered invocation uses bun --no-env-file and ends a scan with the exit echo; no bash block runs npx impeccable', () => {
|
||
for (const content of [designReview(), designHtml(), readSkillUnion('ship'), readSkillUnion('review'), fs.readFileSync(path.join(ROOT, 'review', 'design-checklist.md'), 'utf-8')]) {
|
||
for (const block of bashBlocksOf(content)) {
|
||
expect(block).not.toContain('npx impeccable');
|
||
for (const line of block.split('\n')) {
|
||
if (!line.includes('gstack-design-detect.ts')) continue;
|
||
expect(line).toContain('bun --no-env-file run ');
|
||
if (/gstack-design-detect\.ts scan /.test(line)) expect(line).toContain('echo "DETECT_EXIT_CODE=$?"');
|
||
}
|
||
}
|
||
}
|
||
});
|
||
|
||
test('--host is rendered per host', () => {
|
||
// Fresh codex render into a temp out-dir: the tracked tree is Claude-only and
|
||
// the gitignored .agents/ copy may be stale.
|
||
const out = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-detector-host-'));
|
||
try {
|
||
const r = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--out-dir', out], { cwd: ROOT, timeout: 120_000 });
|
||
expect(r.exitCode).toBe(0);
|
||
const codex = fs.readFileSync(path.join(out, '.agents', 'skills', 'gstack-design-review', 'SKILL.md'), 'utf-8');
|
||
expect(codex).toContain('gstack-design-detect.ts probe --host codex');
|
||
expect(codex).not.toContain('probe --host claude');
|
||
expect(codex).toContain('$GSTACK_ROOT/lib/dom-dump.js');
|
||
} finally {
|
||
fs.rmSync(out, { recursive: true, force: true });
|
||
}
|
||
});
|
||
});
|
||
|
||
// --- {{DESIGN_MD_CHECK}} resolver + open DESIGN.md adoption ---
|
||
|
||
describe('DESIGN_MD_CHECK resolver and open DESIGN.md adoption', () => {
|
||
test('design-consultation asks the conversion question once and writes the spec form', () => {
|
||
const c = readSkillUnion('design-consultation');
|
||
expect(c).toContain('gstack-design-md.ts check DESIGN.md');
|
||
expect(c).toContain('DESIGN_MD_FORMAT: spec');
|
||
expect(c).toContain('mark legacy-keep');
|
||
expect(c).toContain('convert --write');
|
||
expect(c).toContain('# gstack: design-md-format=spec');
|
||
expect(c).toContain("## Do's and Don'ts");
|
||
expect(c).toContain('## Elevation & Depth');
|
||
expect(c).toContain('fontFeature: tnum');
|
||
expect(c).toContain('"{colors.primary}"');
|
||
// the legacy template is gone
|
||
expect(c).not.toContain('## Product Context\n- **What this is:**');
|
||
});
|
||
|
||
test('design-review calibrates against tokens and never re-offers conversion; design-html writes the spec form', () => {
|
||
const dr = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expect(dr).toContain('gstack-design-md.ts check DESIGN.md');
|
||
expect(dr).toContain('gstack-design-md.ts tokens DESIGN.md');
|
||
expect(dr).toMatch(/never offer a conversion/i);
|
||
expect(dr).not.toContain('mark legacy-keep');
|
||
const dh = fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
||
expect(dh).toContain('# gstack: design-md-format=spec');
|
||
const pdr = readSkillUnion('plan-design-review');
|
||
expect(pdr).toContain('{colors.primary}');
|
||
const checklist = fs.readFileSync(path.join(ROOT, 'review', 'design-checklist.md'), 'utf-8');
|
||
expect(checklist).toContain('gstack-design-md.ts tokens DESIGN.md');
|
||
expect(readSkillUnion('ship')).toContain('gstack-design-md.ts tokens DESIGN.md');
|
||
});
|
||
|
||
test('every rendered gstack-design-md invocation uses bun --no-env-file', () => {
|
||
for (const content of [readSkillUnion('design-consultation'), fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8'), readSkillUnion('ship')]) {
|
||
for (const line of content.split('\n')) {
|
||
if (line.includes('gstack-design-md.ts')) expect(line).toContain('bun --no-env-file run ');
|
||
}
|
||
}
|
||
});
|
||
});
|
||
|
||
// --- PRODUCT.md prefill + /impeccable handoffs ---
|
||
|
||
describe('PRODUCT.md prefill and /impeccable handoffs', () => {
|
||
test('design-consultation and design-shotgun read PRODUCT.md and never open the impeccable skill', () => {
|
||
for (const skill of ['design-consultation', 'design-shotgun']) {
|
||
const c = readSkillUnion(skill);
|
||
expect(c).toContain('cat PRODUCT.md 2>/dev/null | head -120 || echo "NO_PRODUCT_MD"');
|
||
expect(c).toContain('do not re-ask');
|
||
expect(c).toContain('Never open `.claude/skills/impeccable/**`');
|
||
}
|
||
});
|
||
|
||
test('handoffs are gated on IMPECCABLE_SKILL: present in review-lite and design-review', () => {
|
||
expect(readSkillUnion('ship')).toContain('IMPECCABLE_SKILL: present`, end each NEEDS INPUT detector row with the `handoff=` command');
|
||
const dr = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
||
expectTokens(dr, ['`handoff=`', '`IMPECCABLE_SKILL: present`'], 'dr');
|
||
expectTokens(dr, ['`/impeccable`'], 'dr');
|
||
});
|
||
});
|
||
|
||
// --- Extended DESIGN_SKETCH resolver tests ---
|
||
|
||
describe('DESIGN_SKETCH extended with outside voices', () => {
|
||
const content = readSkillUnion('office-hours'); // carved: Phase 5/6 prose moved to section
|
||
|
||
test('contains outside design voices step', () => {
|
||
expect(content).toContain('Outside design voices');
|
||
});
|
||
|
||
test('offers opt-in via AskUserQuestion', () => {
|
||
expect(content).toContain('outside design perspectives');
|
||
});
|
||
|
||
test('still contains original wireframe steps', () => {
|
||
expect(content).toContain('wireframe');
|
||
expect(content).toContain('gstack-render.ts');
|
||
});
|
||
});
|
||
|
||
// --- Extended DESIGN_REVIEW_LITE resolver tests ---
|
||
|
||
describe('DESIGN_REVIEW_LITE extended with Codex', () => {
|
||
const content = readShipUnion();
|
||
|
||
test('contains Codex design voice block', () => {
|
||
expect(content).toContain('Codex design voice');
|
||
expect(content).toContain('CODEX (design)');
|
||
});
|
||
|
||
test('still contains original checklist steps', () => {
|
||
expect(content).toContain('design-checklist.md');
|
||
expect(content).toContain('SCOPE_FRONTEND');
|
||
});
|
||
|
||
test('design-checklist path uses installed gstack/review root (#2694)', () => {
|
||
// #2694: generateDesignReviewLite used to emit
|
||
// `.claude/skills/review/design-checklist.md` (missing the gstack/ segment).
|
||
// After install the file lives at ~/.claude/skills/gstack/review/design-checklist.md.
|
||
// The bad relative form must not appear — the good path does not contain it
|
||
// as a substring because `gstack/` sits between `skills/` and `review/`.
|
||
expect(content).toContain('~/.claude/skills/gstack/review/design-checklist.md');
|
||
expect(content).not.toContain('.claude/skills/review/design-checklist.md');
|
||
});
|
||
|
||
});
|
||
|
||
// ─── Codex Generation Tests ─────────────────────────────────
|
||
|
||
describe('Codex generation (--host codex)', () => {
|
||
// .agents/ is gitignored (v0.11.2.0) — read the module-level out-dir render
|
||
// (--host all covers codex) instead of regenerating the live tree in place.
|
||
const AGENTS_DIR = path.join(EXTERNAL_OUT, '.agents', 'skills');
|
||
|
||
// Dynamic discovery of expected Codex skills: all templates except /codex.
|
||
// The out-dir is a fresh mkdtemp, so the vendored-dev-mode symlink loop
|
||
// (.agents/skills/{name} → repo root) that made the generator skip skills
|
||
// in-place can never occur here — every template renders.
|
||
const CODEX_SKILLS = (() => {
|
||
const skills: Array<{ dir: string; codexName: string }> = [];
|
||
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
||
skills.push({ dir: '.', codexName: 'gstack' });
|
||
}
|
||
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
||
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||
if (entry.name === 'codex') continue; // /codex is excluded from Codex output
|
||
if (!fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) continue;
|
||
const codexName = entry.name.startsWith('gstack-') ? entry.name : `gstack-${entry.name}`;
|
||
skills.push({ dir: entry.name, codexName });
|
||
}
|
||
return skills;
|
||
})();
|
||
|
||
test('--host codex generates correct output paths', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const skillMd = path.join(AGENTS_DIR, skill.codexName, 'SKILL.md');
|
||
expect(fs.existsSync(skillMd)).toBe(true);
|
||
}
|
||
});
|
||
|
||
test('root gstack bundle has OpenAI metadata for Codex skill browsing', () => {
|
||
const rootMetadata = path.join(ROOT, 'agents', 'openai.yaml');
|
||
expect(fs.existsSync(rootMetadata)).toBe(true);
|
||
const content = fs.readFileSync(rootMetadata, 'utf-8');
|
||
expect(content).toContain('display_name: "gstack"');
|
||
expect(content).toContain('allow_implicit_invocation: true');
|
||
});
|
||
|
||
test('externalSkillName mapping: root is gstack, others are gstack-{dir}', () => {
|
||
// Root → gstack
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack', 'SKILL.md'))).toBe(true);
|
||
// Subdirectories → gstack-{dir}
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'))).toBe(true);
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'))).toBe(true);
|
||
// gstack-upgrade doesn't double-prefix
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-upgrade', 'SKILL.md'))).toBe(true);
|
||
// No double-prefix: gstack-gstack-upgrade must NOT exist
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-gstack-upgrade', 'SKILL.md'))).toBe(false);
|
||
});
|
||
|
||
test('Codex frontmatter has ONLY name + description', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
expect(content.startsWith('---\n')).toBe(true);
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
expect(fmEnd).toBeGreaterThan(0);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
// Must have name and description
|
||
expect(frontmatter).toContain('name:');
|
||
expect(frontmatter).toContain('description:');
|
||
// Must NOT have allowed-tools, version, or hooks
|
||
expect(frontmatter).not.toContain('allowed-tools:');
|
||
expect(frontmatter).not.toContain('version:');
|
||
expect(frontmatter).not.toContain('hooks:');
|
||
}
|
||
});
|
||
|
||
test('all Codex skills have agents/openai.yaml metadata', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const metadata = path.join(AGENTS_DIR, skill.codexName, 'agents', 'openai.yaml');
|
||
expect(fs.existsSync(metadata)).toBe(true);
|
||
const content = fs.readFileSync(metadata, 'utf-8');
|
||
expect(content).toContain(`display_name: "${skill.codexName}"`);
|
||
expect(content).toContain('short_description:');
|
||
expect(content).toContain('allow_implicit_invocation: true');
|
||
}
|
||
});
|
||
|
||
test('no .claude/skills/ in Codex output', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('.claude/skills');
|
||
}
|
||
});
|
||
|
||
test('no ~/.claude/ runtime paths in Codex output', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
// Outside prompts may explicitly forbid reading Claude's skill directory.
|
||
// Every executable/runtime path must still use the selected host root.
|
||
const withoutBoundary = content.replace(/^.*do not read or execute any files under[^\n]*$/gim, '');
|
||
expect(withoutBoundary).not.toContain('~/.claude/');
|
||
}
|
||
});
|
||
|
||
test('/codex skill excluded from Codex output', () => {
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-codex', 'SKILL.md'))).toBe(false);
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-codex'))).toBe(false);
|
||
});
|
||
|
||
test('Codex output includes the renamed Claude Code skill and restricted runner', () => {
|
||
const content = readExternalSkillUnion(EXTERNAL_OUT, '.agents', 'claude-code');
|
||
expect(content).toContain('name: claude-code');
|
||
expect(content).toContain('gstack-claude-code');
|
||
expect(content).toContain('--access none');
|
||
expect(content).toContain('--access read-only');
|
||
expect(content).toContain('--timeout-ms');
|
||
expect(content).toContain('claude-session-id');
|
||
expect(content).not.toContain('$HOME/.claude/.credentials.json');
|
||
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-claude'))).toBe(false);
|
||
});
|
||
|
||
test('Codex ship and review preserve outside review through Claude Code', () => {
|
||
for (const skill of ['ship', 'review']) {
|
||
const content = readExternalSkillUnion(EXTERNAL_OUT, '.agents', skill);
|
||
expect(content).not.toMatch(/codex\s+(?:exec|review)\s/);
|
||
expect(content).toContain('gstack-claude-code');
|
||
expect(content).toContain('codex_reviews');
|
||
expect(content).toContain('adversarial-review');
|
||
}
|
||
});
|
||
|
||
test('--host codex --dry-run freshness', () => {
|
||
// Dry-run against the out-dir render: determinism/idempotency check
|
||
// (regenerating produces the same bytes the module-level render did).
|
||
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
// Every Codex skill should be FRESH
|
||
for (const skill of CODEX_SKILLS) {
|
||
expect(output).toContain(`FRESH: .agents/skills/${skill.codexName}/SKILL.md`);
|
||
}
|
||
expect(output).not.toContain('STALE');
|
||
});
|
||
|
||
test('--host agents alias produces same output as --host codex', () => {
|
||
const codexResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
const agentsResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'agents', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
expect(codexResult.exitCode).toBe(0);
|
||
expect(agentsResult.exitCode).toBe(0);
|
||
// Both should produce the same output (same FRESH lines)
|
||
expect(codexResult.stdout.toString()).toBe(agentsResult.stdout.toString());
|
||
});
|
||
|
||
test('multiline descriptions preserved in Codex output', () => {
|
||
// office-hours has a multiline description — verify it survives the frontmatter transform
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-office-hours', 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
// Description should span multiple lines (block scalar)
|
||
const descLines = frontmatter.split('\n').filter(l => l.startsWith(' '));
|
||
expect(descLines.length).toBeGreaterThan(1);
|
||
// Verify key phrases survived
|
||
expect(frontmatter).toContain('YC Office Hours');
|
||
});
|
||
|
||
test('hook skills have safety prose and no hooks: in frontmatter', () => {
|
||
const HOOK_SKILLS = ['gstack-careful', 'gstack-freeze', 'gstack-guard'];
|
||
for (const skillName of HOOK_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skillName, 'SKILL.md'), 'utf-8');
|
||
// Must have safety advisory prose
|
||
expect(content).toContain('Safety Advisory');
|
||
// Must NOT have hooks: in frontmatter
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
expect(frontmatter).not.toContain('hooks:');
|
||
}
|
||
});
|
||
|
||
test('all Codex SKILL.md files have auto-generated header', () => {
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('AUTO-GENERATED from SKILL.md.tmpl');
|
||
expect(content).toContain('Regenerate: bun run gen:skill-docs');
|
||
}
|
||
});
|
||
|
||
test('Codex preamble resolves runtime assets from repo-local or global gstack roots', () => {
|
||
// Check a skill that has a preamble (review is a good candidate)
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('GSTACK_ROOT');
|
||
// C1: one shared root resolution — repo-local first, then CODEX_HOME's global root.
|
||
expect(content).toContain('_r=$(git rev-parse --show-toplevel 2>/dev/null)/.agents/skills/gstack');
|
||
expect(content).toContain('_r=${CODEX_HOME:-~/.codex}/skills/gstack');
|
||
// Phase 1/2: config reads moved into gstack-skill-start — the fence itself
|
||
// is the bin asset the preamble must resolve through $GSTACK_BIN, and the
|
||
// question-preference runtime call still resolves the same way.
|
||
expect(content).toContain('$GSTACK_BIN/gstack-skill-start');
|
||
expect(content).toContain('$GSTACK_BIN/gstack-question-preference');
|
||
// The upgrade-skill doc reference moved into the script's upgrade-flow
|
||
// block, resolved $0-relative ($_ROOT_DIR) — host-neutral by construction,
|
||
// so the Codex render no longer needs its own copy.
|
||
expect(SKILL_START_SCRIPT).toContain('$_ROOT_DIR/gstack-upgrade/SKILL.md');
|
||
expect(SKILL_START_SCRIPT).toContain('_ROOT_DIR=$(dirname "$_BIN")');
|
||
expect(content).not.toContain('~/.codex/skills/gstack/bin/gstack-config get telemetry');
|
||
});
|
||
|
||
// ─── Path rewriting regression tests ─────────────────────────
|
||
|
||
test('sidecar paths resolve through $GSTACK_ROOT (not gstack-review/)', () => {
|
||
// #2518: templates now anchor sidecars at the installed skill root
|
||
// (~/.claude/skills/gstack/review/...), which the codex path rewrite turns
|
||
// into $GSTACK_ROOT/review/... — resolved by the preamble against the
|
||
// repo-local .agents root or the global install. The old repo-relative
|
||
// form (.claude/skills/review/) only resolved inside gstack's own checkout.
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('$GSTACK_ROOT/review/checklist.md');
|
||
// design-checklist.md is now referenced via Review Army specialist (Claude only, stripped for Codex)
|
||
// Wrong: must NOT reference gstack-review/checklist.md (file doesn't exist there)
|
||
expect(content).not.toContain('.agents/skills/gstack-review/checklist.md');
|
||
});
|
||
|
||
test('sidecar paths in ship skill point to gstack/review/ for pre-landing review', () => {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
// Ship references the review checklist in its pre-landing review step
|
||
if (content.includes('checklist.md')) {
|
||
expect(content).toContain('.agents/skills/gstack/review/');
|
||
expect(content).not.toContain('.agents/skills/gstack-review/checklist');
|
||
}
|
||
});
|
||
|
||
test('greptile-triage sidecar path is correct', () => {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
if (content.includes('greptile-triage')) {
|
||
expect(content).toContain('$GSTACK_ROOT/review/greptile-triage.md');
|
||
expect(content).not.toContain('.agents/skills/gstack-review/greptile-triage');
|
||
}
|
||
});
|
||
|
||
test('all four path rewrite rules produce correct output', () => {
|
||
// Test each of the 4 path rewrite rules individually
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
|
||
// Rule 1: ~/.claude/skills/gstack → $GSTACK_ROOT
|
||
expect(content).not.toContain('~/.claude/skills/gstack');
|
||
expect(content).toContain('$GSTACK_ROOT');
|
||
|
||
// Rule 2: .claude/skills/gstack → .agents/skills/gstack
|
||
expect(content).not.toContain('.claude/skills/gstack');
|
||
|
||
// Rule 3: .claude/skills/review → .agents/skills/gstack/review
|
||
expect(content).not.toContain('.claude/skills/review');
|
||
|
||
// Rule 4: .claude/skills → .agents/skills (catch-all)
|
||
expect(content).not.toContain('.claude/skills');
|
||
});
|
||
|
||
test('path rewrite rules apply to all Codex skills with sidecar references', () => {
|
||
// Verify across ALL generated skills, not just review
|
||
for (const skill of CODEX_SKILLS) {
|
||
const content = fs.readFileSync(path.join(AGENTS_DIR, skill.codexName, 'SKILL.md'), 'utf-8');
|
||
// No skill should reference Claude paths
|
||
expect(content).not.toContain('~/.claude/skills');
|
||
expect(content).not.toContain('.claude/skills');
|
||
if (content.includes('gstack-config') || content.includes('gstack-update-check') || content.includes('gstack-telemetry-log')) {
|
||
expect(content).toContain('$GSTACK_ROOT');
|
||
}
|
||
// If a skill references checklist.md, it must use the correct sidecar path
|
||
if (content.includes('checklist.md') && !content.includes('design-checklist.md')) {
|
||
expect(content).not.toContain('gstack-review/checklist.md');
|
||
}
|
||
}
|
||
});
|
||
|
||
// ─── Claude output regression guard ─────────────────────────
|
||
|
||
test('Claude output uses installed-root review paths (#2518)', () => {
|
||
// Codex changes must NOT affect Claude output; the Claude form is the
|
||
// installed-root anchor, not the old repo-relative path that only
|
||
// resolved inside gstack's own checkout.
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('~/.claude/skills/gstack/review/checklist.md');
|
||
expect(content).toContain('~/.claude/skills/gstack');
|
||
// Must NOT contain Codex HOST paths. `~/.codex/sessions/` is exempt: the
|
||
// timeout-wrapper guidance documents the Codex CLI's own rollout-log
|
||
// location (a user-facing CLI path, same class as ~/.codex/logs/ in the
|
||
// codex skill), not the gstack Codex host install path.
|
||
// `~/.codex/config.toml` is the same user-facing class: the shared
|
||
// codexPreflight's model_unusable branch (#2477) points at the CLI's own
|
||
// config file, where the rejected `model =` pin lives.
|
||
expect(content).not.toContain('.agents/skills');
|
||
expect(
|
||
content
|
||
.replaceAll('~/.codex/sessions/', '')
|
||
.replaceAll('~/.codex/config.toml', ''),
|
||
).not.toContain('~/.codex/');
|
||
});
|
||
|
||
test('Claude output unchanged: ship skill still uses .claude/skills/ paths', () => {
|
||
const content = readShipUnion();
|
||
expect(content).toContain('~/.claude/skills/gstack');
|
||
expect(content).not.toContain('.agents/skills');
|
||
// ~/.codex/sessions/ is the Codex CLI's rollout-log path (user-facing),
|
||
// documented by the adversarial-pass timeout guidance; ~/.codex/config.toml
|
||
// is the CLI's own config file (model_unusable guidance, #2477) — see the
|
||
// review test above.
|
||
expect(
|
||
content
|
||
.replaceAll('~/.codex/sessions/', '')
|
||
.replaceAll('~/.codex/config.toml', ''),
|
||
).not.toContain('~/.codex/');
|
||
});
|
||
|
||
test('Claude output unchanged: all Claude skills have zero Codex paths', () => {
|
||
for (const skill of CLAUDE_GENERATED_SKILLS) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill.dir, 'SKILL.md'), 'utf-8');
|
||
// pair-agent legitimately documents how Codex agents store credentials.
|
||
// codex + autoplan document the Codex CLI auth file (~/.codex/auth.json)
|
||
// and log path (~/.codex/logs/) — those are user-facing Codex CLI paths,
|
||
// not the gstack Codex host install path. ~/.codex/sessions/ (rollout
|
||
// logs, referenced by the review/ship timeout guidance) and
|
||
// ~/.codex/config.toml (the model_unusable guidance in the shared
|
||
// codexPreflight, #2477) are the same user-facing class, so they are
|
||
// scrubbed before the ban.
|
||
if (skill.dir !== 'pair-agent' && skill.dir !== 'codex' && skill.dir !== 'autoplan') {
|
||
expect(
|
||
content
|
||
.replaceAll('~/.codex/sessions/', '')
|
||
.replaceAll('~/.codex/config.toml', ''),
|
||
).not.toContain('~/.codex/');
|
||
}
|
||
// gstack-upgrade legitimately references .agents/skills for cross-platform detection
|
||
if (skill.dir !== 'gstack-upgrade') {
|
||
expect(content).not.toContain('.agents/skills');
|
||
}
|
||
}
|
||
});
|
||
|
||
// ─── Design outside voices: Codex host guard ─────────────────
|
||
|
||
test('codex host retains design outside voices through Claude Code', () => {
|
||
const codexContent = readExternalSkillUnion(EXTERNAL_OUT, '.agents', 'design-review');
|
||
expect(codexContent).toContain('Design Outside Voices');
|
||
expect(codexContent).toContain('gstack-claude-code');
|
||
});
|
||
|
||
test('codex host does not include Codex design block in ship', () => {
|
||
const codexContent = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
expect(codexContent).not.toContain('Codex design voice');
|
||
});
|
||
|
||
// ─── Explicit --model override wins over the host default ────
|
||
// Without --model the codex host renders its defaultModel (gpt) — pinned by
|
||
// the golden test. This pins the OTHER direction through the real CLI:
|
||
// `./setup --host codex --model <id>` depends on it. The override renders
|
||
// into its OWN out-dir, so no restore pass is needed — the host-default
|
||
// render (EXTERNAL_OUT) is untouched and asserted directly.
|
||
test('explicit --model overrides the codex host default', () => {
|
||
const overrideOut = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-model-override-'));
|
||
try {
|
||
const override = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--model', 'claude', '--out-dir', overrideOut], {
|
||
cwd: ROOT,
|
||
stdout: 'pipe',
|
||
stderr: 'pipe',
|
||
timeout: 120_000,
|
||
});
|
||
expect(override.exitCode).toBe(0);
|
||
const content = fs.readFileSync(path.join(overrideOut, '.agents', 'skills', 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Model-Specific Behavioral Patch (claude)');
|
||
// The overlay now travels as --model into gstack-skill-start, which
|
||
// echoes MODEL_OVERLAY at runtime.
|
||
expect(content).toContain('--model "claude"');
|
||
} finally {
|
||
fs.rmSync(overrideOut, { recursive: true, force: true });
|
||
}
|
||
// Host-default direction: the untouched EXTERNAL_OUT render carries gpt.
|
||
const hostDefault = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
expect(hostDefault).toContain('Model-Specific Behavioral Patch (gpt)');
|
||
expect(hostDefault).toContain('--model "gpt"');
|
||
});
|
||
});
|
||
|
||
// ─── Factory generation tests ────────────────────────────────
|
||
|
||
describe('Factory generation (--host factory)', () => {
|
||
// .factory/ is gitignored — read the module-level out-dir render
|
||
// (--host all covers factory) instead of regenerating in place.
|
||
const FACTORY_DIR = path.join(EXTERNAL_OUT, '.factory', 'skills');
|
||
|
||
// Fresh out-dir → the vendored-dev-mode symlink loop can never occur, so
|
||
// every template renders (see the Codex discovery note above).
|
||
const FACTORY_SKILLS = (() => {
|
||
const skills: Array<{ dir: string; factoryName: string }> = [];
|
||
if (fs.existsSync(path.join(ROOT, 'SKILL.md.tmpl'))) {
|
||
skills.push({ dir: '.', factoryName: 'gstack' });
|
||
}
|
||
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
||
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||
if (entry.name === 'codex') continue;
|
||
if (!fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) continue;
|
||
const factoryName = entry.name.startsWith('gstack-') ? entry.name : `gstack-${entry.name}`;
|
||
skills.push({ dir: entry.name, factoryName });
|
||
}
|
||
return skills;
|
||
})();
|
||
|
||
test('--host factory generates correct output paths', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const skillMd = path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md');
|
||
expect(fs.existsSync(skillMd)).toBe(true);
|
||
}
|
||
});
|
||
|
||
test('Factory frontmatter has name + description + user-invocable', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
expect(frontmatter).toContain('name:');
|
||
expect(frontmatter).toContain('description:');
|
||
expect(frontmatter).toContain('user-invocable: true');
|
||
expect(frontmatter).not.toContain('allowed-tools:');
|
||
expect(frontmatter).not.toContain('preamble-tier:');
|
||
expect(frontmatter).not.toContain('sensitive:');
|
||
}
|
||
});
|
||
|
||
test('sensitive skills have disable-model-invocation', () => {
|
||
const SENSITIVE = ['gstack-ship', 'gstack-land-and-deploy', 'gstack-guard', 'gstack-careful', 'gstack-freeze', 'gstack-unfreeze'];
|
||
for (const name of SENSITIVE) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, name, 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
expect(frontmatter).toContain('disable-model-invocation: true');
|
||
}
|
||
});
|
||
|
||
test('non-sensitive skills lack disable-model-invocation', () => {
|
||
const NON_SENSITIVE = ['gstack-qa', 'gstack-review', 'gstack-investigate', 'gstack-browse'];
|
||
for (const name of NON_SENSITIVE) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, name, 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(4, fmEnd);
|
||
expect(frontmatter).not.toContain('disable-model-invocation');
|
||
}
|
||
});
|
||
|
||
test('no .claude/skills/ in Factory output', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('.claude/skills');
|
||
}
|
||
});
|
||
|
||
test('no ~/.claude/skills/ paths in Factory output', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, skill.factoryName, 'SKILL.md'), 'utf-8');
|
||
// ~/.claude/skills should be rewritten, but ~/.claude/plans is legitimate
|
||
// (plan directory lookup) and ~/.claude/ in codex prompts is intentional
|
||
expect(content).not.toContain('~/.claude/skills');
|
||
}
|
||
});
|
||
|
||
test('Factory installs both outside-review wrapper skills', () => {
|
||
for (const skill of ['gstack-codex', 'gstack-claude-code']) {
|
||
expect(fs.existsSync(path.join(FACTORY_DIR, skill, 'SKILL.md'))).toBe(true);
|
||
}
|
||
});
|
||
|
||
test('Factory keeps Codex integration blocks', () => {
|
||
// Factory users CAN use Codex second opinions (codex exec is a standalone binary)
|
||
const shipContent = fs.readFileSync(path.join(FACTORY_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
|
||
expect(shipContent).toContain('codex');
|
||
});
|
||
|
||
test('no agents/openai.yaml in Factory output', () => {
|
||
for (const skill of FACTORY_SKILLS) {
|
||
const yamlPath = path.join(FACTORY_DIR, skill.factoryName, 'agents', 'openai.yaml');
|
||
expect(fs.existsSync(yamlPath)).toBe(false);
|
||
}
|
||
});
|
||
|
||
test('--host droid alias works', () => {
|
||
const factoryResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
|
||
});
|
||
const droidResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'droid', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
|
||
});
|
||
expect(factoryResult.exitCode).toBe(0);
|
||
expect(droidResult.exitCode).toBe(0);
|
||
expect(factoryResult.stdout.toString()).toBe(droidResult.stdout.toString());
|
||
});
|
||
|
||
test('--host factory --dry-run freshness', () => {
|
||
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
|
||
});
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
for (const skill of FACTORY_SKILLS) {
|
||
expect(output).toContain(`FRESH: .factory/skills/${skill.factoryName}/SKILL.md`);
|
||
}
|
||
expect(output).not.toContain('STALE');
|
||
});
|
||
|
||
test('Factory preamble uses .factory paths', () => {
|
||
const content = fs.readFileSync(path.join(FACTORY_DIR, 'gstack-review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('GSTACK_ROOT');
|
||
expect(content).toContain('_r=$(git rev-parse --show-toplevel 2>/dev/null)/.factory/skills/gstack');
|
||
expect(content).toContain('$GSTACK_BIN/gstack-config');
|
||
});
|
||
});
|
||
|
||
// ─── Parameterized host smoke tests (config-driven) ─────────
|
||
|
||
import { ALL_HOST_CONFIGS, getExternalHosts } from '../hosts/index';
|
||
import { sliceBetween } from './helpers/skill-fixture';
|
||
|
||
describe('Parameterized host smoke tests', () => {
|
||
// Every external host was rendered up front by the module-level
|
||
// `--host all --out-dir EXTERNAL_OUT` render, so the per-host `--dry-run`
|
||
// freshness checks are deterministic: they compare a regeneration against
|
||
// that render — an idempotency/determinism check that catches
|
||
// non-deterministic gen without ever writing (or depending on) the live
|
||
// gitignored host dirs. The tracked-claude freshness test
|
||
// (`generated files are fresh`) runs earlier and is unaffected.
|
||
for (const hostConfig of getExternalHosts()) {
|
||
describe(`${hostConfig.displayName} (--host ${hostConfig.name})`, () => {
|
||
const hostDir = path.join(EXTERNAL_OUT, hostConfig.hostSubdir, 'skills');
|
||
|
||
test('generates output that exists on disk', () => {
|
||
// The module-level --host all render must have produced this host's tree.
|
||
expect(fs.existsSync(hostDir)).toBe(true);
|
||
const skills = fs.readdirSync(hostDir).filter(d =>
|
||
fs.existsSync(path.join(hostDir, d, 'SKILL.md'))
|
||
);
|
||
expect(skills.length).toBeGreaterThan(0);
|
||
});
|
||
|
||
test('no .claude/skills path leakage outside repo-root sidecar symlinks', () => {
|
||
expect(fs.existsSync(hostDir)).toBe(true);
|
||
const skills = fs.readdirSync(hostDir);
|
||
for (const skill of skills) {
|
||
// Dev installs may mount the repo root at host/skills/gstack as a runtime
|
||
// sidecar. The generator skips that symlink loop, so leakage checks should too.
|
||
if (isRepoRootSymlink(path.join(hostDir, skill))) continue;
|
||
const skillMd = path.join(hostDir, skill, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
const leaks = externalHostPathLeaks(content);
|
||
if (leaks.length > 0) {
|
||
throw new Error(`${skill}: .claude/skills leakage:\n${leaks.slice(0, 3).join('\n')}`);
|
||
}
|
||
}
|
||
});
|
||
|
||
test('frontmatter has name and description', () => {
|
||
expect(fs.existsSync(hostDir)).toBe(true);
|
||
const skills = fs.readdirSync(hostDir);
|
||
for (const skill of skills) {
|
||
const skillMd = path.join(hostDir, skill, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
expect(content).toMatch(/^---\n/);
|
||
expect(content).toMatch(/^name:\s/m);
|
||
expect(content).toMatch(/^description:\s/m);
|
||
}
|
||
});
|
||
|
||
test('--dry-run freshness check passes', () => {
|
||
const result = Bun.spawnSync(
|
||
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', hostConfig.name, '--dry-run', '--out-dir', EXTERNAL_OUT],
|
||
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 }
|
||
);
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
expect(output).not.toContain('STALE');
|
||
});
|
||
|
||
if (hostConfig.generation.skipSkills?.includes('codex')) {
|
||
test('/codex skill excluded', () => {
|
||
expect(fs.existsSync(path.join(hostDir, 'gstack-codex', 'SKILL.md'))).toBe(false);
|
||
});
|
||
}
|
||
});
|
||
}
|
||
});
|
||
|
||
// ─── --host all tests ────────────────────────────────────────
|
||
|
||
describe('--host all', () => {
|
||
// Same determinism guard as the parameterized block: the module-level
|
||
// `--host all --out-dir EXTERNAL_OUT` render is the comparison baseline, so
|
||
// this dry-run reports FRESH regardless of live-tree state — and proves the
|
||
// claude host plus every external host regenerate deterministically.
|
||
test('--host all generates for all registered hosts', () => {
|
||
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--dry-run', '--out-dir', EXTERNAL_OUT], {
|
||
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
|
||
});
|
||
expect(result.exitCode).toBe(0);
|
||
const output = result.stdout.toString();
|
||
// All hosts should appear in output
|
||
expect(output).toContain('FRESH: SKILL.md'); // claude
|
||
for (const hostConfig of getExternalHosts()) {
|
||
expect(output).toContain(`FRESH: ${hostConfig.hostSubdir}/skills/`);
|
||
}
|
||
});
|
||
});
|
||
|
||
// ─── Setup script validation ─────────────────────────────────
|
||
// These tests verify the setup script's install layout matches
|
||
// what the generator produces — catching the bug where setup
|
||
// installed Claude-format source dirs for Codex users.
|
||
|
||
describe('setup script validation', () => {
|
||
const setupContent = fs.readFileSync(path.join(ROOT, 'setup'), 'utf-8');
|
||
|
||
test('setup has separate link functions for Claude and Codex', () => {
|
||
expect(setupContent).toContain('link_claude_skill_dirs');
|
||
expect(setupContent).toContain('link_codex_skill_dirs');
|
||
// Old unified function must not exist
|
||
expect(setupContent).not.toMatch(/^link_skill_dirs\(\)/m);
|
||
});
|
||
|
||
test('Claude install uses link_claude_skill_dirs', () => {
|
||
// The Claude install section (section 4) should use the Claude function
|
||
const claudeSection = setupContent.slice(
|
||
setupContent.indexOf('# 4. Install for Claude'),
|
||
setupContent.indexOf('# 5. Install for Codex')
|
||
);
|
||
expect(claudeSection).toContain('link_claude_skill_dirs');
|
||
expect(claudeSection).not.toContain('link_codex_skill_dirs');
|
||
});
|
||
|
||
test('Codex install uses link_codex_skill_dirs', () => {
|
||
// The Codex install section (section 5) should use the Codex function
|
||
// End marker: the next numbered section header (a marker that doesn't
|
||
// exist slices to EOF and the assertion reads unrelated sections).
|
||
const codexSection = setupContent.slice(
|
||
setupContent.indexOf('# 5. Install for Codex'),
|
||
setupContent.indexOf('# 6. Install for Kiro')
|
||
);
|
||
expect(setupContent.indexOf('# 6. Install for Kiro')).toBeGreaterThan(-1);
|
||
expect(codexSection).toContain('create_codex_runtime_root');
|
||
expect(codexSection).toContain('link_codex_skill_dirs');
|
||
expect(codexSection).not.toContain('link_claude_skill_dirs');
|
||
expect(codexSection).not.toContain('_link_or_copy "$GSTACK_DIR" "$CODEX_GSTACK"');
|
||
});
|
||
|
||
test('Codex install prefers repo-local .agents/skills when setup runs from there', () => {
|
||
expect(setupContent).toContain('SKILLS_PARENT_BASENAME');
|
||
expect(setupContent).toContain('CODEX_REPO_LOCAL=0');
|
||
expect(setupContent).toContain('[ "$SKILLS_PARENT_BASENAME" = ".agents" ]');
|
||
expect(setupContent).toContain('CODEX_REPO_LOCAL=1');
|
||
expect(setupContent).toContain('CODEX_SKILLS="$INSTALL_SKILLS_DIR"');
|
||
});
|
||
|
||
test('setup separates install path from source path for symlinked repo-local installs', () => {
|
||
expect(setupContent).toContain('INSTALL_GSTACK_DIR=');
|
||
expect(setupContent).toContain('SOURCE_GSTACK_DIR=');
|
||
expect(setupContent).toContain('INSTALL_SKILLS_DIR=');
|
||
expect(setupContent).toContain('CODEX_GSTACK="$INSTALL_GSTACK_DIR"');
|
||
// Links come from the source checkout, or its per-install render (#1882).
|
||
expect(setupContent).toContain('link_codex_skill_dirs "${_CODEX_RENDER_ROOT:-$SOURCE_GSTACK_DIR}" "$CODEX_SKILLS"');
|
||
});
|
||
|
||
test('Codex installs always create sidecar runtime assets for the real skill target', () => {
|
||
expect(setupContent).toContain('if [ "$INSTALL_CODEX" -eq 1 ]; then');
|
||
expect(setupContent).toContain('create_agents_sidecar "$SOURCE_GSTACK_DIR"');
|
||
});
|
||
|
||
test('link_codex_skill_dirs reads from .agents/skills/', () => {
|
||
// The Codex link function must reference .agents/skills for generated Codex skills
|
||
const fnStart = setupContent.indexOf('link_codex_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('.agents/skills');
|
||
expect(fnBody).toContain('gstack*');
|
||
});
|
||
|
||
test('link_claude_skill_dirs creates real directories with absolute SKILL.md symlinks', () => {
|
||
// Claude links should be real directories with absolute SKILL.md symlinks
|
||
// to ensure Claude Code discovers them as top-level skills (not nested under gstack/)
|
||
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('mkdir -p "$target"');
|
||
// v1.36.0.0: routes through _link_or_copy helper for Windows fallback (cp on MSYS2/Git Bash).
|
||
// v1.67 (#2569): the source is render-aware — canonical SKILL.md, or the
|
||
// rendered :user variant from ${GSTACK_HOME}/render/claude when present.
|
||
expect(fnBody).toContain('_skill_md_src="$gstack_dir/$dir_name/SKILL.md"');
|
||
expect(fnBody).toContain('_link_or_copy "$_skill_md_src" "$target/SKILL.md"');
|
||
});
|
||
|
||
// REGRESSION: cleanup functions must handle both old symlinks AND new real-directory pattern
|
||
test('cleanup functions handle real directories with symlinked SKILL.md', () => {
|
||
// cleanup_old_claude_symlinks must detect and remove real dirs with SKILL.md symlinks
|
||
const cleanupOldStart = setupContent.indexOf('cleanup_old_claude_symlinks()');
|
||
const cleanupOldEnd = setupContent.indexOf('}', setupContent.indexOf('cleaned up old', cleanupOldStart));
|
||
const cleanupOldBody = setupContent.slice(cleanupOldStart, cleanupOldEnd);
|
||
expect(cleanupOldBody).toContain('-d "$old_target"');
|
||
expect(cleanupOldBody).toContain('-L "$old_target/SKILL.md"');
|
||
expect(cleanupOldBody).toContain('rm -rf "$old_target"');
|
||
|
||
// cleanup_prefixed_claude_symlinks must also handle the new pattern
|
||
const cleanupPrefixedStart = setupContent.indexOf('cleanup_prefixed_claude_symlinks()');
|
||
const cleanupPrefixedEnd = setupContent.indexOf('}', setupContent.indexOf('cleaned up prefixed', cleanupPrefixedStart));
|
||
const cleanupPrefixedBody = setupContent.slice(cleanupPrefixedStart, cleanupPrefixedEnd);
|
||
expect(cleanupPrefixedBody).toContain('-d "$prefixed_target"');
|
||
expect(cleanupPrefixedBody).toContain('-L "$prefixed_target/SKILL.md"');
|
||
expect(cleanupPrefixedBody).toContain('rm -rf "$prefixed_target"');
|
||
});
|
||
|
||
// REGRESSION: link function must upgrade old directory symlinks
|
||
test('link_claude_skill_dirs removes old directory symlinks before creating real dirs', () => {
|
||
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
// Must check for and remove old symlinks before mkdir
|
||
expect(fnBody).toContain('if [ -L "$target" ]');
|
||
expect(fnBody).toContain('rm -f "$target"');
|
||
});
|
||
|
||
test('setup links root gstack skill through a thin Claude wrapper alias', () => {
|
||
const fnStart = setupContent.indexOf('link_claude_root_skill_alias()');
|
||
const fnEnd = setupContent.indexOf('# ─── Helper: remove old unprefixed Claude skill entries', fnStart);
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('_gstack-command');
|
||
// #2511: the alias must be a rewritten COPY (unique frontmatter name),
|
||
// never a verbatim symlink of the canonical SKILL.md.
|
||
expect(fnBody).toContain('_install_alias_skill_md "$gstack_dir/SKILL.md" "$target" "_gstack-command"');
|
||
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir/SKILL.md"');
|
||
|
||
const claudeSection = setupContent.slice(
|
||
setupContent.indexOf('# 4. Install for Claude'),
|
||
setupContent.indexOf('# 5. Install for Codex')
|
||
);
|
||
expect(claudeSection).toContain('link_claude_root_skill_alias "$SOURCE_GSTACK_DIR" "$INSTALL_SKILLS_DIR"');
|
||
});
|
||
|
||
test('setup supports --host auto|claude|codex|kiro|opencode|cursor; slate is informational', () => {
|
||
expect(setupContent).toContain('--host');
|
||
// #2361: slate moved OUT of the install accept-list (it was accepted but
|
||
// never dispatched — a silent exit-0 no-op) into an informational arm.
|
||
expect(setupContent).toContain('claude|codex|kiro|factory|opencode|cursor|copilot|auto');
|
||
expect(setupContent).toMatch(/^ {2}slate\)/m);
|
||
});
|
||
|
||
test('auto mode detects claude, codex, kiro, and opencode binaries', () => {
|
||
expect(setupContent).toContain('command -v claude');
|
||
expect(setupContent).toContain('command -v codex');
|
||
expect(setupContent).toContain('command -v kiro-cli');
|
||
expect(setupContent).toContain('command -v opencode');
|
||
});
|
||
|
||
// T1: Sidecar skip guard — prevents .agents/skills/gstack from being linked as a skill
|
||
test('link_codex_skill_dirs skips the gstack sidecar directory', () => {
|
||
const fnStart = setupContent.indexOf('link_codex_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('[ "$skill_name" = "gstack" ] && continue');
|
||
});
|
||
|
||
// T2: Dynamic $GSTACK_ROOT paths in generated Codex preambles
|
||
test('generated Codex preambles use dynamic GSTACK_ROOT paths', () => {
|
||
// Read the module-level out-dir render (always present).
|
||
const codexSkillDir = path.join(EXTERNAL_OUT, '.agents', 'skills', 'gstack-ship');
|
||
const content = fs.readFileSync(path.join(codexSkillDir, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('GSTACK_ROOT=');
|
||
expect(content).toContain('$GSTACK_BIN/');
|
||
});
|
||
|
||
test('setup installs Kiro-native output and runtime assets', () => {
|
||
expect(setupContent).toContain('INSTALL_KIRO=');
|
||
expect(setupContent).toContain('kiro-cli');
|
||
expect(setupContent).toContain('KIRO_SKILLS=');
|
||
expect(setupContent).toContain('KIRO_SKILLS="$HOME/.kiro/skills"');
|
||
expect(setupContent).toContain('KIRO_DIR="$SOURCE_GSTACK_DIR/.kiro/skills"');
|
||
expect(setupContent).toContain('gen:skill-docs --host kiro');
|
||
expect(setupContent).toContain('$KIRO_GSTACK/lib');
|
||
});
|
||
|
||
test('setup supports --host opencode with install section and OpenCode skill path vars', () => {
|
||
expect(setupContent).toContain('INSTALL_OPENCODE=');
|
||
expect(setupContent).toContain('OPENCODE_SKILLS="$HOME/.config/opencode/skills"');
|
||
expect(setupContent).toContain('OPENCODE_GSTACK="$OPENCODE_SKILLS/gstack"');
|
||
});
|
||
|
||
// --host cursor full install slice (#1358, PR #2547 by @szsunyuan re-derived)
|
||
test('auto mode detects Cursor via binary or ~/.cursor directory', () => {
|
||
expect(setupContent).toContain('command -v cursor');
|
||
expect(setupContent).toContain('[ -d "$HOME/.cursor" ] && INSTALL_CURSOR=1');
|
||
});
|
||
|
||
test('setup supports --host cursor with install section and Cursor skill path vars', () => {
|
||
expect(setupContent).toContain('INSTALL_CURSOR=');
|
||
expect(setupContent).toContain('CURSOR_SKILLS="$HOME/.cursor/skills"');
|
||
expect(setupContent).toContain('CURSOR_GSTACK="$CURSOR_SKILLS/gstack"');
|
||
expect(setupContent).toContain('create_cursor_runtime_root');
|
||
expect(setupContent).toContain('create_cursor_sidecar');
|
||
expect(setupContent).toContain('link_cursor_skill_dirs');
|
||
expect(setupContent).toContain('gstack ready (cursor).');
|
||
});
|
||
|
||
test('create_cursor_runtime_root exposes only Cursor runtime assets', () => {
|
||
const fnStart = setupContent.indexOf('create_cursor_runtime_root()');
|
||
const fnEnd = setupContent.indexOf('create_cursor_sidecar()', fnStart);
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('gstack/SKILL.md');
|
||
expect(fnBody).toContain('browse/dist');
|
||
expect(fnBody).toContain('browse/bin');
|
||
expect(fnBody).toContain('gstack-upgrade/SKILL.md');
|
||
expect(fnBody).toContain('checklist.md');
|
||
expect(fnBody).toContain('TODOS-format.md');
|
||
// bin scripts import ../lib — the two must travel together.
|
||
expect(fnBody).toContain('$cursor_gstack/lib');
|
||
expect(fnBody).not.toContain('design-checklist.md');
|
||
expect(fnBody).not.toContain('greptile-triage.md');
|
||
expect(fnBody).not.toContain('review/specialists');
|
||
expect(fnBody).not.toContain('qa/templates');
|
||
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir" "$cursor_gstack"');
|
||
});
|
||
|
||
test('create_cursor_sidecar plants runtime assets without wiping generated SKILL.md', () => {
|
||
const fnStart = setupContent.indexOf('create_cursor_sidecar()');
|
||
const fnEnd = setupContent.indexOf('link_cursor_skill_dirs()', fnStart);
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('.cursor/skills/gstack');
|
||
expect(fnBody).toContain('bin');
|
||
expect(fnBody).toContain('browse/dist');
|
||
expect(fnBody).toContain('browse/bin');
|
||
expect(fnBody).toContain('ETHOS.md');
|
||
expect(fnBody).not.toContain('rm -rf');
|
||
});
|
||
|
||
test('link_cursor_skill_dirs skips the gstack runtime root directory', () => {
|
||
const fnStart = setupContent.indexOf('link_cursor_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('[ "$skill_name" = "gstack" ] && continue');
|
||
// #2444-aware guard: Windows bypass, else only replace symlink-or-missing.
|
||
expect(fnBody).toContain('[ "$IS_WINDOWS" -eq 1 ] || [ -L "$target" ] || [ ! -e "$target" ]');
|
||
});
|
||
|
||
// #2142 deleted existing ~/.cursor/skills/<name> dirs with `rm -rf "$target"`
|
||
// before relinking. That can wipe unowned Cursor skills. Only replace a
|
||
// symlink or a missing path; never the whole skills directory.
|
||
test('link_cursor_skill_dirs does not delete unowned Cursor skill directories', () => {
|
||
const fnStart = setupContent.indexOf('link_cursor_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).not.toContain('rm -rf "$target"');
|
||
expect(fnBody).not.toContain('rm -rf "$skills_dir"');
|
||
expect(setupContent).not.toContain('rm -rf "$CURSOR_SKILLS"');
|
||
});
|
||
|
||
test('Cursor install links generated skills before planting the sidecar', () => {
|
||
const cursorInstall = setupContent.slice(
|
||
setupContent.indexOf('# 6d. Install for Cursor'),
|
||
setupContent.indexOf('# 7. Create .agents/ sidecar'),
|
||
);
|
||
const linkCall = cursorInstall.indexOf('link_cursor_skill_dirs "$SOURCE_GSTACK_DIR"');
|
||
const sidecarCall = cursorInstall.indexOf('create_cursor_sidecar "$SOURCE_GSTACK_DIR"');
|
||
expect(linkCall).toBeGreaterThan(-1);
|
||
expect(sidecarCall).toBeGreaterThan(-1);
|
||
expect(linkCall).toBeLessThan(sidecarCall);
|
||
});
|
||
|
||
test('setup installs OpenCode skills into a nested gstack runtime root', () => {
|
||
expect(setupContent).toContain('create_opencode_runtime_root');
|
||
expect(setupContent).toContain('.opencode/skills');
|
||
expect(setupContent).toContain('review/specialists');
|
||
expect(setupContent).toContain('qa/templates');
|
||
expect(setupContent).toContain('qa/references');
|
||
expect(setupContent).toContain('dx-hall-of-fame.md');
|
||
expect(setupContent).toContain('$opencode_gstack/lib');
|
||
});
|
||
|
||
test('create_agents_sidecar links runtime assets', () => {
|
||
// Sidecar must link bin with its shared lib modules, plus browse, review, qa
|
||
const fnStart = setupContent.indexOf('create_agents_sidecar()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('bin');
|
||
expect(fnBody).toContain('lib');
|
||
expect(fnBody).toContain('browse');
|
||
expect(fnBody).toContain('review');
|
||
expect(fnBody).toContain('qa');
|
||
});
|
||
|
||
test('create_codex_runtime_root exposes only runtime assets', () => {
|
||
const fnStart = setupContent.indexOf('create_codex_runtime_root()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('done', setupContent.indexOf('review/', fnStart)));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('gstack/SKILL.md');
|
||
expect(fnBody).toContain('$codex_gstack/lib');
|
||
expect(fnBody).toContain('browse/dist');
|
||
expect(fnBody).toContain('browse/bin');
|
||
expect(fnBody).toContain('gstack-upgrade/SKILL.md');
|
||
// Review runtime assets (individual files, not the whole dir)
|
||
expect(fnBody).toContain('checklist.md');
|
||
expect(fnBody).toContain('design-checklist.md');
|
||
expect(fnBody).toContain('greptile-triage.md');
|
||
expect(fnBody).toContain('TODOS-format.md');
|
||
expect(fnBody).not.toContain('_link_or_copy "$gstack_dir" "$codex_gstack"');
|
||
});
|
||
|
||
test('create_factory_runtime_root links shared lib modules beside bin', () => {
|
||
const fnStart = setupContent.indexOf('create_factory_runtime_root()');
|
||
const fnEnd = setupContent.indexOf('create_opencode_runtime_root()', fnStart);
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('$factory_gstack/bin');
|
||
expect(fnBody).toContain('$factory_gstack/lib');
|
||
});
|
||
|
||
test('direct Codex installs are migrated out of ~/.codex/skills/gstack', () => {
|
||
expect(setupContent).toContain('migrate_direct_codex_install');
|
||
expect(setupContent).toContain('$HOME/.gstack/repos/gstack');
|
||
expect(setupContent).toContain('avoid duplicate skill discovery');
|
||
});
|
||
|
||
// --- Symlink prefix tests (PR #503) ---
|
||
|
||
test('link_claude_skill_dirs applies gstack- prefix by default', () => {
|
||
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('SKILL_PREFIX');
|
||
expect(fnBody).toContain('link_name="gstack-$skill_name"');
|
||
});
|
||
|
||
test('link_claude_skill_dirs preserves already-prefixed dirs', () => {
|
||
const fnStart = setupContent.indexOf('link_claude_skill_dirs()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('linked[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
// gstack-* dirs should keep their name (e.g., gstack-upgrade stays gstack-upgrade)
|
||
expect(fnBody).toContain('gstack-*) link_name="$skill_name"');
|
||
});
|
||
|
||
test('setup supports --no-prefix flag', () => {
|
||
expect(setupContent).toContain('--no-prefix');
|
||
expect(setupContent).toContain('SKILL_PREFIX=0');
|
||
});
|
||
|
||
test('cleanup_old_claude_symlinks removes only gstack-pointing symlinks', () => {
|
||
expect(setupContent).toContain('cleanup_old_claude_symlinks');
|
||
const fnStart = setupContent.indexOf('cleanup_old_claude_symlinks()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('removed[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
// Should check readlink before removing
|
||
expect(fnBody).toContain('readlink');
|
||
expect(fnBody).toContain('gstack/*');
|
||
// Should skip already-prefixed dirs
|
||
expect(fnBody).toContain('gstack-*) continue');
|
||
});
|
||
|
||
test('cleanup runs before link when prefix is enabled', () => {
|
||
// In the Claude install section, cleanup should happen before linking
|
||
const claudeInstallSection = setupContent.slice(
|
||
setupContent.indexOf('INSTALL_CLAUDE'),
|
||
setupContent.lastIndexOf('link_claude_skill_dirs')
|
||
);
|
||
expect(claudeInstallSection).toContain('cleanup_old_claude_symlinks');
|
||
});
|
||
|
||
// --- Persistent config + interactive prompt tests ---
|
||
|
||
test('setup reads skill_prefix from config', () => {
|
||
expect(setupContent).toContain('get skill_prefix');
|
||
expect(setupContent).toContain('GSTACK_CONFIG');
|
||
});
|
||
|
||
test('setup supports --prefix flag', () => {
|
||
expect(setupContent).toContain('--prefix)');
|
||
expect(setupContent).toContain('SKILL_PREFIX=1; SKILL_PREFIX_FLAG=1');
|
||
});
|
||
|
||
test('--prefix and --no-prefix persist to config', () => {
|
||
expect(setupContent).toContain('set skill_prefix');
|
||
});
|
||
|
||
test('interactive prompt shows when no config', () => {
|
||
expect(setupContent).toContain('Short names');
|
||
expect(setupContent).toContain('Namespaced');
|
||
expect(setupContent).toContain('Choice [1/2]');
|
||
});
|
||
|
||
test('non-TTY defaults to flat names', () => {
|
||
// Should check if stdin is a TTY before prompting
|
||
expect(setupContent).toContain('-t 0');
|
||
});
|
||
|
||
test('cleanup_prefixed_claude_symlinks exists and uses readlink', () => {
|
||
expect(setupContent).toContain('cleanup_prefixed_claude_symlinks');
|
||
const fnStart = setupContent.indexOf('cleanup_prefixed_claude_symlinks()');
|
||
const fnEnd = setupContent.indexOf('}', setupContent.indexOf('removed[@]}', fnStart));
|
||
const fnBody = setupContent.slice(fnStart, fnEnd);
|
||
expect(fnBody).toContain('readlink');
|
||
expect(fnBody).toContain('gstack-$skill_name');
|
||
});
|
||
|
||
test('reverse cleanup runs before link when prefix is disabled', () => {
|
||
const claudeInstallSection = setupContent.slice(
|
||
setupContent.indexOf('INSTALL_CLAUDE'),
|
||
setupContent.lastIndexOf('link_claude_skill_dirs')
|
||
);
|
||
expect(claudeInstallSection).toContain('cleanup_prefixed_claude_symlinks');
|
||
});
|
||
|
||
test('welcome message references SKILL_PREFIX', () => {
|
||
// gstack-upgrade is always called gstack-upgrade (it's the actual dir name)
|
||
// but the welcome section should exist near the prefix logic
|
||
expect(setupContent).toContain('Run /gstack-upgrade anytime');
|
||
});
|
||
});
|
||
|
||
describe('discover-skills hidden directory filtering', () => {
|
||
test('discoverTemplates skips dot-prefixed directories', () => {
|
||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-discover-'));
|
||
try {
|
||
// Create a hidden dir with a template (should be excluded)
|
||
fs.mkdirSync(path.join(tmpDir, '.hidden'), { recursive: true });
|
||
fs.writeFileSync(path.join(tmpDir, '.hidden', 'SKILL.md.tmpl'), '---\nname: evil\n---\ntest');
|
||
// Create a visible dir with a template (should be included)
|
||
fs.mkdirSync(path.join(tmpDir, 'visible'), { recursive: true });
|
||
fs.writeFileSync(path.join(tmpDir, 'visible', 'SKILL.md.tmpl'), '---\nname: good\n---\ntest');
|
||
|
||
const { discoverTemplates } = require('../scripts/discover-skills');
|
||
const results = discoverTemplates(tmpDir);
|
||
const dirs = results.map((r: { tmpl: string }) => r.tmpl);
|
||
|
||
expect(dirs).toContain('visible/SKILL.md.tmpl');
|
||
expect(dirs).not.toContain('.hidden/SKILL.md.tmpl');
|
||
} finally {
|
||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('telemetry', () => {
|
||
test('telemetry start block lives in gstack-skill-start; render notes the handoff keys', () => {
|
||
// The start-block bash moved into the script (Phase 1): it reads the
|
||
// config, mints the session identity, and echoes the STATUS keys.
|
||
expect(SKILL_START_SCRIPT).toContain('_TEL_START=$(date +%s)');
|
||
expect(SKILL_START_SCRIPT).toContain('_SESSION_ID=');
|
||
expect(SKILL_START_SCRIPT).toContain('echo "TELEMETRY:');
|
||
expect(SKILL_START_SCRIPT).toContain('echo "TEL_PROMPTED:');
|
||
expect(SKILL_START_SCRIPT).toMatch(/gstack-config" get telemetry/);
|
||
// The render must tell the model to carry SESSION_ID/TEL_START to skill end.
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('SESSION_ID');
|
||
expect(content).toContain('TEL_START');
|
||
});
|
||
|
||
test('telemetry opt-in prompt lives in gstack-skill-start (marker-gated emit)', () => {
|
||
// Token-reduction Phase 2: the one-time consent prompt left the renders
|
||
// (absence pinned by test/onboarding-moved-literals.test.ts); the script
|
||
// gates it on the marker files and emits it as a GSTACK_INSTRUCTION block
|
||
// with all three config-set outcomes and the ack INSIDE the block.
|
||
expect(SKILL_START_SCRIPT).toContain(
|
||
'if [ "$_TEL_PROMPTED" = "no" ] && [ "$_LAKE_SEEN" = "yes" ]; then',
|
||
);
|
||
expect(SKILL_START_SCRIPT).toContain('_emit_block telemetry-prompt');
|
||
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry community');
|
||
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry anonymous');
|
||
expect(SKILL_START_SCRIPT).toContain('gstack-config set telemetry off');
|
||
expect(SKILL_START_SCRIPT).toContain('touch "$_GH/.telemetry-prompted"');
|
||
});
|
||
|
||
test('generated SKILL.md contains telemetry epilogue (one gstack-skill-end call)', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Telemetry (run last)');
|
||
expect(content).toContain('gstack-skill-end --skill "gstack" --outcome OUTCOME');
|
||
expect(content).toContain('--tel-start "TEL_START"');
|
||
expect(content).toContain('PLAN MODE EXCEPTION');
|
||
// The duration math + remote-log dispatch moved into gstack-skill-end.
|
||
expect(SKILL_END_SCRIPT).toContain('_TEL_END');
|
||
expect(SKILL_END_SCRIPT).toContain('_TEL_DUR');
|
||
expect(SKILL_END_SCRIPT).toContain('SKILL_NAME');
|
||
expect(SKILL_END_SCRIPT).toContain('OUTCOME');
|
||
expect(SKILL_END_SCRIPT).toContain('gstack-telemetry-log');
|
||
});
|
||
|
||
test('pending marker handling lives in the scripts', () => {
|
||
// gstack-skill-start finalizes stale markers; gstack-skill-end clears the
|
||
// session's own marker.
|
||
expect(SKILL_START_SCRIPT).toContain("-name '.pending-*'");
|
||
expect(SKILL_START_SCRIPT).toContain('_pending_finalize');
|
||
expect(SKILL_END_SCRIPT).toContain('.pending-$SESSION_ID');
|
||
});
|
||
|
||
test('telemetry blocks appear in all skill files that use PREAMBLE', () => {
|
||
const skills = ['qa', 'ship', 'review', 'plan-ceo-review', 'plan-eng-review', 'retro'];
|
||
for (const skill of skills) {
|
||
const skillPath = path.join(ROOT, skill, 'SKILL.md');
|
||
if (fs.existsSync(skillPath)) {
|
||
const content = fs.readFileSync(skillPath, 'utf-8');
|
||
expect(content).toContain('Telemetry (run last)');
|
||
expect(content).toContain(`gstack-skill-end --skill "${skill}"`);
|
||
expect(content).toContain('--tel-start "TEL_START"');
|
||
}
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('community fixes wave', () => {
|
||
// Helper to get all generated SKILL.md files
|
||
function getAllSkillMds(): Array<{ name: string; content: string }> {
|
||
const results: Array<{ name: string; content: string }> = [];
|
||
const rootPath = path.join(ROOT, 'SKILL.md');
|
||
if (fs.existsSync(rootPath)) {
|
||
results.push({ name: 'root', content: fs.readFileSync(rootPath, 'utf-8') });
|
||
}
|
||
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
|
||
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||
const skillPath = path.join(ROOT, entry.name, 'SKILL.md');
|
||
if (fs.existsSync(skillPath)) {
|
||
results.push({ name: entry.name, content: fs.readFileSync(skillPath, 'utf-8') });
|
||
}
|
||
}
|
||
return results;
|
||
}
|
||
|
||
// #594 — Discoverability: every SKILL.md.tmpl description contains "gstack"
|
||
test('every SKILL.md.tmpl description contains "gstack"', () => {
|
||
for (const skill of ALL_SKILLS) {
|
||
const tmplPath = skill.dir === '.' ? path.join(ROOT, 'SKILL.md.tmpl') : path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
||
const content = fs.readFileSync(tmplPath, 'utf-8');
|
||
const desc = extractDescription(content);
|
||
expect(desc.toLowerCase()).toContain('gstack');
|
||
}
|
||
});
|
||
|
||
// #594 — Discoverability: first line of each description is under 120 chars
|
||
test('every SKILL.md.tmpl description first line is under 120 chars', () => {
|
||
for (const skill of ALL_SKILLS) {
|
||
const tmplPath = skill.dir === '.' ? path.join(ROOT, 'SKILL.md.tmpl') : path.join(ROOT, skill.dir, 'SKILL.md.tmpl');
|
||
const content = fs.readFileSync(tmplPath, 'utf-8');
|
||
const desc = extractDescription(content);
|
||
const firstLine = desc.split('\n')[0];
|
||
expect(firstLine.length).toBeLessThanOrEqual(120);
|
||
}
|
||
});
|
||
|
||
// #573 — Feature signals: ship/SKILL.md contains feature signal detection
|
||
test('ship/SKILL.md contains feature signal detection in Step 4', () => {
|
||
const content = readShipUnion();
|
||
expect(content.toLowerCase()).toContain('feature signal');
|
||
});
|
||
|
||
// #510 — Context warnings: no SKILL.md contains "running low on context"
|
||
test('no generated SKILL.md contains "running low on context"', () => {
|
||
const skills = getAllSkillMds();
|
||
for (const { name, content } of skills) {
|
||
expect(content).not.toContain('running low on context');
|
||
}
|
||
});
|
||
|
||
// #510 — Context warnings: plan-eng-review has explicit anti-warning
|
||
test('plan-eng-review/SKILL.md explicitly forbids preemptive context warnings', () => {
|
||
const content = readSkillUnion('plan-eng-review'); // carved: review body moved to section
|
||
expect(content.toLowerCase()).toContain('do not preemptively warn');
|
||
});
|
||
|
||
// #474 — Safety Net: no SKILL.md uses find with -delete
|
||
test('no generated SKILL.md contains find with -delete flag', () => {
|
||
const skills = getAllSkillMds();
|
||
for (const { name, content } of skills) {
|
||
// Match find commands that use -delete (but not prose mentioning the word "delete")
|
||
const lines = content.split('\n');
|
||
for (const line of lines) {
|
||
if (line.includes('find ') && line.includes('-delete')) {
|
||
throw new Error(`${name}/SKILL.md contains find with -delete: ${line.trim()}`);
|
||
}
|
||
}
|
||
}
|
||
});
|
||
|
||
// #467 — Telemetry: preamble JSONL writes are gated by telemetry setting
|
||
test('preamble JSONL writes are inside telemetry conditional', () => {
|
||
const preamble = fs.readFileSync(path.join(ROOT, 'scripts/resolvers/preamble.ts'), 'utf-8');
|
||
// Find all skill-usage.jsonl write lines
|
||
const lines = preamble.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
if (lines[i].includes('skill-usage.jsonl') && lines[i].includes('>>')) {
|
||
// Look backwards for a telemetry conditional within 5 lines
|
||
let foundConditional = false;
|
||
for (let j = i - 1; j >= Math.max(0, i - 5); j--) {
|
||
if (lines[j].includes('_TEL') && lines[j].includes('off')) {
|
||
foundConditional = true;
|
||
break;
|
||
}
|
||
}
|
||
expect(foundConditional).toBe(true);
|
||
}
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('codex commands must not use inline $(git rev-parse --show-toplevel) for cwd', () => {
|
||
// Regression test: inline $(git rev-parse --show-toplevel) in codex exec -C
|
||
// or codex review without cd evaluates in whatever cwd the background shell
|
||
// inherits, which may be a different project in Conductor workspaces.
|
||
// The fix is to resolve _REPO_ROOT eagerly at the top of each bash block.
|
||
|
||
// Scan all source files that could contain codex commands
|
||
// Use Bun.Glob to avoid ELOOP from .claude/skills/gstack symlink back to ROOT
|
||
const tmplGlob = new Bun.Glob('**/*.tmpl');
|
||
const sourceFiles = [
|
||
...Array.from(tmplGlob.scanSync({ cwd: ROOT, followSymlinks: false })),
|
||
...fs.readdirSync(path.join(ROOT, 'scripts/resolvers'))
|
||
.filter(f => f.endsWith('.ts'))
|
||
.map(f => `scripts/resolvers/${f}`),
|
||
'scripts/gen-skill-docs.ts',
|
||
];
|
||
|
||
test('no codex exec command uses inline $(git rev-parse --show-toplevel) in -C flag', () => {
|
||
const violations: string[] = [];
|
||
for (const rel of sourceFiles) {
|
||
const abs = path.join(ROOT, rel);
|
||
if (!fs.existsSync(abs)) continue;
|
||
const content = fs.readFileSync(abs, 'utf-8');
|
||
const lines = content.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
const line = lines[i];
|
||
if (line.includes('codex exec') && line.includes('-C') && line.includes('$(git rev-parse --show-toplevel)')) {
|
||
violations.push(`${rel}:${i + 1}`);
|
||
}
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
|
||
test('no generated SKILL.md has codex exec with inline $(git rev-parse --show-toplevel) in -C flag', () => {
|
||
const violations: string[] = [];
|
||
const skillMdGlob = new Bun.Glob('**/SKILL.md');
|
||
const skillMdFiles = Array.from(skillMdGlob.scanSync({ cwd: ROOT, followSymlinks: false }));
|
||
for (const rel of skillMdFiles) {
|
||
const abs = path.join(ROOT, rel);
|
||
if (!fs.existsSync(abs)) continue;
|
||
const content = fs.readFileSync(abs, 'utf-8');
|
||
const lines = content.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
const line = lines[i];
|
||
if (line.includes('codex exec') && line.includes('-C') && line.includes('$(git rev-parse --show-toplevel)')) {
|
||
violations.push(`${rel}:${i + 1}`);
|
||
}
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
|
||
test('codex review commands must be preceded by cd "$_REPO_ROOT" (no -C support)', () => {
|
||
// codex review does not support -C, so the pattern must be:
|
||
// _REPO_ROOT=$(git rev-parse --show-toplevel) || { ... }
|
||
// cd "$_REPO_ROOT"
|
||
// codex review ...
|
||
// NOT: codex review ... with inline $(git rev-parse --show-toplevel)
|
||
const allFiles = [
|
||
...Array.from(tmplGlob.scanSync({ cwd: ROOT, followSymlinks: false })),
|
||
...Array.from(new Bun.Glob('**/SKILL.md').scanSync({ cwd: ROOT, followSymlinks: false })),
|
||
...fs.readdirSync(path.join(ROOT, 'scripts/resolvers'))
|
||
.filter(f => f.endsWith('.ts'))
|
||
.map(f => `scripts/resolvers/${f}`),
|
||
'scripts/gen-skill-docs.ts',
|
||
];
|
||
const violations: string[] = [];
|
||
for (const rel of allFiles) {
|
||
const abs = path.join(ROOT, rel);
|
||
if (!fs.existsSync(abs)) continue;
|
||
const content = fs.readFileSync(abs, 'utf-8');
|
||
const lines = content.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
const line = lines[i];
|
||
// Skip non-executable lines (markdown table cells, prose references)
|
||
if (line.includes('|') && line.includes('`/codex review`')) continue;
|
||
if (line.includes('`codex review`')) continue;
|
||
// Check for codex review with inline $(git rev-parse)
|
||
if (line.includes('codex review') && line.includes('$(git rev-parse --show-toplevel)')) {
|
||
violations.push(`${rel}:${i + 1} — inline git rev-parse in codex review`);
|
||
}
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
|
||
test('codex review commands take their scope from a flag, never from prompt text', () => {
|
||
// `codex review` scope comes ONLY from --base/--commit/--uncommitted. The
|
||
// positional [PROMPT] is mutually exclusive with all three (#1428, #1479),
|
||
// and a prompt-only `codex review` silently falls back to the *uncommitted
|
||
// working-tree* scope (`git status --short; git diff`) — so describing the
|
||
// diff range in prompt text produces a confident review of the wrong
|
||
// changes, with no error. Both halves are pinned here:
|
||
// (a) every `codex review` invocation carries a scope flag, and
|
||
// (b) no invocation puts a positional prompt in front of that flag.
|
||
//
|
||
// This does NOT apply to `codex exec`, which is agentic and really does run
|
||
// the git command it's told to — the adversarial pass legitimately scopes
|
||
// itself in prompt text.
|
||
const checkedFiles = [
|
||
'scripts/resolvers/review-dashboard.ts',
|
||
'scripts/resolvers/plan-gates.ts',
|
||
'scripts/resolvers/spec-review.ts',
|
||
'scripts/resolvers/outside-voice-steps.ts',
|
||
'scripts/resolvers/review-scope.ts',
|
||
'review/SKILL.md',
|
||
'ship/SKILL.md',
|
||
'codex/SKILL.md.tmpl',
|
||
'codex/SKILL.md',
|
||
// codex's scoped invocations moved into the carved review-mode section
|
||
// (T9) — keep sweeping both the .tmpl source and the generated section.
|
||
'codex/sections/review-mode.md.tmpl',
|
||
'codex/sections/review-mode.md',
|
||
];
|
||
|
||
const violations: string[] = [];
|
||
for (const rel of checkedFiles) {
|
||
// ship's codex/adversarial command moved into sections/adversarial.md (T9 carve).
|
||
const content = rel === 'ship/SKILL.md' ? readShipUnion() : fs.readFileSync(path.join(ROOT, rel), 'utf-8');
|
||
const lines = content.split('\n');
|
||
for (let i = 0; i < lines.length; i++) {
|
||
const line = lines[i];
|
||
// Only inspect real shell invocations, not prose mentioning the command.
|
||
if (line.includes('`codex review`')) continue;
|
||
const match = line.match(/(?:^|[;&|]\s*|\s)codex\s+review\b(.*)$/);
|
||
if (!match) continue;
|
||
const rest = match[1];
|
||
const scopeFlag = /--base\b|--commit\b|--uncommitted\b/;
|
||
if (!scopeFlag.test(rest)) {
|
||
// A quoted prompt with no scope flag is the silent-wrong-scope bug.
|
||
if (/^\s*["'$]/.test(rest)) {
|
||
violations.push(`${rel}:${i + 1} — prompt-only codex review (falls back to working-tree scope)`);
|
||
}
|
||
continue;
|
||
}
|
||
const beforeFlag = rest.split(scopeFlag)[0].trim();
|
||
if (/^["'$]|^--\s*["']/.test(beforeFlag)) {
|
||
violations.push(`${rel}:${i + 1} — positional prompt passed alongside a scope flag`);
|
||
}
|
||
}
|
||
}
|
||
expect(violations).toEqual([]);
|
||
});
|
||
});
|
||
|
||
// ─── Learnings + Confidence Resolver Tests ─────────────────────
|
||
|
||
describe('LEARNINGS_SEARCH resolver', () => {
|
||
const SEARCH_SKILLS = ['review', 'ship', 'plan-eng-review', 'investigate', 'office-hours', 'plan-ceo-review'];
|
||
|
||
for (const skill of SEARCH_SKILLS) {
|
||
test(`${skill} generated SKILL.md contains learnings search`, () => {
|
||
const content = readSkillUnion(skill); // ship: moved to sections/plan-completion.md
|
||
expect(content).toContain('Prior Learnings');
|
||
expect(content).toContain('gstack-learnings-search');
|
||
});
|
||
}
|
||
|
||
test('learnings search includes cross-project config check', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('cross_project_learnings');
|
||
expect(content).toContain('--cross-project');
|
||
});
|
||
|
||
test('learnings search includes AskUserQuestion for first-time cross-project opt-in', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Enable cross-project learnings');
|
||
expect(content).toContain('project-scoped only');
|
||
});
|
||
|
||
test('Eng clarifies its full brief without changing any host options or other skills', async () => {
|
||
const { generateLearningsSearch } = await import('../scripts/resolvers/learnings');
|
||
const { ALL_HOST_CONFIGS } = await import('../hosts');
|
||
const { HOST_PATHS } = await import('../scripts/resolvers/types');
|
||
const fullBrief = 'Build a full decision brief from these facts and options using the preamble format, then ask and wait:';
|
||
for (const host of ALL_HOST_CONFIGS) {
|
||
const ctx = { skillName: 'plan-eng-review', tmplPath: 'plan-eng-review/SKILL.md.tmpl', host: host.name, paths: HOST_PATHS[host.name]! };
|
||
const eng = generateLearningsSearch(ctx);
|
||
const standard = generateLearningsSearch({ ...ctx, skillName: 'review' });
|
||
expect(generateLearningsSearch({ ...ctx, skillName: 'plan-ceo-review' })).toBe(standard);
|
||
if (host.learningsMode === 'basic') {
|
||
expect(eng).toBe(standard);
|
||
expect(eng).not.toContain('CROSS_PROJECT');
|
||
} else {
|
||
expect(eng).toContain(fullBrief);
|
||
expect(eng.replace(fullBrief, 'Use AskUserQuestion:')).toBe(standard);
|
||
expect(eng).toContain('A) Enable cross-project learnings (recommended)');
|
||
expect(eng).toContain('B) Keep learnings project-scoped only');
|
||
expect(eng).toContain('gstack-config set cross_project_learnings true');
|
||
expect(eng).toContain('gstack-config set cross_project_learnings false');
|
||
}
|
||
}
|
||
});
|
||
|
||
test('learnings search mentions prior learning applied display format', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Prior learning applied');
|
||
});
|
||
});
|
||
|
||
describe('LEARNINGS_LOG resolver', () => {
|
||
const LOG_SKILLS = ['review', 'retro', 'investigate'];
|
||
|
||
for (const skill of LOG_SKILLS) {
|
||
test(`${skill} generated SKILL.md contains learnings log`, () => {
|
||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Capture Learnings');
|
||
expect(content).toContain('gstack-learnings-log');
|
||
});
|
||
}
|
||
|
||
test('learnings log documents all type values', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
for (const type of ['pattern', 'pitfall', 'preference', 'architecture', 'tool']) {
|
||
expect(content).toContain(type);
|
||
}
|
||
});
|
||
|
||
test('learnings log documents all source values', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
for (const source of ['observed', 'user-stated', 'inferred', 'cross-model']) {
|
||
expect(content).toContain(source);
|
||
}
|
||
});
|
||
|
||
test('learnings log includes files field for staleness detection', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('"files"');
|
||
expect(content).toContain('staleness detection');
|
||
});
|
||
});
|
||
|
||
describe('CONFIDENCE_CALIBRATION resolver', () => {
|
||
// CSO owns a distinct evidence rubric; the shared numerical confidence
|
||
// resolver would conflict with that contract and its private startup.
|
||
const CONFIDENCE_SKILLS = ['review', 'ship', 'plan-eng-review'];
|
||
|
||
for (const skill of CONFIDENCE_SKILLS) {
|
||
test(`${skill} generated SKILL.md contains confidence calibration`, () => {
|
||
const content = readSkillUnion(skill); // ship: moved to sections/review-army.md
|
||
expect(content).toContain('Confidence Calibration');
|
||
expect(content).toContain(skill === 'review' ? 'score every finding (1-10)' : 'confidence score');
|
||
});
|
||
}
|
||
|
||
test('confidence calibration includes scoring rubric with all tiers', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('9-10');
|
||
expect(content).toContain('7-8');
|
||
expect(content).toContain('5-6');
|
||
expect(content).toContain('3-4');
|
||
expect(content).toContain('1-2');
|
||
});
|
||
|
||
test('confidence calibration includes display rules', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('Show normally');
|
||
expect(content).toContain('Suppress from main report');
|
||
});
|
||
|
||
test('confidence calibration includes finding format example', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('[CRITICAL] (confidence:');
|
||
expect(content).toContain('SQL injection');
|
||
});
|
||
|
||
test('confidence calibration includes calibration learning feedback loop', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
||
const flat = content.replace(/\s+/g, ' ');
|
||
expect(flat).toContain('Calibration learning');
|
||
expect(flat).toContain('log the corrected pattern as a learning');
|
||
});
|
||
|
||
test('skills without confidence calibration do NOT contain it', () => {
|
||
// office-hours and retro do NOT use confidence calibration
|
||
for (const skill of ['office-hours', 'retro']) {
|
||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
expect(content).not.toContain('## Confidence Calibration');
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('gen-skill-docs prefix warning (#620/#578)', () => {
|
||
const { execSync } = require('child_process');
|
||
|
||
test('warns about skill_prefix when config has prefix=true', () => {
|
||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-prefix-warn-'));
|
||
try {
|
||
// Create a fake ~/.gstack/config.yaml with skill_prefix: true
|
||
const fakeHome = tmpDir;
|
||
const fakeGstack = path.join(fakeHome, '.gstack');
|
||
fs.mkdirSync(fakeGstack, { recursive: true });
|
||
fs.writeFileSync(path.join(fakeGstack, 'config.yaml'), 'skill_prefix: true\n');
|
||
|
||
// Render into an out-dir under the fixture (the warning fires on any
|
||
// non-dry-run generation) so the live tree is never rewritten.
|
||
const outDir = path.join(tmpDir, 'out');
|
||
const output = execSync(`bun run scripts/gen-skill-docs.ts --out-dir "${outDir}"`, {
|
||
cwd: ROOT,
|
||
env: { ...process.env, HOME: fakeHome },
|
||
encoding: 'utf-8',
|
||
timeout: 30000,
|
||
});
|
||
expect(output).toContain('skill_prefix is true');
|
||
expect(output).toContain('gstack-relink');
|
||
} finally {
|
||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||
}
|
||
});
|
||
|
||
test('no warning when skill_prefix is false or absent', () => {
|
||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-prefix-warn-'));
|
||
try {
|
||
const fakeHome = tmpDir;
|
||
const fakeGstack = path.join(fakeHome, '.gstack');
|
||
fs.mkdirSync(fakeGstack, { recursive: true });
|
||
fs.writeFileSync(path.join(fakeGstack, 'config.yaml'), 'skill_prefix: false\n');
|
||
|
||
const outDir = path.join(tmpDir, 'out');
|
||
const output = execSync(`bun run scripts/gen-skill-docs.ts --out-dir "${outDir}"`, {
|
||
cwd: ROOT,
|
||
env: { ...process.env, HOME: fakeHome },
|
||
encoding: 'utf-8',
|
||
timeout: 30000,
|
||
});
|
||
expect(output).not.toContain('skill_prefix is true');
|
||
} finally {
|
||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('voice-triggers processing', () => {
|
||
const { extractVoiceTriggers, processVoiceTriggers } = require('../scripts/gen-skill-docs') as {
|
||
extractVoiceTriggers: (content: string) => string[];
|
||
processVoiceTriggers: (content: string) => string;
|
||
};
|
||
|
||
test('extractVoiceTriggers parses valid YAML list', () => {
|
||
const content = `---\nname: cso\ndescription: |\n Security audit.\nvoice-triggers:\n - "see-so"\n - "security review"\n---\nBody`;
|
||
const triggers = extractVoiceTriggers(content);
|
||
expect(triggers).toEqual(['see-so', 'security review']);
|
||
});
|
||
|
||
test('extractVoiceTriggers returns [] when no field present', () => {
|
||
const content = `---\nname: qa\ndescription: |\n QA testing.\n---\nBody`;
|
||
expect(extractVoiceTriggers(content)).toEqual([]);
|
||
});
|
||
|
||
test('processVoiceTriggers appends voice triggers to description', () => {
|
||
const content = `---\nname: cso\ndescription: |\n Security audit. (gstack)\nvoice-triggers:\n - "see-so"\n - "security review"\n---\nBody`;
|
||
const result = processVoiceTriggers(content);
|
||
expect(result).toContain('Voice triggers (speech-to-text aliases): "see-so", "security review".');
|
||
});
|
||
|
||
test('processVoiceTriggers strips voice-triggers field from output', () => {
|
||
const content = `---\nname: cso\ndescription: |\n Security audit. (gstack)\nvoice-triggers:\n - "see-so"\n---\nBody`;
|
||
const result = processVoiceTriggers(content);
|
||
expect(result).not.toContain('voice-triggers:');
|
||
});
|
||
|
||
test('processVoiceTriggers returns content unchanged when no voice-triggers', () => {
|
||
const content = `---\nname: qa\ndescription: |\n QA testing.\n---\nBody`;
|
||
expect(processVoiceTriggers(content)).toBe(content);
|
||
});
|
||
|
||
test('generated CSO SKILL.md contains voice triggers in description', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
||
expect(content).toContain('"see-so"');
|
||
expect(content).toContain('Voice triggers (speech-to-text aliases):');
|
||
});
|
||
|
||
test('generated CSO SKILL.md does NOT contain raw voice-triggers field', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = content.slice(0, fmEnd);
|
||
expect(frontmatter).not.toContain('voice-triggers:');
|
||
});
|
||
|
||
test('generated Claude CSO skill preauthorizes only the trusted launcher', () => {
|
||
const expected = [
|
||
'Bash(~/.claude/skills/gstack/bin/gstack-cso-launcher *)',
|
||
'Bash(~/.claude/skills/gstack/bin/gstack-cso-launcher.exe *)',
|
||
];
|
||
for (const file of ['cso/SKILL.md.tmpl', 'cso/SKILL.md']) {
|
||
const content = fs.readFileSync(path.join(ROOT, file), 'utf-8');
|
||
const fmEnd = content.indexOf('\n---', 4);
|
||
const frontmatter = Bun.YAML.parse(content.slice(4, fmEnd)) as Record<string, unknown>;
|
||
expect(frontmatter['allowed-tools'], file).toEqual(expected);
|
||
}
|
||
});
|
||
|
||
test('generated CSO host variants retain challenge fallback and host-containment disclosure', () => {
|
||
const claude = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
|
||
const codex = fs.readFileSync(path.join(EXTERNAL_OUT, '.agents', 'skills', 'gstack-cso', 'SKILL.md'), 'utf-8');
|
||
for (const content of [claude, codex]) {
|
||
expect(content).toContain('sequential challenge; independent agent unavailable');
|
||
expectMentions(content, [['does not','containment','sandbox']], 'content');
|
||
expectMentions(content, [['do not','independent','reviewer']], 'content');
|
||
}
|
||
});
|
||
|
||
// Gen-time-only keys: interactive + benefits-from are read from the .tmpl by
|
||
// buildContext; the generated copy has no reader (the host reads name/
|
||
// description/allowed-tools/hooks; gbrain: is runtime-read and NOT stripped).
|
||
// Pin the strip so a stripFields refactor can't silently re-add the always-on
|
||
// frontmatter weight — mirrors the voice-triggers pins above.
|
||
test('generated SKILL.md strips gen-time-only keys the .tmpl still declares', () => {
|
||
const tmpl = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md.tmpl'), 'utf-8');
|
||
const tmplFm = tmpl.slice(0, tmpl.indexOf('\n---', 4));
|
||
expect(tmplFm).toContain('interactive:');
|
||
expect(tmplFm).toContain('benefits-from:');
|
||
|
||
const generated = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||
const genFm = generated.slice(0, generated.indexOf('\n---', 4));
|
||
expect(genFm).not.toContain('interactive:');
|
||
expect(genFm).not.toContain('benefits-from:');
|
||
|
||
// The runtime-read and host-read keys survive the strip.
|
||
const investigate = fs.readFileSync(path.join(ROOT, 'investigate', 'SKILL.md'), 'utf-8');
|
||
const invFm = investigate.slice(0, investigate.indexOf('\n---', 4));
|
||
expect(invFm).toContain('hooks:');
|
||
expect(invFm).toContain('gbrain:');
|
||
});
|
||
});
|
||
|
||
describe('CEO accepted requirement preservation', () => {
|
||
test('HOLD SCOPE includes implementation work needed to meet accepted requirements', () => {
|
||
const main = fs.readFileSync(path.join(ROOT, 'plan-ceo-review/SKILL.md'), 'utf8');
|
||
const hold = main.slice(main.indexOf('**For HOLD SCOPE**'), main.indexOf('**For SCOPE REDUCTION**'));
|
||
expectMentions(hold, [['acceptance criteria', 'in scope']], 'HOLD SCOPE');
|
||
});
|
||
|
||
test('loaded sections keep requirement conflicts unresolved until an authorized decision', () => {
|
||
const section = fs.readFileSync(path.join(ROOT, 'plan-ceo-review/sections/review-sections.md'), 'utf8');
|
||
const policy = section.slice(section.indexOf('**Preserve accepted requirements.**'), section.indexOf('### Section 1:')).replace(/\s+/g, ' ');
|
||
expect(policy).toContain('Report gaps and propose remedies, including omitted mechanisms in HOLD SCOPE');
|
||
expectMentions(policy, [['never','guarantee','violation']], 'policy');
|
||
expectMentions(policy, [['do not','documentation','requirements']], 'policy');
|
||
expect(policy).toContain('Changing a requirement needs explicit authority');
|
||
expectMentions(policy, [['cannot','auto-decide','constraints']], 'policy');
|
||
});
|
||
});
|
||
|
||
describe('plan-mode-info resolver (handshake-replacement)', () => {
|
||
const REVIEW_SKILLS = [
|
||
'plan-ceo-review',
|
||
'plan-eng-review',
|
||
'plan-design-review',
|
||
'plan-devex-review',
|
||
];
|
||
|
||
// Header for the vestigial handshake that was removed. If it ever reappears,
|
||
// someone accidentally re-introduced the resolver.
|
||
const HANDSHAKE_MARKER = '## Plan Mode Handshake';
|
||
// Header for the new plan-mode-info section (previously lived at the tail
|
||
// of completion-status.ts; now hoisted to position 1 of the preamble).
|
||
const PLAN_MODE_INFO_MARKER = '## Skill Invocation During Plan Mode';
|
||
|
||
test('vestigial handshake is absent from all generated Claude SKILL.md files', () => {
|
||
// Scan every generated SKILL.md under ROOT (top-level directory per skill).
|
||
// Using fs.readdirSync + filter instead of a glob so we catch any skill
|
||
// that gets added later without updating this list.
|
||
const entries = fs.readdirSync(ROOT, { withFileTypes: true });
|
||
let checked = 0;
|
||
for (const entry of entries) {
|
||
if (!entry.isDirectory()) continue;
|
||
const skillMd = path.join(ROOT, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
expect(content, `handshake marker in ${entry.name}/SKILL.md`).not.toContain(HANDSHAKE_MARKER);
|
||
checked++;
|
||
}
|
||
expect(checked).toBeGreaterThan(0);
|
||
});
|
||
|
||
test('vestigial handshake is absent from non-Claude host outputs', () => {
|
||
// Non-Claude hosts render to hostSubdirs (.agents/, .openclaw/, etc). The
|
||
// plan-mode-info resolver has no host-scoping — all hosts get the new
|
||
// section, none get the old handshake. Scan every candidate host tree in
|
||
// the module-level out-dir render (--host all), which is always present —
|
||
// so the check can no longer silently degrade to a console warning.
|
||
const hostDirs = ['.agents', '.openclaw', '.opencode', '.factory', '.hermes', '.kiro', '.cursor', '.slate'];
|
||
let checked = 0;
|
||
for (const host of hostDirs) {
|
||
const skillsRoot = path.join(EXTERNAL_OUT, host, 'skills');
|
||
if (!fs.existsSync(skillsRoot)) continue;
|
||
const entries = fs.readdirSync(skillsRoot, { withFileTypes: true });
|
||
for (const entry of entries) {
|
||
if (!entry.isDirectory()) continue;
|
||
const skillMd = path.join(skillsRoot, entry.name, 'SKILL.md');
|
||
if (!fs.existsSync(skillMd)) continue;
|
||
const content = fs.readFileSync(skillMd, 'utf-8');
|
||
expect(content, `handshake marker in ${host}/skills/${entry.name}/SKILL.md`).not.toContain(HANDSHAKE_MARKER);
|
||
checked++;
|
||
}
|
||
}
|
||
expect(checked).toBeGreaterThan(0);
|
||
});
|
||
|
||
test.each(REVIEW_SKILLS)(
|
||
'%s/SKILL.md contains the new plan-mode-info section near the top',
|
||
(skill) => {
|
||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
const idx = content.indexOf(PLAN_MODE_INFO_MARKER);
|
||
expect(idx).toBeGreaterThan(0);
|
||
// Position 1 in preamble composition = within the first ~300 lines.
|
||
// Roughly translates to first ~15KB of text.
|
||
expect(idx).toBeLessThan(15_000);
|
||
},
|
||
);
|
||
|
||
test('plan-mode-info is wired BEFORE generateUpgradeCheck in preamble', () => {
|
||
// Token-reduction Phase 2: generateUpgradeCheck's render output is now
|
||
// ONLY the steady-state PROACTIVE-false + SKILL_PREFIX rules (the
|
||
// UPGRADE_AVAILABLE prose emits from bin/gstack-skill-start at runtime),
|
||
// so those rules are the resolver's order marker.
|
||
const content = fs.readFileSync(
|
||
path.join(ROOT, 'plan-ceo-review', 'SKILL.md'),
|
||
'utf-8',
|
||
);
|
||
const planModeIdx = content.indexOf(PLAN_MODE_INFO_MARKER);
|
||
const upgradeIdx = content.search(/If `PROACTIVE` is `"?false"?`/i);
|
||
expect(planModeIdx).toBeGreaterThan(0);
|
||
expect(upgradeIdx).toBeGreaterThan(0);
|
||
expect(planModeIdx).toBeLessThan(upgradeIdx);
|
||
});
|
||
|
||
test('0D authority and fresh-approval paths precede mode selection', () => {
|
||
const content = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8').replace(/\s+/g, ' ');
|
||
const approachIdx = content.indexOf('### 0D.');
|
||
const constructIdx = content.indexOf('Build one `currentDecision`', approachIdx);
|
||
const saveIdx = content.indexOf('**Pre-question checkpoint:**', constructIdx);
|
||
const presentIdx = content.indexOf('Ask one row per call', saveIdx);
|
||
const stopOffset = content.slice(presentIdx).search(/stop for the actual answer/i);
|
||
const stopIdx = stopOffset < 0 ? -1 : presentIdx + stopOffset;
|
||
const modeIdx = content.indexOf('### 0E. Mode Selection');
|
||
const preludeIdx = content.indexOf('### 0F');
|
||
const positions = [approachIdx, constructIdx, saveIdx, presentIdx, stopIdx, modeIdx, preludeIdx];
|
||
expect(positions.every(position => position > 0)).toBe(true);
|
||
expect(positions).toEqual([...positions].sort((a, b) => a - b));
|
||
const approach = content.slice(approachIdx, modeIdx);
|
||
expect(approach).toMatch(/Reuse (?:an )?exact approvals?/);
|
||
expect(content).toMatch(/(?:the actual instruction\/answer|Cite actual instructions\/answers) and exact scope/);
|
||
const reopenRule = approach.match(/Reuse exact approvals\. Reopen only for contradictions, changed assumptions or user instructions, never speculation or reviewer agreement/)?.[0];
|
||
expect(reopenRule).toBeDefined();
|
||
expect(reopenRule!).toMatch(/reopen only|requires reopening it/i);
|
||
expect(content.indexOf(reopenRule!)).toBeGreaterThan(approachIdx);
|
||
expect(content.indexOf(reopenRule!)).toBeLessThan(presentIdx);
|
||
const gate = content.slice(stopIdx, modeIdx);
|
||
const startup = content.slice(content.indexOf('### 0C.'), approachIdx);
|
||
expect(startup).toContain('Before 0E, call 0D for unresolved approaches');
|
||
expect(startup).toContain('A) current/requested plan, B) smallest scoped alternative');
|
||
expectMentions(startup, [['no','required','choices']], 'startup');
|
||
expectMentions(approach, [['not','selection','returns']], 'approach');
|
||
expectTokens(approach, ['**Mode change**'], 'approach');
|
||
const modeChange = content.slice(content.indexOf('**Mode change:**'), preludeIdx);
|
||
expectMentions(modeChange, [['ask','four-mode','answered']], 'modeChange');
|
||
expectMentions(gate, [['do not','calling','return']], 'gate');
|
||
expect(gate).toContain('even for a lone option');
|
||
expect(approach).toContain('A recommendation is not approval');
|
||
expectMentions(approach, [['without','recomposing','unchanged']], 'approach');
|
||
expect(gate).toMatch(/(?:Record its reference|Save the answer reference) and scope in Exact approval and scope/);
|
||
expect(gate).toMatch(/update Status,? and amend only (?:what it authorizes|authorized work)/);
|
||
expectMentions(gate, [['before','storage','another']], 'gate');
|
||
});
|
||
});
|
||
|
||
// Every skill that renders {{PLAN_FILE_REVIEW_REPORT}} / {{EXIT_PLAN_MODE_GATE}},
|
||
// on every host: the union covers each skill-specific branch of both resolvers.
|
||
function renderPlanReportAndGate(): string {
|
||
const { generatePlanFileReviewReport } = require('../scripts/resolvers/review-dashboard');
|
||
const { generateExitPlanModeGate } = require('../scripts/resolvers/plan-gates');
|
||
const { HOST_PATHS } = require('../scripts/resolvers/types');
|
||
const skills = ['codex', 'devex-review', 'plan-ceo-review', 'plan-design-review', 'plan-devex-review', 'plan-eng-review'];
|
||
return ALL_HOST_CONFIGS.flatMap((host) => skills.map((skillName) => {
|
||
const ctx = { skillName, tmplPath: `${skillName}/SKILL.md.tmpl`, host: host.name, paths: HOST_PATHS[host.name] };
|
||
return `${generatePlanFileReviewReport(ctx)}\n${generateExitPlanModeGate(ctx)}`;
|
||
})).join('\n');
|
||
}
|
||
|
||
// GSTACK REVIEW REPORT report-at-bottom contract — verifies the prompt-text
|
||
// fix in scripts/resolvers/review-dashboard.ts (the load-bearing change for the
|
||
// "report not at bottom of plan in plan mode" bug). The bug is in the
|
||
// prompt's contradictory write-flow instructions, not in observable
|
||
// runtime behavior we can cheaply gate in CI. Verifying the prompt text
|
||
// directly is the deterministic equivalent of the regression test the
|
||
// PTY harness can't reliably drive (autoplan needs auto-progression of
|
||
// AskUserQuestions to reach the report-write step, which the harness
|
||
// doesn't support today).
|
||
describe('GSTACK REVIEW REPORT delete-then-append flow', () => {
|
||
const PLAN_REVIEW_SKILLS = [
|
||
'plan-ceo-review',
|
||
'plan-design-review',
|
||
'plan-devex-review',
|
||
'plan-eng-review',
|
||
];
|
||
|
||
for (const skill of PLAN_REVIEW_SKILLS) {
|
||
test(`${skill}/SKILL.md prescribes delete-then-append, not in-place replace`, () => {
|
||
// Carved skills (v2 plan Phase B) relocate the review-report prose into
|
||
// sections/*.md; readSkillUnion follows the content wherever the carve put it.
|
||
const content = readSkillUnion(skill);
|
||
|
||
// The new (correct) instruction must be present.
|
||
expect(content).toContain('delete-then-append flow');
|
||
|
||
// The old contradictory bullets must be gone. The signature phrase
|
||
// from the buggy prompt was 'replace it entirely using the Edit tool'
|
||
// which is what allowed mid-file reports to stay mid-file.
|
||
expect(content).not.toContain('replace it** entirely using the Edit tool');
|
||
expect(content).not.toContain('If it was found mid-file, move it');
|
||
});
|
||
}
|
||
|
||
test('plan-file review report resolver renders the rewritten flow', () => {
|
||
const src = renderPlanReportAndGate();
|
||
expect(src).toContain('delete-then-append flow');
|
||
// Old contradictory bullets are gone from the source resolver.
|
||
expect(src).not.toContain('replace it** entirely using the Edit tool');
|
||
expect(src).not.toContain('If it was found mid-file, move it');
|
||
});
|
||
});
|
||
|
||
describe('LEARNINGS_SEARCH resolver: query parameter', () => {
|
||
// Lazy-load resolver and types after describe block to keep test file self-contained.
|
||
const { generateLearningsSearch } = require('../scripts/resolvers/learnings');
|
||
const { HOST_PATHS } = require('../scripts/resolvers/types');
|
||
|
||
const claudeCtx = {
|
||
skillName: 'test',
|
||
tmplPath: 'test/SKILL.md.tmpl',
|
||
host: 'claude',
|
||
paths: HOST_PATHS.claude,
|
||
};
|
||
const codexCtx = { ...claudeCtx, host: 'codex', paths: HOST_PATHS.codex };
|
||
|
||
test('no args → bash does not contain --query (backwards-compat)', () => {
|
||
const out = generateLearningsSearch(claudeCtx);
|
||
expect(out).not.toContain('--query');
|
||
});
|
||
|
||
test('claude host + query=foo bar → both cross-project and project-scoped branches contain --query', () => {
|
||
const out = generateLearningsSearch(claudeCtx, ['query=foo bar']);
|
||
// Both branches of the if/else must carry the flag.
|
||
const lines = out.split('\n').filter(l => l.includes('gstack-learnings-search'));
|
||
expect(lines.length).toBeGreaterThanOrEqual(2);
|
||
for (const line of lines) {
|
||
expect(line).toContain('--query "foo bar"');
|
||
}
|
||
});
|
||
|
||
test('codex host + query=foo bar → codex bash variant contains --query', () => {
|
||
const out = generateLearningsSearch(codexCtx, ['query=foo bar']);
|
||
expect(out).toContain('--query "foo bar"');
|
||
expect(out).toContain('$GSTACK_BIN/gstack-learnings-search');
|
||
});
|
||
|
||
test('empty value query= → bash does not contain --query (locked semantics: falls through)', () => {
|
||
const claudeOut = generateLearningsSearch(claudeCtx, ['query=']);
|
||
expect(claudeOut).not.toContain('--query');
|
||
const codexOut = generateLearningsSearch(codexCtx, ['query=']);
|
||
expect(codexOut).not.toContain('--query');
|
||
});
|
||
|
||
test('shell-injection chars in query= → throws at gen-time (defense in depth)', () => {
|
||
for (const bad of ['$(whoami)', '`cmd`', 'a;b', 'a&b', 'a"b', 'a\\b', 'foo$x']) {
|
||
expect(() => generateLearningsSearch(claudeCtx, [`query=${bad}`])).toThrow(/alphanumeric/);
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('EXIT PLAN MODE GATE placement', () => {
|
||
// Fresh skill list — do NOT reuse REVIEW_SKILLS upstream (3 entries, missing plan-devex).
|
||
const planSkills = ['plan-eng-review', 'plan-ceo-review', 'plan-design-review', 'plan-devex-review'];
|
||
|
||
// Strip fenced code blocks before matching headings — PLAN_FILE_REVIEW_REPORT
|
||
// already contains `## GSTACK REVIEW REPORT` inside a markdown example fence,
|
||
// and the gate text itself shows `## GSTACK REVIEW REPORT` inside a fence too.
|
||
const stripFences = (md: string) => md.replace(/```[\s\S]*?```/g, '');
|
||
|
||
test('gate follows review work, with only the declared cache hook after CEO and Eng verification', () => {
|
||
for (const skill of planSkills) {
|
||
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
const stripped = stripFences(md);
|
||
const headings = [...stripped.matchAll(/^## .+$/gm)].map(m => m[0]);
|
||
const lastH2 = headings.at(-1);
|
||
if (['plan-ceo-review', 'plan-eng-review'].includes(skill)) {
|
||
const gate = headings.indexOf('## EXIT PLAN MODE GATE (BLOCKING)');
|
||
expect(gate).toBeGreaterThan(headings.indexOf('## Section self-check (before you finish)'));
|
||
expect(headings.slice(gate)).toEqual(['## EXIT PLAN MODE GATE (BLOCKING)', '## Brain Cache Background Refresh']);
|
||
const tail = md.slice(md.indexOf('## EXIT PLAN MODE GATE (BLOCKING)'));
|
||
const telemetry = tail.indexOf('**Telemetry (run last)**');
|
||
expect(telemetry).toBeGreaterThan(0);
|
||
const cache = tail.indexOf('## Brain Cache Background Refresh');
|
||
if (skill === 'plan-ceo-review') expect(telemetry).toBeGreaterThan(cache);
|
||
else expect(telemetry).toBeLessThan(cache);
|
||
expect(tail).toContain(skill === 'plan-ceo-review' ? 'Passed with a verified persisted report' : 'After the gate passes');
|
||
const finalHandoff = tail.slice(tail.indexOf('## Brain Cache Background Refresh'));
|
||
expect(finalHandoff).toContain(skill === 'plan-ceo-review' ? 'Call ExitPlanMode where required or return to the caller' : 'After success telemetry and cache dispatch, call ExitPlanMode');
|
||
if (skill === 'plan-ceo-review') {
|
||
expect(finalHandoff.indexOf('Call ExitPlanMode')).toBeGreaterThan(finalHandoff.indexOf('**Telemetry (run last)**'));
|
||
expect(finalHandoff).toContain('handoff starts a separate workflow');
|
||
}
|
||
if (skill === 'plan-eng-review') {
|
||
expectMentions(tail.replace(/\s+/g, ' '), [['cannot','persistence','unrecovered']], 'tail.replace(/\s+/g, )');
|
||
expectMentions(finalHandoff, [['only', 'host', 'plan']], 'finalHandoff');
|
||
expectMentions(finalHandoff, [['do not','conversation','exitplanmode']], 'finalHandoff');
|
||
} else {
|
||
expect(tail.replace(/\s+/g, ' ')).toContain('**Pass with log-only gaps:**');
|
||
expectTokens(tail.replace(/\s+/g, ' '), ['**not persisted**'], 'tail.replace(/\s+/g, )');
|
||
expectMentions(tail.replace(/\s+/g, ' '), [['do not', 'unwritten', 'artifacts']], 'tail.replace(/\s+/g, )');
|
||
expectMentions(tail.replace(/\s+/g, ' '), [['without','exitplanmode','telemetry']], 'tail.replace(/\s+/g, )');
|
||
}
|
||
} else {
|
||
expect(lastH2, `${skill}/SKILL.md last ## heading (fences stripped)`).toBe('## EXIT PLAN MODE GATE (BLOCKING)');
|
||
}
|
||
if (skill === 'plan-eng-review') {
|
||
const gate = extractMarkdownSection(md, '## EXIT PLAN MODE GATE (BLOCKING)').replace(/\s+/g, ' ');
|
||
expectTokens(gate, ['**Blocked outcome**'], 'gate');
|
||
expectMentions(gate, [['without', 'exitplanmode', 'telemetry']], 'gate');
|
||
} else if (skill === 'plan-ceo-review') {
|
||
expect(md, `${skill}/SKILL.md gate body`).toContain('Failed checks use **Gate outcome: Blocked**');
|
||
} else {
|
||
expect(extractMarkdownSection(md, '## EXIT PLAN MODE GATE (BLOCKING)'), `${skill}/SKILL.md gate body`).toMatch(/do not call ExitPlanMode/i);
|
||
}
|
||
}
|
||
});
|
||
|
||
test('codex/SKILL.md contains gate (mid-file per D5; Step 2B/2C follow)', () => {
|
||
const codex = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
|
||
expect(codex).toContain('## EXIT PLAN MODE GATE (BLOCKING)');
|
||
expect(extractMarkdownSection(codex, '## EXIT PLAN MODE GATE (BLOCKING)')).toMatch(/do not call ExitPlanMode/i);
|
||
});
|
||
});
|
||
|
||
describe('scope-gate exceptions drift-guard', () => {
|
||
// The plan-mode auto-select-B exceptions block is hand-duplicated in the
|
||
// plan-eng-review and plan-design-review templates (matching the gate
|
||
// around it, which predates this block). The two copies must stay
|
||
// identical in their target-selection policy. Eng resolves that policy
|
||
// before announcing its target and has a separate bootstrap transport;
|
||
// Design retains its own startup sequence and page vocabulary.
|
||
// A future edit to one copy that silently misses the other fails here
|
||
// instead of drifting. The real fix (shared {{SCOPE_GATE}} resolver) is a
|
||
// filed TODO — this guard is the stopgap that makes the duplication safe.
|
||
const START_MARKER = '**Exceptions — check in this order, BEFORE asking:**';
|
||
const END_MARKER = 'When no exception above applied:';
|
||
|
||
function extractExceptionsBlock(skill: string): string {
|
||
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
const start = md.indexOf(START_MARKER);
|
||
expect(start, `${skill}/SKILL.md: exceptions block start marker present`).toBeGreaterThan(-1);
|
||
const end = md.indexOf(END_MARKER, start);
|
||
expect(end, `${skill}/SKILL.md: exceptions block end marker present`).toBeGreaterThan(start);
|
||
return md.slice(start, end + END_MARKER.length);
|
||
}
|
||
|
||
const normalizeTargetPolicy = (block: string) => block
|
||
.split('\n').filter(line => /^[12]\. /.test(line)).join('\n')
|
||
.replace(/ Announce (?:it|an auto-selected plan) in one line so the user can interrupt: "Scope gate: plan mode — auto-selected B \(reviewing <target>\)\."/g, '')
|
||
.replace(/ Then run the pre-review audit, .*? against that plan\./, '')
|
||
.replace('a path, a page, a doc they pasted,', 'a path, a doc they pasted,');
|
||
|
||
test('eng and design retain the same target-selection policy across distinct startup sequences', () => {
|
||
const engRaw = extractExceptionsBlock('plan-eng-review');
|
||
const eng = normalizeTargetPolicy(engRaw);
|
||
const design = normalizeTargetPolicy(extractExceptionsBlock('plan-design-review'));
|
||
expect(eng).toBe(design);
|
||
expect(eng).toContain('their choice wins — use it instead');
|
||
expect(eng).toContain('still ambiguous — ask');
|
||
expectMentions(eng, [['ask','default','doubt']], 'eng');
|
||
expect(engRaw.indexOf('their choice wins')).toBeLessThan(engRaw.indexOf('Announce an auto-selected plan'));
|
||
const engMd = fs.readFileSync(path.join(ROOT, 'plan-eng-review/SKILL.md'), 'utf8');
|
||
const transport = engMd.indexOf('Choose listed, enabled MCP AskUserQuestion, otherwise listed native');
|
||
expect(transport).toBeGreaterThan(engMd.indexOf(END_MARKER));
|
||
expect(transport).toBeLessThan(engMd.indexOf('## Preamble (after scope gate)'));
|
||
});
|
||
|
||
test('exceptions block carries the announcement string the PTY detectors pin', () => {
|
||
for (const skill of ['plan-eng-review', 'plan-design-review']) {
|
||
const block = extractExceptionsBlock(skill);
|
||
expect(block, `${skill}: verbatim announcement`).toContain(
|
||
'Scope gate: plan mode — auto-selected B (reviewing <target>).',
|
||
);
|
||
}
|
||
});
|
||
|
||
test('gate menu carries the question strings the PTY question detector pins', () => {
|
||
// isScopeGateQuestionVisible (claude-pty-runner.ts) anchors on the
|
||
// question text + option A's body. If the menu is reworded without
|
||
// updating the detector, the paid smokes' must-stay-false assertions go
|
||
// vacuous — this free pin fails first.
|
||
for (const skill of ['plan-eng-review', 'plan-design-review']) {
|
||
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
expect(md, `${skill}: gate question text`).toContain('What should I review?');
|
||
expect(md, `${skill}: option A body text`).toContain('The current branch diff');
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('GSTACK REVIEW REPORT mandatory unresolved-decisions status', () => {
|
||
// Report text rides in PLAN_FILE_REVIEW_REPORT → every report consumer gets it.
|
||
// devex-review is a report consumer but NOT a gate consumer, so the two target
|
||
// sets differ (CP5/CX5). Regression guard: a future token-cut that drops the
|
||
// unresolved-status line again fails here. See plan-flag-unresolved-issues.
|
||
const REPORT_CONSUMERS = [
|
||
'plan-ceo-review',
|
||
'plan-eng-review',
|
||
'plan-design-review',
|
||
'plan-devex-review',
|
||
'codex',
|
||
'devex-review',
|
||
];
|
||
// Gate text rides in EXIT_PLAN_MODE_GATE (lives in SKILL.md, not sections).
|
||
const GATE_SKILLS = [
|
||
'plan-ceo-review',
|
||
'plan-eng-review',
|
||
'plan-design-review',
|
||
'plan-devex-review',
|
||
'codex',
|
||
];
|
||
|
||
for (const skill of REPORT_CONSUMERS) {
|
||
test(`${skill}: report mandates the unresolved-decisions status as final content`, () => {
|
||
const content = readSkillUnion(skill);
|
||
expect(content).toContain('NO UNRESOLVED DECISIONS');
|
||
// \s+ tolerates prose line-wraps within "final non-whitespace line".
|
||
expect(content).toMatch(/final\s+non-whitespace\s+line/i);
|
||
});
|
||
}
|
||
|
||
for (const skill of GATE_SKILLS) {
|
||
test(`${skill}: exit gate blocks unless the unresolved status is the final line`, () => {
|
||
const md = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||
// Gate check #4 — present, sentinel named, and explicitly blocking (no escape).
|
||
expect(md).toContain('NO UNRESOLVED DECISIONS');
|
||
if (['plan-ceo-review', 'plan-eng-review'].includes(skill)) {
|
||
const gate = md.split('## EXIT PLAN MODE GATE (BLOCKING)')[1]!.replace(/\s+/g, ' ');
|
||
expect(gate).toMatch(/final\s+non-whitespace\s+line is the exact unbolded `NO UNRESOLVED DECISIONS`/i);
|
||
expect(gate).toContain('`**UNRESOLVED DECISIONS:**`');
|
||
expect(gate).toMatch(/missing\s+status[^.]*fails/i);
|
||
expect(gate).toContain(skill === 'plan-eng-review'
|
||
? 'If any check fails, follow **Blocked outcome** without success telemetry or ExitPlanMode'
|
||
: 'Failed checks use **Gate outcome: Blocked**');
|
||
} else {
|
||
const gate = extractMarkdownSection(md, '## EXIT PLAN MODE GATE (BLOCKING)');
|
||
expect(gate).toMatch(/final\s+non-whitespace\s+line/i);
|
||
expect(gate).toContain('`**UNRESOLVED DECISIONS:**`');
|
||
expect(gate).toMatch(/missing\s+status[^.]*fails/i);
|
||
}
|
||
});
|
||
}
|
||
|
||
test('plan-file review report and exit gate resolvers render the mandatory block + blocking gate', () => {
|
||
const src = renderPlanReportAndGate();
|
||
expect(src).toContain('NO UNRESOLVED DECISIONS');
|
||
expect(src).toMatch(/avoids double-counting/i);
|
||
expect(src).toMatch(/final\s+non-whitespace\s+line/i);
|
||
expect(src).toMatch(/missing\s+status[^.]*fails/i);
|
||
// The old soft wording must be gone from the gate.
|
||
expect(src).not.toContain('absorbs CODEX / CROSS-MODEL / UNRESOLVED lines if applicable');
|
||
});
|
||
});
|
||
|
||
// ─── {{PREAMBLE}} requires an explicit preamble-tier ────────
|
||
|
||
describe('PREAMBLE resolution requires declared preamble-tier', () => {
|
||
test('resolving {{PREAMBLE}} without preamble-tier throws with the template path', async () => {
|
||
const { generatePreamble } = await import('../scripts/resolvers/preamble');
|
||
const { HOST_PATHS } = await import('../scripts/resolvers/types');
|
||
const ctx = {
|
||
skillName: 'tierless-skill',
|
||
tmplPath: 'tierless-skill/SKILL.md.tmpl',
|
||
host: 'claude' as const,
|
||
paths: HOST_PATHS.claude,
|
||
// preambleTier deliberately absent — the generator must refuse to default it.
|
||
};
|
||
expect(() => generatePreamble(ctx)).toThrow(/tierless-skill\/SKILL\.md\.tmpl/);
|
||
expect(() => generatePreamble(ctx)).toThrow(/preamble-tier/);
|
||
});
|
||
|
||
test('every template that resolves {{PREAMBLE}} declares preamble-tier in frontmatter', () => {
|
||
const entries = fs.readdirSync(ROOT, { withFileTypes: true });
|
||
const offenders: string[] = [];
|
||
const checkTmpl = (tmplPath: string) => {
|
||
const tmpl = fs.readFileSync(tmplPath, 'utf-8');
|
||
if (tmpl.includes('{{PREAMBLE}}') && !/^preamble-tier:\s*\d+$/m.test(tmpl)) {
|
||
offenders.push(path.relative(ROOT, tmplPath));
|
||
}
|
||
};
|
||
checkTmpl(path.join(ROOT, 'SKILL.md.tmpl'));
|
||
for (const e of entries) {
|
||
if (!e.isDirectory() || e.name.startsWith('.') || e.name === 'node_modules') continue;
|
||
const tmplPath = path.join(ROOT, e.name, 'SKILL.md.tmpl');
|
||
if (fs.existsSync(tmplPath)) checkTmpl(tmplPath);
|
||
}
|
||
expect(offenders).toEqual([]);
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// #2499: gbrain MCP detection must read BOTH ~/.claude.json scopes.
|
||
// Claude Code registers MCP servers at user scope (.mcpServers) and project
|
||
// scope (.projects["/abs/path"].mcpServers — what `claude mcp add` without
|
||
// --scope user writes). The rendered brain-sync block previously read only
|
||
// user scope, so a correctly configured project-scoped brain was invisible.
|
||
// ---------------------------------------------------------------------------
|
||
describe('brain-sync block reads project-scoped MCP registrations (#2499)', () => {
|
||
// Phase 1: the artifacts-sync bash (including the MCP-scope jq probe) moved
|
||
// from the rendered SKILL.md into bin/gstack-skill-start. Pin the LIVE
|
||
// script bytes — same assertions, new home. The render carries only the
|
||
// ARTIFACTS_SYNC interpretation prose.
|
||
const rendered = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
|
||
|
||
test('rendered _GBRAIN_MCP_ENTRY jq resolves project scope with nearest-ancestor cwd match', () => {
|
||
const line = rendered.split('\n').find((l) => l.includes('_GBRAIN_MCP_ENTRY=$('));
|
||
expect(line).toBeDefined();
|
||
// Project-scope read present, driven by $PWD.
|
||
expect(line!).toContain('--arg cwd "$PWD"');
|
||
expect(line!).toContain('.projects');
|
||
// User scope still resolved first.
|
||
expect(line!).toContain('.mcpServers.gbrain');
|
||
// The old user-scope-only filter is gone from the rendered output.
|
||
expect(rendered).not.toContain('.mcpServers.gbrain.type // .mcpServers.gbrain.transport');
|
||
expect(rendered).not.toContain(".mcpServers.gbrain.url // empty");
|
||
});
|
||
|
||
test('rendered _GBRAIN_MCP_TYPE and _GBRAIN_HOST extract from the resolved entry', () => {
|
||
const typeLine = rendered.split('\n').find((l) => l.includes('_GBRAIN_MCP_TYPE=$('));
|
||
const hostLine = rendered.split('\n').find((l) => l.includes('_GBRAIN_HOST=$('));
|
||
expect(typeLine).toBeDefined();
|
||
expect(hostLine).toBeDefined();
|
||
expect(typeLine!).toContain('_GBRAIN_MCP_ENTRY');
|
||
expect(hostLine!).toContain('_GBRAIN_MCP_ENTRY');
|
||
});
|
||
|
||
test('rendered jq lines FUNCTION: project-scoped registration resolves for a cwd inside the project', () => {
|
||
// Execute the exact rendered bytes, not a re-derivation: extract the
|
||
// _GBRAIN_MCP_ENTRY + _GBRAIN_MCP_TYPE lines from the generated SKILL.md
|
||
// and run them in bash against a fixture ~/.claude.json that carries ONLY
|
||
// a project-scoped gbrain registration.
|
||
const lines = rendered.split('\n');
|
||
const entryLine = lines.find((l) => l.includes('_GBRAIN_MCP_ENTRY=$('));
|
||
const typeLine = lines.find((l) => l.includes('_GBRAIN_MCP_TYPE=$('));
|
||
expect(entryLine).toBeDefined();
|
||
expect(typeLine).toBeDefined();
|
||
|
||
const tmpHome = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-2499-home-'));
|
||
const projectDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-2499-proj-'));
|
||
const nestedCwd = path.join(projectDir, 'src', 'deep');
|
||
fs.mkdirSync(nestedCwd, { recursive: true });
|
||
try {
|
||
fs.writeFileSync(
|
||
path.join(tmpHome, '.claude.json'),
|
||
JSON.stringify({
|
||
projects: {
|
||
[projectDir]: {
|
||
mcpServers: { gbrain: { type: 'http', url: 'https://brain.example.com/mcp' } },
|
||
},
|
||
},
|
||
}),
|
||
);
|
||
const script = `cd "$1" || exit 1\n${entryLine!.trim()}\n${typeLine!.trim()}\necho "RESOLVED:$_GBRAIN_MCP_TYPE"`;
|
||
const r = spawnSync('bash', ['-c', script, 'bash', nestedCwd], {
|
||
encoding: 'utf-8',
|
||
env: { ...process.env, HOME: tmpHome },
|
||
timeout: 10_000,
|
||
});
|
||
expect(r.stdout).toContain('RESOLVED:http');
|
||
|
||
// Discriminator: a cwd OUTSIDE the project must NOT resolve it.
|
||
const outside = spawnSync('bash', ['-c', script, 'bash', os.tmpdir()], {
|
||
encoding: 'utf-8',
|
||
env: { ...process.env, HOME: tmpHome },
|
||
timeout: 10_000,
|
||
});
|
||
expect(outside.stdout).toContain('RESOLVED:\n');
|
||
} finally {
|
||
fs.rmSync(tmpHome, { recursive: true, force: true });
|
||
fs.rmSync(projectDir, { recursive: true, force: true });
|
||
}
|
||
});
|
||
});
|
||
|
||
// #2896: Claude Code replaces `$<digit>` tokens in a SKILL.md body with the
|
||
// invocation's argument words before the model reads it (2.1.271+:
|
||
// /\$(\d+)(?!\w)/g), so shell/awk fields arrive rewritten yet still parse.
|
||
// Write awk fields as $(N) and shell positionals as ${N}. Section files are
|
||
// read later with Read and are not substituted.
|
||
describe('generated Claude skill bodies carry no $N token Claude Code rewrites (#2896)', () => {
|
||
const CLAUDE_ARG_TOKEN = /\$\d+(?!\w)/g;
|
||
test('tripwire', () => {
|
||
const offenders: string[] = [];
|
||
let checked = 0;
|
||
for (const file of ['SKILL.md', ...fs.readdirSync(ROOT).map(d => path.join(d, 'SKILL.md'))]) {
|
||
const full = path.join(ROOT, file);
|
||
if (!fs.existsSync(full)) continue;
|
||
const lines = fs.readFileSync(full, 'utf-8').split('\n');
|
||
const bodyStart = lines[0] === '---' ? lines.indexOf('---', 1) + 1 : 0;
|
||
checked++;
|
||
lines.slice(bodyStart).forEach((line, i) => {
|
||
for (const m of line.matchAll(CLAUDE_ARG_TOKEN)) offenders.push(`${file}:${bodyStart + i + 1}: ${m[0]} in ${line.trim().slice(0, 120)}`);
|
||
});
|
||
}
|
||
expect(checked).toBeGreaterThan(40);
|
||
expect(offenders).toEqual([]);
|
||
});
|
||
|
||
test('the tripwire pattern matches what Claude Code substitutes', () => {
|
||
expect('awk \'{print $1}\' "$0" ~$0.05 $12'.match(CLAUDE_ARG_TOKEN)).toEqual(['$1', '$0', '$0', '$12']);
|
||
expect('awk \'{print $(1)}\' "${1}" $ARGUMENTS $1a $_x'.match(CLAUDE_ARG_TOKEN)).toBeNull();
|
||
});
|
||
});
|