Files
gstack/test/skill-validation.test.ts
T
Garry Tanandcapy-ai[bot] 14d7414af3 v1.93.33.0 wave(review): #3705 #3744 #3745 #3764 #3765 #3795 (#3852)
* fix(ship): base merges, adjacent base edits, test-only and audit-requested fixes no longer force a second review loop (GST-407, GST-384, GST-391)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ship driver: trim two phrases for headroom with #3662

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ship driver: keep the Next steps heading phrase the handoff check reads; trim elsewhere

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: regenerate ship goldens

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes

comparePath returns same only when every hunk sits where the base change puts it (patch-ids ignore positions) and treats a file-mode change as changed; the scoped-review packet says when a fence is truncated; --docs-result must be the posted audit's result; the --context-reviewed record goes beside the packet, outside the worktree. Negative controls for each.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pass-2 review fixes

A test edit routes as tests= only when it adds lines with no gate tag; a removed or changed line or an unreadable test diff goes through the full loop. Diffs skip textconv. A hunkless patch (binary, empty, mode-only) is same when its patch-id matches. Without --reviewed-base test files route as before. finish's packet path and attempt's state directory have fallbacks.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: Step 11 review fixes; a version-only stamp_files edit after review is release metadata (GBRA-251)

gateEditedPaths reads removed lines only inside hunks; carry refuses when a path listing fails; docs freshness stops on a failed listing; attempt counts an unreadable state file. releasePaths includes stamp_files paths whose edit is a version swap under stamp_patterns.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: no test edit skips review; guard packet writes; exclusive attempt state

A test-only edit with no gate tag gets the scoped fix-check of its test hunks instead of a lanes-only rerun. Packet write failures print a STOP or RECEIPT_CARRY_REFUSED. attempt's allowance uses an exclusive create.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: attempt prints gstack's install command; carry checks the product diff outside context paths

The attempt STOP no longer echoes the audit child's fix text. A scoped-review carry now requires the product patch-id without the context paths to match at both heads.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: oversized packets take the full loop; exact --docs-requested match; name-only test globs only in test dirs

A scoped-review packet section over 200 lines sends that path to the full loop instead of a truncated packet. The docs-result path match is exact. test_*.py, *Test.java and similar count as test code only under a test directory.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(unattended): carry's scoped-review refusal and what it keeps

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: carry's scoped-review route in ship-policy, review-policy and troubleshooting (docs-a-1)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(windows): curate ship-rereview-triggers out of the Windows lane like ship-release (run 41193336209)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(review-log): record a notes-only /ship pass clean; name the cause and repair when a pass's tree moved (GST-430)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(receipt): carry and write handle PRs with thousands of files; tree receipt fails closed on a failed diff (GST-448)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ship): preflight routes RECEIPT_CARRY_SCOPED_REVIEW to the scoped review; carry --dry-run writes no packet (GST-442)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(ship): Step 1.4 and Step 16 route the scoped-review refusal and the --docs-requested fix; trims keep the cap (GST-442, GST-444)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: /ship review pass 1: notes-only floor, scoped carry order, named temp and diff refusals

- A notes-only pass is recorded clean only when every finding is fixed, or noted or deferred below the
  blocking floor (never CRITICAL or blocking), by its logger-stamped action, and the findings account
  for the claimed issue and critical counts (GST-430).
- The preflight scoped carry route commits and pushes the base-merge fix before writing the packet (GST-442).
- patchId refuses by name on an unwritable TMPDIR; the tree receipt names why its diff failed (GST-448).
- Step 16 stage 3 spells out --docs-requested with its --docs-result (GST-444).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: pin the Step 16 --docs-requested route with its --docs-result

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: a claimed clean review row with a logger-stamped open gate finding reads UNVERIFIED (red-team, pass 1)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: Step 11 findings: gate rows from --findings are stamped, one open rule, stale patch-id dirs swept

- deriveFromFindings stamps gate rows with the session kind before deriving status, so a findings file cannot
  dispose of a gate finding the unattended logger forces open (outside-voice#1).
- reviewFreshness reads a claimed row's findings with isOpenByAction, the rule the notes-only rewrite uses;
  noted and deferred clear a finding only when it is marked blocking: false and is not CRITICAL.
- patchId sweeps gstack-patch-id-* directories over an hour old that a killed carry left in TMPDIR.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: Step 11 round 2: stamped-open gates win over status and disposition; resolved gate, skipped and below-floor actions stay CURRENT

- isOpenFinding treats action 'open' (the logger's forced-open stamp) as open before a row's status or disposition.
- reviewFreshness flags a claimed clean row only for an 'open' action or a CRITICAL or blocking deferral or note, so
  interactive kept and restored gates, skipped advisories and INFORMATIONAL deferrals read CURRENT as before.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: carry dry run offers the scoped route last; patch-id diff under the git dir; changelog matches the review-log floor

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: a claimed clean review row with any unresolved non-gate CRITICAL finding reads UNVERIFIED (pass 2 red-team)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: carry dry-run scoped route, patch-id directory, tree-receipt refusal (docs-d-1 audit)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ship): Step 3 names merge conflicts in its STOP sentence again (ship-workflow-clarity pin)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(tree-receipt): keep the gated-to-current diff at the file helper timeout (60 s); pregate literals check

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(carry): large-diff fixture sets core.longpaths so Git for Windows can add its long receipt paths (windows-free-shard 1, run 41210080055)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(review): a stale local base branch yields to its origin twin in gate-diff and diff-scope (GST-462)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ship): Step 11 saves the targeted record and the receipt writes carry it; Step 9 clears it (GST-464)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: GST-464 and GST-462

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(review): pass 1: ahead and diverged labels, diff-scope needs shared history; preflight refreshes origin/<base> and stops on a diff far beyond the PR (GST-462, HARN-15)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pass-2 review fixes for base twin, prSized stop and targeted record

Resolve the tracking twin by refs/remotes/<remote>/<ref> and only for a bare
branch name, keep a local base that is ahead or diverged, quote the prSized
stop's base and name the fork case, and have every Step 9 entry create,
exclude and clear .gstack/tmp/targeted.json.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: origin's tracking base wins whenever related; prSized lower bound

The fix-check found that keeping a local base ahead of origin left unpushed
base commits out of the gate scan while review-plan listed them. The tracking
branch now wins whenever it shares history with HEAD, an explicit origin/main
reads refs/remotes/origin/main, the preflight also stops when the diff lists
fewer files than the PR, and the Step 9 exclude line survives a missing
.git/info and an exclude file without a final newline.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: the tracking-ref claim holds while the tracking branch exists

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: repin the two ship section briefs under no-loss replay GST-464

GST-464 brief-identity check: every reviewer brief lib/review-brief.ts cuts
from ship/sections/review-army.md and adversarial.md stays byte-identical to
the bodies the L and GST-257 replays admitted; only lead instructions outside
the cut spans changed. The check is a test CI runs.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ship: keep Step 9's PASS_HEAD sentence whole; repin review-army under GST-464

The targeted-record line follows the PASS_HEAD/RANGE_BASE sentence instead of
splitting it (test/review-fix-range.test.ts pins that sentence). The brief-body
check still holds: every reviewer brief is byte-identical.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: restore bin/gstack-ship-receipt after the main merge

The merge commit b10e2db wrote a truncated copy of the file. This is the
three-way merge of both sides with the refused() union: main's codes plus
RECEIPT_CARRY_SCOPED_REVIEW.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review-collect: narrow the consumer scan and give it one 300 s deadline (GST-544)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review-collect: keep the consumer scan deadline under the 180 s plan timeout (review fix)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review-collect: plain paths under --literal-pathspecs; whole-millisecond timeout override (review fix)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review-collect: whole-tree grep for non-UTF-8 names, no argument spread, name spawn errors (review fix)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review-collect: cap the consumer grep output at 256 MB across calls; per-scan deadline wording (review fix, cycle 4)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review-collect: read a missing spawn status as null (Bun 1.4.2 returns undefined on ENOENT)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: review-collect whole-tree helper reads a null status as -1 (typecheck:test ratchet)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review-bots: /review and /ship triage every configured review bot, not only Greptile (#3122)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes (gh before 2.48 pages without --slurp; driver routes Step 10; tests for guards, review-bot fence, REMOTE_SLUG owner)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: drop the review-bot fence test that could not fail (the tool-fence WARN degrade is unreachable: placeholder spans are skipped first)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: Step 11 review fixes (record by comment id so a filename never reaches a shell; lookup failures are errors; tojson pages; longest bot prefix wins)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pass-2 review fixes (record refuses * paths, colliding ids and a missing fetch dir; parent requires id and dir; botOwning doc)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review: classify the PR type and always check repo conventions (#356)

The checklist now opens with a PR-type line (application, CI/infra,
scripts/tooling, docs-only or mixed). Application and mixed diffs run
every category; other types always run five categories, including the
new Repo Conventions & Platform Fit, and report the rest as not
applicable in one line instead of skipping silently.

No-loss replay: GSTA-224 H1 preregistered no-loss run (run id
gsta224-h1-20261031, Opus 5.5, 81 calls, $9.53) against main's checklist
with a no-gstack arm; every preregistered bar passed. The
review/checklist.md pin in test/review-packets-removed.test.ts moves to
a32963ef with replay 'GSTA-224 H1'.

Refs GSTA-224

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review-bots: allowlist the record placeholders and match Step 10's id rule in tests

The record fence uses <comment-id>, <fix|fp|already-fixed> and <category>,
each with an enum or numeric grammar; <greptile-dir> becomes <bot-dir>.
review/greptile-triage.md also lists every REVIEW_BOTS skip reason.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: review-bot triage covers every configured bot (#3122)

README, CONTRIBUTING, the contributor reference and standalone-binaries
name gstack-review-bots and the review-bot fence; TODOS drops the #3122
item; the changelog fragment covers record-by-id and lookup errors.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: the PR-type test also pins the compose, Terraform and README/CHANGELOG phrases

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: follow the review-bot rename in the remote free tier's three reds

The cross-skill consistency judge describes review-bot-history.md and its
record helper; the tracker-guard exemption for the removed raw Greptile
fetch in review/greptile-triage.md is dropped (the fetch now lives in
gstack-review-bots, whose bodies still reach context only through
gstack-issue-guard); the Step 10 PR-check test expects 'Review bots: not
run'. HARNESS_VERSION stays 6: skill-fixture.ts and llm-judge.ts are
recorded non-behavioral for every other case.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep the cross-skill judge's rubric to one renamed bullet

Name review-bot-history.md beside the older greptile-history.md so the
calibration corpus still matches, and keep the REMOTE_SLUG bullet. One
live 3-sample panel at this tree: consistent 3/3, scores 4/4/4.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: H1 fragment passes prose lint

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: name the H1 fragment after its branch; TODOS drops #356

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: /review's PR type step in docs/skills.md (#356)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ship-policy: review_exclude_paths key and loadPolicyAt (GST-508)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* review: apply the base policy's review_exclude_paths and the reviewer diff ceiling (GST-508)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* receipt: bind review_exclude in write, carry and the queue (GST-508)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs, tests and changelog for review_exclude_paths (GST-508)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes (binary by bytes, links kept, untracked files out of the selection)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: keep binary files at code paths under review_exclude_paths

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: name the reasons that keep a binary file under review_exclude_paths

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: keep every review_exclude_paths candidate when the gate scan is unknown; docs example names its version

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep the NUL fixture clear of the sync-spawn tripwire

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: split the ceiling sentence

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: keep a live binary file whose bytes cannot be read; tests render outside voices from a clean checkout and read the ship fixture text

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(review-exclude): explicit +x for the census script; skip the chmod-000 read case on Windows and root

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* refactor: move loadPolicyAt to lib/ship-policy-at.ts so lib/ship-policy.ts stays under the 800-line module cap

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: refuse a base that contains HEAD; skip a self-pointing origin/HEAD (GST-774)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: regenerate gstack-doctor-components.sh for BASE_CONTAINS_HEAD

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: land resumes a merged merge-only PR instead of refusing BASE_CONTAINS_HEAD (GST-774 review)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: changelog fragment for GST-774

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: troubleshooting names diff-scope base_contains_head and the self-pointing no_base case (GST-774 docs audit)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: repin the reviewer briefs to GST-508's no-loss-tested bytes

ship/sections/review-army.md, ship/sections/adversarial.md and lib/review-depth.ts now pin
the bytes the GST-508 preregistered no-loss replay ran at ef2760f (handoffs/GSTA-202/noloss-results.md
in the GStack Minder drive): Arm V 33/41 vs R 33/41, delta 4, exact sign flip, p_NI 0.0279 (pass).
The merge of main 83ae96e leaves these three files byte-identical to ef2760f, and an Arm P unit
(3593) renders byte-identical briefs at ef2760f and at the merge.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: GST-774 review round 1 — merged resume depth, aliased and detached PR checkouts, develop/trunk defaults

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ship: Step 20 says a metrics row, keeping DRIVER.md at 12 KB after the main merge

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: ship-parts resend case leaves room for the first phase-3 part after the Step 9 targeted-record sentence

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: /ship delivers review/checklist.md in two parts so checklist v1 fits the 24576-byte part cap (#3670)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: drop the GST-407 and GST-430 fragments, already released on main (stacked history)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* wave review-53cfb5bfe011: lib/ship-policy.ts back to 800 lines (module-size ratchet after merging #3744 and #3795)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* wave review-53cfb5bfe011: /ship pass 1 fixes

- lib/ship-receipt-depth.ts: a merged resume (land after a true merge) no longer recomputes the review_exclude digest over the emptied range, so it lands instead of RECEIPT_INVALID (failing-first land test with review_exclude_paths active).
- bin/gstack-review-dispatch render: a standalone targeted render under an active review_exclude_paths refuses and points at the pass manifest voices stage, instead of writing a brief with no selection that leaves verify dirty (failing-first test).
- test/default-base.test.ts: #3795 expectations follow main R51 depth (light for an 80-line code diff; runtime_lines=80 still proves the real base).
- test/review-dispatch.test.ts: the GST-897 fingerprint test renders from the clean checkout, like its neighbor, because CI rebuilds the tracked lib/diagram-render/dist/diagram-render.html and the reviewer diff ceiling refused the ROOT render.

* wave review-53cfb5bfe011: /ship pass 2 fixes

- lib/review-exclude.ts: binaryContent reads each blob's first 8,000 bytes (git cat-file blob, 8 KB buffer) instead of one --batch read capped at 256 MB, so head mode (receipt, carry, queue) and the plan agree past 256 MB of blobs; an unreadable blob is kept, as in live mode. Failing-first test with two 270 MB blobs.
- lib/review-bots.ts: a file-level bot review comment (subject_type file, no line position) is triaged instead of read as outdated. Failing-first test.
- test/ship-queue-land.test.ts: the merged-resume exclusion test runs its gate scan under the fixture state root, not the real home (CI free-suite 6 home containment).

* fix: merge follow-ups for main 3d99c22: origin/* stays qualified in diff-scope's twin step, retro skeleton back under its cap, combined no-loss pin GST-508+R51+GST-911

* fix(review): receipt write gate-scans whenever the base declares review_exclude_paths; collectLive never re-qualifies origin/<base> (#3852 /ship red-team#1, outside-voice#1)

* test: the docs-only receipt digest test keeps its gate scan out of the real home (free-suite shard 5 FREE_SUBSHARD_REJECTED)

* fix(merge fc292e22f): tentative REVIEW_EXCLUDE_KEY_VERSION, classify targeted.json, render main's new outside-voice test from a clean checkout

- lib/ship-policy.ts: REVIEW_EXCLUDE_KEY_VERSION 1.93.0.0 -> 1.94.0.0, the next-minor tentative form stamp_paths moves (docs/ship-policy.md, #3512); 1.93.0.0 is a released heading that never names the key.
- test/gstack-clean-producers.test.ts (new from main): KEPT gains targeted.json, the wave's --review-targeted input.
- test/review-dispatch.test.ts: main's wrong-tree test rendered the outside voice from ROOT; a CI build that dirties ROOT pushes that diff past the wave's REVIEW_DIFF_MAX_BYTES ceiling, so render refused and verify saw no brief (exit 3). It now runs in cleanCheckout like the wave's other outside-voice tests.

* test(merge 68410828f): GST-967's brief-body pin carries the wave's GST-508 placeholders

The merged sectionTexts differ from main's by exactly the wave's GST-508 exclude-pathspec lines (merged minus main equals wave minus fc292e22f); the 1,440-case plan grid still hashes to main's MAIN_PLANS. Checks: GStack Minder drive handoffs/GSTA-202/merge-6841-checks.md.

* fix(merge e85005482): targeted classes since the read accept an empty range; the wave's receipt tests pass main's targeted record

- lib/ship-receipt-depth.ts targetedClassesSince (main, GST-986) reads the range from the commit the targeted reader read to HEAD through headInputs, which since #3795 runs gstack-diff-scope's BASE_CONTAINS_HEAD check. When the reader read HEAD itself the range is empty, diff-scope reported base_contains_head, and every write with --review-targeted was refused (the product patch matched). It now passes merged=true (--merged), which skips only that check; an ancestor check already guards the range.
- test/review-exclude.test.ts and test/review-targeted.test.ts: since #3887 write refuses a money head with no targeted record and requires the record's head, so the wave's two writes that must succeed pass one, as main's own tests do (targetedAtHead in test/ship-receipt-depth.test.ts).

* test(merge e85005482): drop the wave's duplicate targeted.json KEPT entry

Main (#3887) classified .gstack/tmp/targeted.json in KEPT too; the auto-merge kept both keys, which typecheck:test rejects (TS1117). Main's entry stays.

* test(merge e85005482): ship size ledger entry for the merged tree (-232 B)

Main's ship size ledger plus the wave's entries summed to 345,334 B; the merged tree measures 345,102 B because DRIVER.md takes main's shorter Step 19 line. Recorded with scripts/size-ledger.ts record ship.

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-03 01:40:07 +00:00

2243 lines
96 KiB
TypeScript

import { describe, test, expect, afterAll } from 'bun:test';
import { validateSkill, extractRemoteSlugPatterns, extractWeightsFromTable } from './helpers/skill-parser';
import { ALL_COMMANDS, COMMAND_DESCRIPTIONS, READ_COMMANDS, WRITE_COMMANDS, META_COMMANDS } from '../browse/src/commands';
import { SNAPSHOT_FLAGS } from '../browse/src/snapshot';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { usePrivateStateRoot } from './helpers/private-state-root';
import { expectMentions } from './helpers/prompt-structure';
import { generateFailureRouting } from '../scripts/resolvers/preamble/generate-test-failure-triage';
import { readShippedSkillRouting } from './helpers/shipped-skill-routing';
import { readSkillStartup } from './helpers/skill-startup';
const ROOT = path.resolve(import.meta.dir, '..');
// ─── Codex-host render isolation ─────────────────────────────
// .agents/ is gitignored and regenerated. This file used to regenerate it IN
// PLACE at three sites (a tree-mutating hazard for concurrent readers).
// Render the codex host ONCE into a module-level out-dir instead; every
// codex-artifact assertion reads from here. Out-dir renders are byte-
// identical to in-place external-host renders (pinned by
// test/gen-skill-docs-out-dir.test.ts).
const CODEX_OUT = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-skillval-codex-'));
{
const render = Bun.spawnSync(
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--out-dir', CODEX_OUT],
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 },
);
if (render.exitCode !== 0) {
throw new Error(
`gen-skill-docs --host codex --out-dir failed (exit ${render.exitCode}):\n${render.stderr.toString()}`,
);
}
}
afterAll(() => {
fs.rmSync(CODEX_OUT, { recursive: true, force: true });
});
// Carved-skill aware (v2 plan T9 / Phase B): a carved skill is a skeleton SKILL.md
// plus sections/*.md. Read the union so validations of content that moved into a
// section still hold. For an uncarved skill (no sections dir) this is just the
// skeleton, so readSkillUnion is safe to use everywhere.
function readSkillUnion(skill: string): string {
let t = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
const secDir = path.join(ROOT, skill, 'sections');
if (fs.existsSync(secDir)) {
for (const f of fs.readdirSync(secDir).sort()) {
if (f.endsWith('.md')) t += '\n' + fs.readFileSync(path.join(secDir, f), 'utf-8');
}
}
return t;
}
function readShipUnion(): string {
return readSkillUnion('ship');
}
describe('CSO host permission boundary', () => {
test('grants no raw source, mutation, search, question, or Agent tools', () => {
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
const fmEnd = content.indexOf('\n---', 4);
const frontmatter = Bun.YAML.parse(content.slice(4, fmEnd)) as Record<string, unknown>;
const allowed = frontmatter['allowed-tools'];
expect(allowed).toEqual([
'Bash(~/.claude/skills/gstack/bin/gstack-cso-launcher *)',
'Bash(~/.claude/skills/gstack/bin/gstack-cso-launcher.exe *)',
]);
for (const broad of ['Bash', 'Read', 'Grep', 'Glob', 'Write', 'Agent', 'WebSearch', 'AskUserQuestion']) expect(allowed).not.toContain(broad);
});
test('retains helper-only source access, sequential challenge, and honest host containment', () => {
const content = readSkillUnion('cso');
expect(content).toContain('Never use host `Read`/`Glob`/`Grep`');
expect(content).toContain('sequential challenge; independent agent unavailable');
expectMentions(content, [['do not', 'independent', 'reviewer']], 'content');
expectMentions(content, [['does not', 'containment', 'sandbox']], 'content');
});
});
function readCodexSkillUnion(skill: string): string {
const dir = path.join(CODEX_OUT, '.agents', 'skills', `gstack-${skill}`);
const sections = path.join(dir, 'sections');
return fs.readFileSync(path.join(dir, 'SKILL.md'), 'utf-8')
+ (fs.existsSync(sections) ? fs.readdirSync(sections).sort()
.filter(file => file.endsWith('.md'))
.map(file => '\n' + fs.readFileSync(path.join(sections, file), 'utf-8')).join('') : '');
}
describe('SKILL.md command validation', () => {
// P2 (v1.2.0): the top-level gstack skill is a pure ROUTER, not the browse
// skill. The browse body lives only in browse/SKILL.md now. This regression
// pins the split: the router carries routing rules and zero browse commands,
// while browse/SKILL.md still advertises the full QA surface (asserted below).
test('top-level SKILL.md is a router with no browse body (P2)', () => {
const md = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
expect(md).not.toContain('gstack browse: QA Testing'); // browse body removed
expect(md).toContain('## Route first'); // router head present
expect(md).toContain('invoke `/investigate`'); // routing rules present
const result = validateSkill(path.join(ROOT, 'SKILL.md'));
expect(result.invalid).toHaveLength(0); // no INVALID browse commands
expect(result.valid.length).toBe(0); // and no browse commands at all — it routes, not browses
});
// Browse carve: the $B command reference renders into the on-demand section
// browse/sections/command-list.md; the skeleton carries the Aside contract
// (Aside is the primary browser, $B its fallback). Validate the union.
const BROWSE_DOCS = ['browse/SKILL.md', 'browse/sections/command-list.md']
.map((rel) => path.join(ROOT, rel)).filter((p) => fs.existsSync(p));
test('all $B commands in browse/SKILL.md + command-list section are valid browse commands', () => {
let validTotal = 0;
for (const doc of BROWSE_DOCS) {
const result = validateSkill(doc);
expect({ doc, invalid: result.invalid }).toEqual({ doc, invalid: [] });
validTotal += result.valid.length;
}
expect(validTotal).toBeGreaterThan(0);
});
test('all snapshot flags in browse/SKILL.md + command-list section are valid', () => {
for (const doc of BROWSE_DOCS) {
const result = validateSkill(doc);
expect({ doc, snapshotFlagErrors: result.snapshotFlagErrors }).toEqual({ doc, snapshotFlagErrors: [] });
}
});
test('all $B commands in qa/SKILL.md are valid browse commands', () => {
const qaSkill = path.join(ROOT, 'qa', 'SKILL.md');
if (!fs.existsSync(qaSkill)) return; // skip if missing
const result = validateSkill(qaSkill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in qa/SKILL.md are valid', () => {
const qaSkill = path.join(ROOT, 'qa', 'SKILL.md');
if (!fs.existsSync(qaSkill)) return;
const result = validateSkill(qaSkill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
// qa carve: the Phases 1-6 methodology (with its $B command examples) moved
// into qa/sections/*.md — validate the section files too so browse-command
// coverage doesn't silently shrink with the carve.
test('all $B commands and snapshot flags in qa/sections/*.md are valid', () => {
const secDir = path.join(ROOT, 'qa', 'sections');
if (!fs.existsSync(secDir)) return; // pre-carve checkout
const sectionMds = fs.readdirSync(secDir).filter(f => f.endsWith('.md') && !f.endsWith('.md.tmpl'));
expect(sectionMds.length).toBeGreaterThan(0);
for (const f of sectionMds) {
const result = validateSkill(path.join(secDir, f));
expect({ file: f, invalid: result.invalid }).toEqual({ file: f, invalid: [] });
expect({ file: f, snapshotFlagErrors: result.snapshotFlagErrors }).toEqual({ file: f, snapshotFlagErrors: [] });
}
});
test('all $B commands in qa-only/SKILL.md are valid browse commands', () => {
const qaOnlySkill = path.join(ROOT, 'qa-only', 'SKILL.md');
if (!fs.existsSync(qaOnlySkill)) return;
const result = validateSkill(qaOnlySkill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in qa-only/SKILL.md are valid', () => {
const qaOnlySkill = path.join(ROOT, 'qa-only', 'SKILL.md');
if (!fs.existsSync(qaOnlySkill)) return;
const result = validateSkill(qaOnlySkill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in plan-design-review/SKILL.md are valid browse commands', () => {
const skill = path.join(ROOT, 'plan-design-review', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in plan-design-review/SKILL.md are valid', () => {
const skill = path.join(ROOT, 'plan-design-review', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in design-review/SKILL.md are valid browse commands', () => {
const skill = path.join(ROOT, 'design-review', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in design-review/SKILL.md are valid', () => {
const skill = path.join(ROOT, 'design-review', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in design-consultation/SKILL.md are valid browse commands', () => {
const skill = path.join(ROOT, 'design-consultation', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in design-consultation/SKILL.md are valid', () => {
const skill = path.join(ROOT, 'design-consultation', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in autoplan/SKILL.md are valid browse commands', () => {
const skill = path.join(ROOT, 'autoplan', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in autoplan/SKILL.md are valid', () => {
const skill = path.join(ROOT, 'autoplan', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('autoplan section skip list includes the scope gate', () => {
// autoplan Step 3 reads plan-eng-review / plan-design-review SKILL.md
// verbatim; without this skip-list entry it ingests their scope gate — a
// hard-STOP AskUserQuestion that contradicts autoplan's auto-decide
// contract. Nothing else pins the skip-list contents.
const md = fs.readFileSync(path.join(ROOT, 'autoplan', 'SKILL.md'), 'utf-8');
expect(md).toContain('- Scope gate (the plan under review is already the target)');
});
});
describe('Command registry consistency', () => {
test('COMMAND_DESCRIPTIONS covers all commands in sets', () => {
const allCmds = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
const descKeys = new Set(Object.keys(COMMAND_DESCRIPTIONS));
for (const cmd of allCmds) {
expect(descKeys.has(cmd)).toBe(true);
}
});
test('COMMAND_DESCRIPTIONS has no extra commands not in sets', () => {
const allCmds = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
for (const key of Object.keys(COMMAND_DESCRIPTIONS)) {
expect(allCmds.has(key)).toBe(true);
}
});
test('ALL_COMMANDS matches union of all sets', () => {
const union = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
expect(ALL_COMMANDS.size).toBe(union.size);
for (const cmd of union) {
expect(ALL_COMMANDS.has(cmd)).toBe(true);
}
});
test('SNAPSHOT_FLAGS option keys are valid SnapshotOptions fields', () => {
const validKeys = new Set([
'interactive', 'compact', 'depth', 'selector',
'diff', 'annotate', 'outputPath', 'cursorInteractive',
'heatmap',
]);
for (const flag of SNAPSHOT_FLAGS) {
expect(validKeys.has(flag.optionKey)).toBe(true);
}
});
});
describe('Usage string consistency', () => {
// Normalize a usage string to its structural skeleton for comparison.
// Replaces <param-names> with <>, [optional] with [], strips parenthetical hints.
// This catches format mismatches (e.g., <name>:<value> vs <name> <value>)
// without tripping on abbreviation differences (e.g., <sel> vs <selector>).
function skeleton(usage: string): string {
return usage
.replace(/\(.*?\)/g, '') // strip parenthetical hints like (e.g., Enter, Tab)
.replace(/<[^>]*>/g, '<>') // normalize <param-name> → <>
.replace(/\[[^\]]*\]/g, '[]') // normalize [optional] → []
.replace(/\s+/g, ' ') // collapse whitespace
.trim();
}
// Cross-check Usage: patterns in implementation against COMMAND_DESCRIPTIONS
test('implementation Usage: structural format matches COMMAND_DESCRIPTIONS', () => {
const implFiles = [
path.join(ROOT, 'browse', 'src', 'write-commands.ts'),
path.join(ROOT, 'browse', 'src', 'read-commands.ts'),
path.join(ROOT, 'browse', 'src', 'meta-commands.ts'),
];
// Extract "Usage: browse <pattern>" from throw new Error(...) calls
const usagePattern = /throw new Error\(['"`]Usage:\s*browse\s+(.+?)['"`]\)/g;
const implUsages = new Map<string, string>();
for (const file of implFiles) {
const content = fs.readFileSync(file, 'utf-8');
let match;
while ((match = usagePattern.exec(content)) !== null) {
const usage = match[1].split('\\n')[0].trim();
const cmd = usage.split(/\s/)[0];
implUsages.set(cmd, usage);
}
}
// Compare structural skeletons
const mismatches: string[] = [];
for (const [cmd, implUsage] of implUsages) {
const desc = COMMAND_DESCRIPTIONS[cmd];
if (!desc) continue;
if (!desc.usage) continue;
const descSkel = skeleton(desc.usage);
const implSkel = skeleton(implUsage);
if (descSkel !== implSkel) {
mismatches.push(`${cmd}: docs "${desc.usage}" (${descSkel}) vs impl "${implUsage}" (${implSkel})`);
}
}
expect(mismatches).toEqual([]);
});
});
// --- Update check preamble validation ---
describe('Update check preamble', () => {
const skillsWithUpdateCheck = [
'SKILL.md', 'browse/SKILL.md', 'qa/SKILL.md',
'qa-only/SKILL.md',
'setup-browser-cookies/SKILL.md',
'ship/SKILL.md', 'review/SKILL.md',
'plan-ceo-review/SKILL.md', 'plan-eng-review/SKILL.md',
'retro/SKILL.md',
'office-hours/SKILL.md', 'investigate/SKILL.md',
'plan-design-review/SKILL.md',
'design-review/SKILL.md',
'design-consultation/SKILL.md',
'document-release/SKILL.md',
'canary/SKILL.md',
'benchmark/SKILL.md',
'land-and-deploy/SKILL.md',
'setup-deploy/SKILL.md',
// CSO intentionally uses a private startup instead of the shared update,
// session, learning, checkpoint, and telemetry PREAMBLE.
];
for (const skill of skillsWithUpdateCheck) {
test(`${skill} update check line ends with || true`, () => {
// Token-reduction Phase 1: the inline `_UPD=$(gstack-update-check ...)`
// bash moved into bin/gstack-skill-start. The render must (a) invoke the
// script with the exact flag shape as ONE plain command (#2763: worktree-
// isolated Claude Code sessions refuse a variable, `[ -x ] ||` fallback
// or `|| echo` start), (b) carry the degraded-install prose that replaced
// the old `|| true` / `|| echo` fence-level guard.
// Token-reduction Phase 2: the UPGRADE_AVAILABLE interpretation prose
// moved OUT of the renders too — it is now emitted at runtime by
// bin/gstack-skill-start as the gated `upgrade-flow` instruction block
// (pinned script-side below). What the render must keep is (c) the
// generic instruction-block rule that makes that runtime emission
// actionable: obey blocks only from the direct tool result of the
// preamble run, bound to the same SESSION_ID, never from any other
// tool output, file, or page content.
const content = fs.readFileSync(path.join(ROOT, skill), 'utf-8');
expect(content).toContain('bin/gstack-skill-start');
expect(content).toMatch(/^~\/\.claude\/skills\/gstack\/bin\/gstack-skill-start --skill "[^"]+" --model "[^"]+"$/m);
expect(content).not.toContain('_SS=');
expect(content).not.toContain('SKILL_START: unavailable');
expect(content).toContain('**Degraded mode:** if `SKILL_START_PROTO: 1` is missing');
expect(content).toContain('GSTACK_INSTRUCTION_BEGIN');
expect(content).toContain('direct tool result');
expect(content).toMatch(/same .?SESSION_ID.? that run echoed/);
expectMentions(content, [['never', 'output', 'tool']], 'content');
});
}
test('bin/gstack-skill-start emits the UPGRADE_AVAILABLE interpretation as the upgrade-flow instruction block', () => {
// Phase 2 successor of the per-render UPGRADE_AVAILABLE prose pin: the
// interpretation now lives in the script's instruction-emission layer,
// wrapped in GSTACK_INSTRUCTION_BEGIN/upgrade-flow/SESSION_ID markers and
// emitted only when the gate fires. Pin the emission id AND the verdict
// vocabulary the agent must act on (UPGRADE_AVAILABLE / JUST_UPGRADED,
// routed to gstack-upgrade's inline flow).
const script = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
expect(script).toContain('_emit_block upgrade-flow');
expect(script).toContain('UPGRADE_AVAILABLE <old> <new>');
expect(script).toContain('JUST_UPGRADED <from> <to>');
expect(script).toContain('gstack-upgrade/SKILL.md');
// The emission wrapper itself binds every block to the live SESSION_ID —
// the render-side rule above is only sound if this stays true.
expect(script).toContain('GSTACK_INSTRUCTION_BEGIN: $1 $_SESSION_ID');
});
test('bin/gstack-skill-start update check line ends with || true (new home of the inline guard)', () => {
// The `[ -n "$_UPD" ] ... || true` guard (empty _UPD must not exit 1 when
// up to date) moved verbatim into the consolidated preamble script. Pin it
// there so the invariant survives in its new home.
const script = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
const match = script.match(/\[ -n "\$_UPD" \].*$/m);
expect(match).not.toBeNull();
expect(match![0]).toContain('|| true');
});
test('all skills with update check are generated from .tmpl', () => {
for (const skill of skillsWithUpdateCheck) {
const tmplPath = path.join(ROOT, skill + '.tmpl');
expect(fs.existsSync(tmplPath)).toBe(true);
}
});
test('update check bash block exits 0 when up to date', () => {
// Simulate the exact update-check lines from bin/gstack-skill-start
// (per-line `|| true`, sanitize pipe included)
const result = Bun.spawnSync(['bash', '-c',
'_sanitize() { sed "s/GSTACK_INSTRUCTION/GSTACK-INSTRUCTION-(stripped)/g"; }; ' +
'_UPD=$(echo "" || true); [ -n "$_UPD" ] && printf "%s\\n" "$_UPD" | _sanitize || true'
], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.exitCode).toBe(0);
});
test('update check bash block exits 0 when upgrade available', () => {
const result = Bun.spawnSync(['bash', '-c',
'_sanitize() { sed "s/GSTACK_INSTRUCTION/GSTACK-INSTRUCTION-(stripped)/g"; }; ' +
'_UPD=$(echo "UPGRADE_AVAILABLE 0.3.3 0.4.0" || true); [ -n "$_UPD" ] && printf "%s\\n" "$_UPD" | _sanitize || true'
], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.exitCode).toBe(0);
expect(result.stdout.toString().trim()).toBe('UPGRADE_AVAILABLE 0.3.3 0.4.0');
});
});
// --- Part 7: Cross-skill path consistency (A1) ---
describe('Cross-skill path consistency', () => {
test('no qa or review skill file hand-rolls a REMOTE_SLUG derivation (gstack-slug and gstack-review-bots own it)', () => {
const patterns = extractRemoteSlugPatterns(ROOT, ['qa', 'review']);
const allPatterns: string[] = [];
for (const [, filePatterns] of patterns) {
allPatterns.push(...filePatterns);
}
// review/greptile-triage.md held the last prose copy until #3122 moved it into lib/review-bots.ts.
expect(allPatterns).toEqual([]);
});
test('every review-bot history write goes through the one writer (project and global rows: test/review-bots.test.ts)', () => {
// ship's bot handling moved into sections/greptile.md (T9 carve).
for (const content of [fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8'), readShipUnion()]) {
expect(content).toContain('History File Writes');
expect(content).not.toContain('greptile-history (type:');
}
});
test('greptile-triage.md records through gstack-review-bots into review-bot-history.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'greptile-triage.md'), 'utf-8');
expect(content).toContain('gstack-review-bots record --dir <bot-dir> --id <comment-id>');
expect(content).toContain('review-bot-history.md');
});
test('retro/SKILL.md reads the global review-bot and greptile histories (not per-project)', () => {
const content = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
expect(content).toContain('`REVIEW_BOT_HISTORY` / `GREPTILE_HISTORY: present`');
const metrics = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-retro-metrics'), 'utf-8');
expect(metrics).toContain('$_GH/greptile-history.md');
expect(metrics).toContain('$_GH/review-bot-history.md');
// Should NOT reference per-project path for reads
expect(content).not.toContain('$REMOTE_SLUG/greptile-history.md');
});
});
// --- Part 7: QA skill structure validation (A2) ---
describe('QA skill structure validation', () => {
// qa carve: modes, Phases 1-6, and the health rubric moved into
// qa/sections/qa-patterns.md — validate the skeleton+sections union.
const qaContent = readSkillUnion('qa');
test('qa/SKILL.md has all 11 phases', () => {
const phases = [
'Phase 1', 'Initialize',
'Phase 2', 'Authenticate',
'Phase 3', 'Orient',
'Phase 4', 'Explore',
'Phase 5', 'Document',
'Phase 6', 'Wrap Up',
'Phase 7', 'Triage',
'Phase 8', 'Fix Loop',
'Phase 9', 'Final QA',
'Phase 10', 'Report',
'Phase 11', 'TODOS',
];
for (const phase of phases) {
expect(qaContent).toContain(phase);
}
});
test('has all four QA modes defined', () => {
const modes = [
'Diff-aware',
'Full',
'Quick',
'Regression',
];
for (const mode of modes) {
expect(qaContent).toContain(mode);
}
// Mode triggers/flags
expect(qaContent).toContain('--quick');
expect(qaContent).toContain('--regression');
});
test('has all three tiers defined', () => {
const tiers = ['Quick', 'Standard', 'Exhaustive'];
for (const tier of tiers) {
expect(qaContent).toContain(tier);
}
});
test('health score weights sum to 100%', () => {
const weights = extractWeightsFromTable(qaContent);
expect(weights.size).toBeGreaterThan(0);
let sum = 0;
for (const pct of weights.values()) {
sum += pct;
}
expect(sum).toBe(100);
});
test('health score has all 8 categories', () => {
const weights = extractWeightsFromTable(qaContent);
const expectedCategories = [
'Console', 'Links', 'Visual', 'Functional',
'UX', 'Performance', 'Content', 'Accessibility',
];
for (const cat of expectedCategories) {
expect(weights.has(cat)).toBe(true);
}
expect(weights.size).toBe(8);
});
test('has four mode definitions (Diff-aware/Full/Quick/Regression)', () => {
expect(qaContent).toContain('### Diff-aware');
expect(qaContent).toContain('### Full');
expect(qaContent).toContain('### Quick');
expect(qaContent).toContain('### Regression');
});
test('output structure references report directory layout', () => {
expect(qaContent).toContain('qa-report-');
expect(qaContent).toContain('baseline.json');
expect(qaContent).toContain('screenshots/');
expect(qaContent).toContain('.gstack/qa-reports/');
});
});
// --- Part 7: Greptile history format consistency (A3) ---
describe('Greptile history format consistency', () => {
test('greptile-triage.md defines the canonical history format', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'greptile-triage.md'), 'utf-8');
expect(content).toContain('<YYYY-MM-DD>');
expect(content).toContain('<owner/repo>');
expect(content).toContain('<type');
expect(content).toContain('<file-pattern>');
expect(content).toContain('<category>');
});
test('review/SKILL.md and ship/SKILL.md both reference greptile-triage.md for write details', () => {
const reviewContent = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
const shipContent = readShipUnion();
expect(reviewContent.toLowerCase()).toContain('greptile-triage.md');
expect(shipContent.toLowerCase()).toContain('greptile-triage.md');
});
test('greptile-triage.md defines all 9 valid categories', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'greptile-triage.md'), 'utf-8');
const categories = [
'race-condition', 'null-check', 'error-handling', 'style',
'type-safety', 'security', 'performance', 'correctness', 'other',
];
for (const cat of categories) {
expect(content).toContain(cat);
}
});
});
// --- Hardcoded branch name detection in templates ---
describe('No hardcoded branch names in SKILL templates', () => {
const tmplFiles = [
'ship/SKILL.md.tmpl',
'review/SKILL.md.tmpl',
'qa/SKILL.md.tmpl',
'plan-ceo-review/SKILL.md.tmpl',
'retro/SKILL.md.tmpl',
'document-release/SKILL.md.tmpl',
'plan-eng-review/SKILL.md.tmpl',
'plan-design-review/SKILL.md.tmpl',
'codex/SKILL.md.tmpl',
];
// Patterns that indicate hardcoded 'main' in git commands
const gitMainPatterns = [
/\bgit\s+diff\s+(?:origin\/)?main\b/,
/\bgit\s+log\s+(?:origin\/)?main\b/,
/\bgit\s+fetch\s+origin\s+main\b/,
/\bgit\s+merge\s+origin\/main\b/,
/\borigin\/main\b/,
];
// Lines that are allowed to mention 'main' (fallback logic, prose)
const allowlist = [
/fall\s*back\s+to\s+`main`/i,
/fall\s*back\s+to\s+`?main`?/i,
/typically\s+`?main`?/i,
/If\s+on\s+`main`/i, // old pattern — should not exist
];
for (const tmplFile of tmplFiles) {
test(`${tmplFile} has no hardcoded 'main' in git commands`, () => {
const filePath = path.join(ROOT, tmplFile);
if (!fs.existsSync(filePath)) return;
const lines = fs.readFileSync(filePath, 'utf-8').split('\n');
const violations: string[] = [];
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
const isAllowlisted = allowlist.some(p => p.test(line));
if (isAllowlisted) continue;
for (const pattern of gitMainPatterns) {
if (pattern.test(line)) {
violations.push(`Line ${i + 1}: ${line.trim()}`);
break;
}
}
}
if (violations.length > 0) {
throw new Error(
`${tmplFile} has hardcoded 'main' in git commands:\n` +
violations.map(v => ` ${v}`).join('\n')
);
}
});
}
});
// --- Part 7b: TODOS-format.md reference consistency ---
describe('TODOS-format.md reference consistency', () => {
test('review/TODOS-format.md exists and defines canonical format', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'TODOS-format.md'), 'utf-8');
expect(content).toContain('**What:**');
expect(content).toContain('**Why:**');
expect(content).toContain('**Priority:**');
expect(content).toContain('**Effort:**');
expect(content).toContain('## Completed');
});
test('skills that write TODOs reference TODOS-format.md', () => {
const shipContent = readShipUnion();
const ceoPlanContent = readSkillUnion('plan-ceo-review'); // carved: TODOS-format ref moved to section
const engPlanContent = readSkillUnion('plan-eng-review');
expect(shipContent).toContain('TODOS-format.md');
expect(ceoPlanContent).toContain('TODOS-format.md');
expect(engPlanContent).toContain('TODOS-format.md');
});
});
// --- v0.4.1 feature coverage: RECOMMENDATION format, session awareness, enum completeness ---
describe('v0.4.1 preamble features', () => {
// Tier 1 skills have core preamble only (no AskUserQuestion format)
const tier1Skills = ['SKILL.md', 'browse/SKILL.md', 'setup-browser-cookies/SKILL.md', 'benchmark/SKILL.md'];
// Tier 2+ skills have AskUserQuestion format with RECOMMENDATION
const tier2PlusSkills = [
'qa/SKILL.md', 'qa-only/SKILL.md',
'ship/SKILL.md', 'review/SKILL.md',
'plan-ceo-review/SKILL.md', 'plan-eng-review/SKILL.md',
'retro/SKILL.md',
'office-hours/SKILL.md', 'investigate/SKILL.md',
'plan-design-review/SKILL.md',
'design-review/SKILL.md',
'design-consultation/SKILL.md',
'document-release/SKILL.md',
'canary/SKILL.md',
'land-and-deploy/SKILL.md',
'setup-deploy/SKILL.md',
// CSO's private startup intentionally omits the generic AUQ/session/
// escalation PREAMBLE; its helper owns readiness and terminal state.
];
const skillsWithPreamble = [...tier1Skills, ...tier2PlusSkills];
for (const skill of tier2PlusSkills) {
test(`${skill} contains AskUserQuestion Pros/Cons format`, () => {
const content = readSkillStartup(path.join(ROOT, skill));
// v1.7.0.0 Pros/Cons format tokens. The preamble resolver
// (generate-ask-user-format.ts) injects all of these into every
// tier-2+ skill. Drop any of them and the test catches it on the
// next `bun test` run.
expect(content).toContain('AskUserQuestion');
expect(content).toContain('Pros / cons:');
expect(content).toContain('Recommendation: <choice>');
expect(content).toContain('Net:');
expect(content).toContain('ELI10');
expect(content).toContain('Stakes if we pick wrong:');
// Concrete format markers must be documented in the resolver text
expect(content).toMatch(/✅/);
expect(content).toMatch(/❌/);
});
}
for (const skill of skillsWithPreamble) {
test(`${skill} contains session awareness`, () => {
// Token-reduction Phase 1: the inline `_SESSIONS=$(find ~/.gstack/sessions ...)`
// bash moved into bin/gstack-skill-start. Session identity (the harness
// pid that names the sessions-dir entry) is derived by the script itself,
// so the render passes no "$PPID" (#2763); it still carries the
// SESSION_KIND STATUS-line interpretation prose.
const content = readSkillStartup(path.join(ROOT, skill));
expect(content).not.toContain('$PPID');
expect(content).toContain('SESSION_KIND');
});
}
test('bin/gstack-skill-start owns the session-tracking machinery (new home of _SESSIONS)', () => {
// The sessions-dir touch + stale-session cleanup that every preamble used
// to inline now lives in the consolidated script — pin it there.
const script = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
expect(script).toContain('mkdir -p "$_GH/sessions"');
expect(script).toContain('touch "$_GH/sessions/$PARENT_PID"');
expect(script).toContain('-mmin +120'); // 120-min freshness window survives the move
});
for (const skill of skillsWithPreamble) {
test(`${skill} contains escalation protocol`, () => {
const content = readSkillStartup(path.join(ROOT, skill));
expect(content).toContain('DONE_WITH_CONCERNS');
expect(content).toContain('BLOCKED');
expect(content).toContain('NEEDS_CONTEXT');
});
}
});
// --- Structural tests for new skills ---
describe('office-hours skill structure', () => {
// Carved (v2 plan T9): Phase 5 (Design Doc) + Phase 6 (handoff) moved into
// sections/design-and-handoff.md, so structural phrases now live there — read
// the skeleton+sections union.
const content = readSkillUnion('office-hours');
// Original structural assertions
for (const section of ['Phase 1', 'Phase 2', 'Phase 3', 'Phase 4', 'Phase 5', 'Phase 6',
'Design Doc', 'Supersedes', 'APPROVED', 'Premise Challenge',
'Alternatives', 'Smart-skip']) {
test(`contains ${section}`, () => expect(content).toContain(section));
}
// Dual-mode structure
for (const section of ['Startup mode', 'Builder mode']) {
test(`contains ${section}`, () => expect(content).toContain(section));
}
// Mode detection question
test('contains explicit mode detection question', () => {
expect(content).toContain("what's your goal");
});
// Six forcing questions (startup mode)
for (const question of ['Demand Reality', 'Status Quo', 'Desperate Specificity',
'Narrowest Wedge', 'Observation & Surprise', 'Future-Fit']) {
test(`contains forcing question: ${question}`, () => expect(content).toContain(question));
}
// Builder mode questions
test('contains builder brainstorming questions', () => {
expect(content).toContain('coolest version');
expect(content).toContain('delightful');
});
// Intrapreneurship adaptation
test('contains intrapreneurship adaptation', () => {
expect(content).toContain('Intrapreneurship');
});
// YC founder discovery engine
test('contains YC apply CTA with ref tracking', () => {
expect(content).toContain('ycombinator.com/apply?ref=gstack');
});
test('contains "What I noticed" design doc section', () => {
expect(content).toContain('What I noticed about how you think');
});
test('contains golden age framing', () => {
expect(content).toContain('golden age');
});
test('contains Garry Tan personal plea', () => {
expect(content).toContain('Garry Tan, the creator of GStack');
});
test('contains founder signal synthesis phase', () => {
expect(content).toContain('Founder Signal Synthesis');
});
test('contains three-tier decision rubric', () => {
expect(content).toContain('Top tier');
expect(content).toContain('Middle tier');
expect(content).toContain('Base tier');
});
test('contains anti-slop examples', () => {
expect(content).toContain('GOOD:');
expect(content).toContain('BAD:');
});
test('contains "One more thing" transition beat', () => {
expect(content).toContain('One more thing');
});
// Operating principles per mode
test('contains startup operating principles', () => {
expect(content).toContain('Specificity is the only currency');
});
test('contains builder operating principles', () => {
expect(content).toContain('Delight is the currency');
});
// Spec Review Loop (Phase 5.5)
test('contains spec review loop', () => {
expect(content).toContain('Spec Review Loop');
});
test('contains adversarial review dimensions', () => {
for (const dim of ['Completeness', 'Consistency', 'Clarity', 'Scope', 'Feasibility']) {
expect(content).toContain(dim);
}
});
test('contains subagent dispatch instruction', () => {
expect(content).toMatch(/Agent.*tool|subagent/i);
});
test('contains max 3 iterations', () => {
expect(content).toMatch(/3.*iteration|maximum.*3/i);
});
test('contains quality score', () => {
expect(content).toContain('quality score');
});
test('contains spec review metrics path', () => {
expect(content).toContain('spec-review.jsonl');
});
test('contains convergence guard', () => {
expect(content).toMatch(/convergence/i);
});
// Visual Sketch (Phase 4.5)
test('contains visual sketch section', () => {
expect(content).toContain('Visual Sketch');
});
test('contains wireframe generation', () => {
expect(content).toMatch(/wireframe|sketch/i);
});
test('contains DESIGN.md awareness', () => {
expect(content).toContain('DESIGN.md');
});
test('wireframes render through gstack-render (Aside first)', () => {
expect(content).toContain('gstack-render.ts');
expect(content).toContain('--screenshot');
});
test('contains rough aesthetic instruction', () => {
expect(content).toMatch(/rough|hand-drawn/i);
});
});
describe('investigate skill structure', () => {
const content = fs.readFileSync(path.join(ROOT, 'investigate', 'SKILL.md'), 'utf-8');
for (const section of ['Iron Law', 'How to work', 'Scope Lock',
'DEBUG REPORT', '3-strike', 'BLOCKED']) {
test(`contains ${section}`, () => expect(content).toContain(section));
}
});
// Contributor mode was removed in v0.13.10.0 — replaced by operational self-improvement.
// Tests for contributor mode preamble structure are no longer applicable.
describe('Enum & Value Completeness in review checklist', () => {
const checklist = fs.readFileSync(path.join(ROOT, 'review', 'checklist.md'), 'utf-8');
test('checklist has Enum & Value Completeness section', () => {
expect(checklist).toContain('Enum & Value Completeness');
});
test('Enum & Value Completeness is classified as CRITICAL', () => {
// It should appear under Pass 1 — CRITICAL, not Pass 2
const pass1Start = checklist.indexOf('### Pass 1');
const pass2Start = checklist.indexOf('### Pass 2');
const enumStart = checklist.indexOf('Enum & Value Completeness');
expect(enumStart).toBeGreaterThan(pass1Start);
expect(enumStart).toBeLessThan(pass2Start);
});
test('Enum & Value Completeness mentions tracing through consumers', () => {
expect(checklist).toContain('Trace it through every consumer');
expect(checklist).toContain('case');
expect(checklist).toContain('allowlist');
});
test('Enum & Value Completeness covers loosened input acceptance and stale user-facing strings', () => {
const start = checklist.indexOf('#### Enum & Value Completeness');
const section = checklist.slice(start, checklist.indexOf('### Pass 2', start));
expect(section).toContain('loosens what an input accepts');
expect(section).toContain('Loosened acceptance breaks unchanged consumers');
const maintainability = fs.readFileSync(path.join(ROOT, 'review', 'specialists', 'maintainability.md'), 'utf-8');
const stale = maintainability.slice(maintainability.indexOf('### Stale Comments & Docstrings'));
expect(stale.slice(0, stale.indexOf('\n### ', 5))).toContain('User-facing strings (errors, toasts, labels) whose guarding condition changed');
});
test('Enum & Value Completeness is in the severity classification as CRITICAL', () => {
const gateSection = checklist.slice(checklist.indexOf('## Severity Classification'));
// The ASCII art has CRITICAL on the left and INFORMATIONAL on the right
// Enum & Value Completeness should appear on a line with the CRITICAL tree (├─ or └─)
const enumLine = gateSection.split('\n').find(l => l.includes('Enum & Value Completeness'));
expect(enumLine).toBeDefined();
// It's on the left (CRITICAL) side — starts with ├─ or └─
expect(enumLine!.trimStart().startsWith('├─') || enumLine!.trimStart().startsWith('└─')).toBe(true);
});
test('Fix-First Heuristic exists in checklist and is referenced by review + ship', () => {
expect(checklist).toContain('## Fix-First Heuristic');
expect(checklist).toContain('AUTO-FIX');
expect(checklist).toContain('ASK');
const reviewSkill = fs.readFileSync(path.join(ROOT, 'review/SKILL.md'), 'utf-8');
const shipSkill = readShipUnion();
expect(reviewSkill).toContain('AUTO-FIX');
expect(reviewSkill).toContain('[AUTO-FIXED]');
expect(shipSkill).toContain('AUTO-FIX');
expect(shipSkill).toContain('[AUTO-FIXED]');
});
});
// --- Completeness Principle spot-check ---
describe('Completeness Principle in generated SKILL.md files', () => {
const skillsWithPreamble = [
'qa/SKILL.md',
'qa-only/SKILL.md',
'ship/SKILL.md', 'review/SKILL.md',
'plan-ceo-review/SKILL.md', 'plan-eng-review/SKILL.md',
'retro/SKILL.md',
'plan-design-review/SKILL.md',
'design-review/SKILL.md',
'design-consultation/SKILL.md',
'document-release/SKILL.md',
// CSO reports complete/partial/not-assessed from its own evidence contract
// and must not inherit the shared numerical completeness rubric.
];
for (const skill of skillsWithPreamble) {
test(`${skill} contains Completeness Principle section`, () => {
const content = readSkillStartup(path.join(ROOT, skill));
expect(content).toContain('Completeness Principle');
expect(content).toContain('Boil the Ocean');
});
}
test('Completeness Principle keeps compact scoring guidance in tier 2+ skills', () => {
// CSO is intentionally exempt; use a regular tier 2+ PREAMBLE consumer.
const content = readSkillStartup(path.join(ROOT, 'review', 'SKILL.md'));
expect(content).toContain('Completeness: X/10');
expect(content).toContain('10 = all edge cases');
expect(content).toContain('Note: options differ in kind, not coverage');
expect(content).toContain('Do not fabricate scores');
});
});
// --- Part 7: Planted-bug fixture validation (A4) ---
describe('Planted-bug fixture validation', () => {
test('qa-eval ground truth has exactly 5 planted bugs', () => {
const groundTruth = JSON.parse(
fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'qa-eval-ground-truth.json'), 'utf-8')
);
expect(groundTruth.bugs).toHaveLength(5);
expect(groundTruth.total_bugs).toBe(5);
});
test('qa-eval-spa ground truth has exactly 5 planted bugs', () => {
const groundTruth = JSON.parse(
fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'qa-eval-spa-ground-truth.json'), 'utf-8')
);
expect(groundTruth.bugs).toHaveLength(5);
expect(groundTruth.total_bugs).toBe(5);
});
test('qa-eval-checkout ground truth has exactly 5 planted bugs', () => {
const groundTruth = JSON.parse(
fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'qa-eval-checkout-ground-truth.json'), 'utf-8')
);
expect(groundTruth.bugs).toHaveLength(5);
expect(groundTruth.total_bugs).toBe(5);
});
test('qa-eval.html contains the planted bugs', () => {
const html = fs.readFileSync(path.join(ROOT, 'browse', 'test', 'fixtures', 'qa-eval.html'), 'utf-8');
// BUG 1: broken link
expect(html).toContain('/nonexistent-404-page');
// BUG 2: disabled submit
expect(html).toContain('disabled');
// BUG 3: overflow
expect(html).toContain('overflow: hidden');
// BUG 4: missing alt
expect(html).toMatch(/<img[^>]*src="\/logo\.png"[^>]*>/);
expect(html).not.toMatch(/<img[^>]*src="\/logo\.png"[^>]*alt=/);
// BUG 5: console error
expect(html).toContain("Cannot read properties of undefined");
});
test('review-eval-vuln.rb contains expected vulnerability patterns', () => {
const content = fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-vuln.rb'), 'utf-8');
expect(content).toContain('params[:id]');
expect(content).toContain('update_column');
});
});
// --- CEO review mode validation ---
describe('CEO review mode validation', () => {
const content = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
test('has all four CEO review modes defined', () => {
const modes = ['SCOPE EXPANSION', 'SELECTIVE EXPANSION', 'HOLD SCOPE', 'SCOPE REDUCTION'];
for (const mode of modes) {
expect(content).toContain(mode);
}
});
test('has CEO plan persistence step', () => {
expect(content).toContain('ceo-plans');
expect(content).toContain('status: ACTIVE');
});
test('has docs/designs promotion section', () => {
// Carved (v2 plan Phase B): the promotion block moved into the review section.
const union = readSkillUnion('plan-ceo-review');
expect(union).toContain('docs/designs');
expect(union).toContain('PROMOTED');
});
test('mode quick reference has four columns', () => {
expect(content).toContain('EXPANSION');
expect(content).toContain('SELECTIVE');
expect(content).toContain('HOLD SCOPE');
expect(content).toContain('REDUCTION');
});
// Skill chaining (benefits-from)
test('contains prerequisite skill offer for office-hours', () => {
expect(content).toContain('Prerequisite Skill Offer');
expect(content).toContain('/office-hours');
});
test('contains mid-session detection', () => {
expect(content).toContain('Mid-session detection');
expect(content).toMatch(/still figuring out|seems lost/i);
});
// Spec review on CEO plans
test('contains spec review loop for CEO plan documents', () => {
expect(content).toContain('Spec Review Loop');
});
});
// --- gstack-slug helper ---
describe('gstack-slug', () => {
const SLUG_BIN = path.join(ROOT, 'bin', 'gstack-slug');
usePrivateStateRoot();
test('binary exists and is executable', () => {
expect(fs.existsSync(SLUG_BIN)).toBe(true);
const stat = fs.statSync(SLUG_BIN);
expect(stat.mode & 0o111).toBeGreaterThan(0);
});
test('outputs SLUG and BRANCH lines in a git repo', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, env: { ...process.env }, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
expect(output).toContain('SLUG=');
expect(output).toContain('BRANCH=');
});
test('SLUG does not contain forward slashes', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, env: { ...process.env }, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const slug = result.stdout.toString().match(/SLUG=(.*)/)?.[1] ?? '';
expect(slug).not.toContain('/');
expect(slug.length).toBeGreaterThan(0);
});
test('BRANCH does not contain forward slashes', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, env: { ...process.env }, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const branch = result.stdout.toString().match(/BRANCH=(.*)/)?.[1] ?? '';
expect(branch).not.toContain('/');
expect(branch.length).toBeGreaterThan(0);
});
test('output is eval-compatible (KEY=VALUE format)', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, env: { ...process.env }, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const lines = result.stdout.toString().trim().split('\n');
expect(lines.length).toBe(2);
expect(lines[0]).toMatch(/^SLUG=.+/);
expect(lines[1]).toMatch(/^BRANCH=.+/);
});
test('output values contain only safe characters (no shell metacharacters)', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, env: { ...process.env }, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const slug = result.stdout.toString().match(/SLUG=(.*)/)?.[1] ?? '';
const branch = result.stdout.toString().match(/BRANCH=(.*)/)?.[1] ?? '';
// Only alphanumeric, dot, dash, underscore are allowed (#133)
expect(slug).toMatch(/^[a-zA-Z0-9._-]+$/);
expect(branch).toMatch(/^[a-zA-Z0-9._-]+$/);
});
test('eval sets variables under bash with set -euo pipefail', () => {
const result = Bun.spawnSync(
['bash', '-c', 'set -euo pipefail; eval "$(./bin/gstack-slug 2>/dev/null)"; echo "SLUG=$SLUG"; echo "BRANCH=$BRANCH"'],
{ cwd: ROOT, env: { ...process.env }, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }
);
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
expect(output).toMatch(/^SLUG=.+/m);
expect(output).toMatch(/^BRANCH=.+/m);
});
test('no templates or bin scripts use source process substitution for gstack-slug', () => {
// Scan project sources, including new untracked templates, without walking
// ignored caches, symlinked installations or private evaluation worktrees.
const result = Bun.spawnSync(
['git', 'ls-files', '--cached', '--others', '--exclude-standard', '-z', '--', '*.tmpl', 'bin/gstack-review-*'],
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 5_000 },
);
expect(result.exitCode, result.stderr.toString()).toBe(0);
const files = [...new Set(result.stdout.toString().split('\0').filter(Boolean))];
expect(files.length).toBeGreaterThan(0);
const violations = files.filter(file => /source <\(.*gstack-slug/.test(fs.readFileSync(path.join(ROOT, file), 'utf8')));
expect(violations).toEqual([]);
});
});
// --- Test Bootstrap validation ---
describe('Test Bootstrap ({{TEST_BOOTSTRAP}}) integration', () => {
test('TEST_BOOTSTRAP resolver produces valid content', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship/sections/tests.md'), 'utf8');
expect(content).toContain('Test Framework Bootstrap');
expect(content).toContain('RUNTIME:ruby');
expect(content).toContain('RUNTIME:node');
expect(content).toContain('RUNTIME:python');
expect(content).toContain('no-test-bootstrap');
expect(content).toContain('BOOTSTRAP_DECLINED');
});
test('TEST_BOOTSTRAP appears in qa/SKILL.md', () => {
const content = readSkillUnion('qa');
expect(content).toContain('Test Framework Bootstrap');
expect(content).toContain('TESTING.md');
expect(content).toContain('CLAUDE.md');
});
test('TEST_BOOTSTRAP appears in ship/SKILL.md', () => {
const content = readShipUnion();
expect(content).toContain('Test Framework Bootstrap');
expect(content).toContain('Step 4');
});
test('TEST_BOOTSTRAP appears in design-review/sections/test-bootstrap.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'sections', 'test-bootstrap.md'), 'utf-8');
expect(content).toContain('Test Framework Bootstrap');
});
test('TEST_BOOTSTRAP does NOT appear in qa-only/SKILL.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md'), 'utf-8');
expect(content).not.toContain('Test Framework Bootstrap');
// But should have the recommendation note
expect(content).toContain('No test framework detected');
expect(content).toContain('Run `/qa` to bootstrap');
});
test('bootstrap includes framework knowledge table', () => {
const content = readSkillUnion('qa');
expect(content).toContain('vitest');
expect(content).toContain('minitest');
expect(content).toContain('pytest');
expect(content).toContain('cargo test');
expect(content).toContain('phpunit');
expect(content).toContain('ExUnit');
});
test('bootstrap includes CI/CD pipeline generation', () => {
const content = readSkillUnion('qa');
expect(content).toContain('.github/workflows/test.yml');
expect(content).toContain('GitHub Actions');
});
test('bootstrap includes first real tests step', () => {
const content = readSkillUnion('qa');
expect(content).toContain('First real tests');
expect(content).toContain('git log --since=30.days');
expect(content).toContain('Prioritize by risk');
});
test('bootstrap includes vibe coding philosophy', () => {
const content = readSkillUnion('qa');
expect(content).toContain('vibe coding');
expect(content).toContain('100% test coverage');
});
test('WebSearch is in allowed-tools for qa, ship, design-review', () => {
const qa = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
const ship = readShipUnion();
const qaDesign = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
expect(qa).toContain('WebSearch');
expect(ship).toContain('WebSearch');
expect(qaDesign).toContain('WebSearch');
});
});
// --- Phase 8e.5 regression test validation ---
describe('Phase 8e.5 regression test generation', () => {
test('qa/SKILL.md contains Phase 8e.5', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('8e.5. Regression Test');
expect(content).toContain('test(qa): regression test');
expectMentions(content, [['do not', 'test-only', 'commits']], 'content');
});
test('qa/SKILL.md Rule 13 is amended for regression tests', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expectMentions(content, [['only', 'codification', 'authorized']], 'content');
expectMentions(content, [['never', 'configuration', 'existing']], 'content');
expect(content.indexOf('### 8a.5. Regression test before repair')).toBeLessThan(content.indexOf('### 8b. Fix'));
expectMentions(content, [['before', 'detected', 'command']], 'content');
const exploratory = fs.readFileSync(path.join(ROOT, 'qa', 'sections', 'exploratory.md'), 'utf-8').replace(/\s+/g, ' ');
expectMentions(exploratory, [['before', 'regression', 'verified']], 'exploratory');
expectMentions(exploratory, [['before', 'command/request', 'failing']], 'exploratory');
expectMentions(exploratory, [['not', 'regression', 'another']], 'exploratory');
});
test('design-review has CSS-aware Phase 8e.5 variant', () => {
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
expect(content).toContain('8e.5. Regression Test (design-review variant)');
expect(content).toContain('CSS-only');
expect(content).toContain('test(design): regression test');
});
test('regression test includes full attribution comment format', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('// Regression: ISSUE-NNN');
expect(content).toContain('// Found by /qa on');
expect(content).toContain('// Report: .gstack/qa-reports/');
});
test('regression test uses auto-incrementing names', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('auto-incrementing');
expect(content).toContain('max number + 1');
});
});
// --- Step 3.4 coverage audit validation ---
describe('Step 3.4 test coverage audit', () => {
test('ship/SKILL.md contains Step 7', () => {
const content = readShipUnion();
expect(content).toContain('Step 7: Test Coverage Audit');
// The coverage diagram collapses code-path and user-flow counts onto one
// summary line. Verify that summary is present (labels are stable).
expect(content).toContain('Code paths:');
});
test('Step 3.4 includes quality scoring rubric', () => {
const content = readShipUnion();
expect(content).toContain('★★★');
expect(content).toContain('★★');
expect(content).toContain('edge cases AND error paths');
expect(content).toContain('happy path only');
});
test('Step 3.4 includes before/after test count', () => {
const content = readShipUnion();
expect(content).toContain('Count test files before');
expect(content).toContain('Count test files after');
});
test('ship PR body includes Test Coverage section', () => {
const content = readShipUnion();
expect(content).toContain('## Test Coverage');
});
test('ship rules include test generation rule', () => {
const content = readShipUnion();
expect(content).toContain('Step 7 generates coverage tests');
expect(content).toContain('Never commit failing tests');
});
test('Step 3.4 states the value-based coverage goal', () => {
const content = readShipUnion();
expect(content).toContain('Coverage goal: every changed behavior is protected by a test that would catch a real regression. Test count is not a goal.');
});
test('Step 3.4 traces actual codepaths, not just syntax', () => {
const content = readShipUnion();
expect(content).toContain('Trace every codepath');
expect(content).toContain('Trace data flow');
expect(content).toContain('Diagram the execution');
});
test('Step 3.4 maps user flows and interaction edge cases', () => {
const content = readShipUnion();
expect(content).toContain('Map user flows');
expect(content).toContain('Interaction edge cases');
expect(content).toContain('Double-click');
expect(content).toContain('Navigate away');
expect(content).toContain('Error states the user can see');
expect(content).toContain('Empty/zero/boundary states');
});
test('Step 3.4 diagram includes user-flow coverage summary', () => {
const content = readShipUnion();
// The diagram was compressed from separate CODE PATH COVERAGE / USER FLOW
// COVERAGE section headers into a single summary line. Assert on the
// labels that still appear on that summary line.
expect(content).toContain('Code paths:');
expect(content).toContain('User flows:');
});
});
// --- Ship step numbering regression guard ---
describe('ship step numbering', () => {
// Allowed sub-steps that are resolver-generated and intentionally nested:
// 0.9 (Apple target detection — MUST precede Step 1's branch gate, R2-pinned
// by test/ship-apple-gate.test.ts), 8.1 (Plan Verification), 8.2 (Scope
// Drift), 9.1 (Review Army), 9.2 (Findings Merge), 9.3 (Cross-review dedup),
// 9.4 (Fix-First and persistence), 15.0 (WIP context), 15.1 (Bisectable commits),
// 15.2 (safe optional WIP consolidation), 6.5 (early Greptile PR, which must
// follow the free tests and precede the reviews it runs in parallel with).
const ALLOWED_SUBSTEPS = new Set(['0.9', '6.5', '8.1', '8.2', '9.1', '9.2', '9.3', '9.4', '11.5', '14.5', '15.0', '15.1', '15.2']);
test('ship/SKILL.md.tmpl contains no unexpected fractional step numbers', () => {
const tmpl = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md.tmpl'), 'utf-8');
// Match "Step X.Y" where X.Y is a decimal step reference (e.g., "Step 3.47", "Step 8.1")
const matches = Array.from(tmpl.matchAll(/Step (\d+\.\d+)/g));
const violations = matches
.map((m) => m[1])
.filter((n) => !ALLOWED_SUBSTEPS.has(n));
if (violations.length > 0) {
const unique = Array.from(new Set(violations)).sort();
throw new Error(
`ship/SKILL.md.tmpl contains fractional step numbers that are not in the allowed sub-step list.\n` +
` Found: ${unique.join(', ')}\n` +
` Allowed sub-steps: ${Array.from(ALLOWED_SUBSTEPS).sort().join(', ')}\n` +
` Fix: use clean integer step numbers (1-20), or add to ALLOWED_SUBSTEPS if intentional.`
);
}
});
test('ship/SKILL.md main headings use clean integer step numbers', () => {
const skill = readShipUnion();
// Headings like "## Step 7: Test Coverage Audit" — NOT sub-steps like "## Step 8.1:"
const headings = Array.from(skill.matchAll(/^## Step (\d+(?:\.\d+)?):/gm)).map(
(m) => m[1]
);
const fractional = headings.filter((n) => n.includes('.'));
const unexpected = fractional.filter((n) => !ALLOWED_SUBSTEPS.has(n));
expect(unexpected).toEqual([]);
expect(headings.filter((n) => n === '11.5')).toHaveLength(1);
});
test('review/SKILL.md step numbers unchanged (regression guard for resolver conditionals)', () => {
// Carved skill: Step 4.5 lives in sections/review-army.md and Step 4.8 in
// sections/adversarial.md — read the skeleton+sections union.
const skill = readSkillUnion('review');
// /review uses its own fractional numbering: 1.5, 2.5, 4.5, 4.8, 5.8
// If the ship-side renumber accidentally touched the review-side of resolver conditionals,
// these would vanish. This test catches that.
expect(skill).toContain('## Step 1.5: Scope Drift Detection');
expect(skill).toContain('## Step 4.5: Review Army');
expect(skill).toContain('## Step 4.8: Adversarial review');
});
});
// --- Retro test health validation ---
describe('Retro test health tracking', () => {
// The inline git pipelines moved into bin/gstack-retro-metrics (retro
// token-reduction wave); the skill interprets its labeled METRIC lines.
test('gstack-retro-metrics gathers the test health data', () => {
const script = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-retro-metrics'), 'utf-8');
expect(script).toContain('TEST_FILES_TOTAL');
expect(script).toContain('REGRESSION_TEST_COMMITS');
expect(script).toContain('TEST_FILES_CHANGED');
// The historical grep targets survive the script absorption.
expect(script).toContain('test(qa):');
expect(script).toContain('test(design):');
expect(script).toContain('git ls-files');
});
test('retro skill interprets the test-health metric lines', () => {
// Template source, not the generated render: the pin must hold across the
// regen boundary (the generated file follows the template at gen time).
const content = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md.tmpl'), 'utf-8');
expect(content).toContain('TEST_FILES_TOTAL');
expect(content).toContain('REGRESSION_TEST_COMMITS');
});
test('retro/SKILL.md has Test Health metrics row', () => {
const content = readSkillUnion('retro');
expect(content).toContain('Test Health');
expect(content).toContain('N test files');
expect(content).toContain('M changed this period');
expect(content).toContain('K regression test commits');
});
test('retro/SKILL.md has Test Health narrative section', () => {
// Carved: the narrative report format lives in sections/report-format.md.
const content = readSkillUnion('retro');
expect(content).toContain('### Test Health');
expect(content).toContain('Total test files');
expect(content).toContain('vibe coding safe');
});
test('retro JSON schema includes test_health field', () => {
const content = readSkillUnion('retro');
expect(content).toContain('test_health');
expect(content).toContain('total_test_files');
expect(content).toContain('regression_test_commits');
});
});
// --- QA report template regression tests section ---
describe('QA report template', () => {
test('qa-report-template.md has Regression Tests section', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'templates', 'qa-report-template.md'), 'utf-8');
expect(content).toContain('## Regression Tests');
expect(content).toContain('committed / deferred / skipped');
expect(content).toContain('### Deferred Tests');
expect(content).toContain('**Precondition:**');
});
});
// --- Codex skill validation ---
describe('Codex skill', () => {
test('codex/SKILL.md exists and has correct frontmatter', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('name: codex');
expect(content).toContain('version: 1.0.0');
expect(content).toContain('allowed-tools:');
});
// Carved skill (T9): the three mode bodies live in codex/sections/
// {review,challenge,consult}-mode.md. Content pins read the union; the
// mode DISPATCH must stay in the always-loaded skeleton.
test('codex union contains all three modes; skeleton keeps the dispatch', () => {
const content = readSkillUnion('codex');
expect(content).toContain('Step 2A: Review Mode');
expect(content).toContain('Step 2B: Challenge');
expect(content).toContain('Step 2C: Consult Mode');
const skeleton = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(skeleton).toContain('## Step 1: Detect mode');
expect(skeleton).toMatch(/mutually exclusive/i);
});
test('codex union contains gate verdict logic', () => {
const content = readSkillUnion('codex');
expect(content).toContain('[P1]');
expect(content).toContain('GATE: PASS');
expect(content).toContain('GATE: FAIL');
});
test('codex union contains session continuity', () => {
const content = readSkillUnion('codex');
expect(content).toContain('codex-session-id');
expect(content).toContain('codex exec resume');
});
test('codex resume command only uses resume-supported flags', () => {
const content = readSkillUnion('codex');
const match = content.match(/codex exec resume[^\n]+/);
expect(match).not.toBeNull();
const resumeCommand = match![0];
expect(resumeCommand).not.toContain(' -C ');
expect(resumeCommand).not.toMatch(/ -s /);
// resume takes the sandbox only as config; the probe's select-model reports it (read-only by default).
expect(resumeCommand).toContain('-c "sandbox_mode=\\"${_CODEX_SANDBOX_MODE:?}\\""');
});
test('codex union contains cost tracking', () => {
const content = readSkillUnion('codex');
expect(content).toContain('tokens used');
expect(content).toContain('Est. cost');
});
test('codex union contains cross-model comparison', () => {
const content = readSkillUnion('codex');
expect(content).toContain('CROSS-MODEL ANALYSIS');
expect(content).toContain('Agreement rate');
});
test('codex union contains review log persistence', () => {
const content = readSkillUnion('codex');
expect(content).toContain('codex-review');
expect(content).toContain('gstack-review-log');
});
test('codex uses command -v for binary discovery, not hardcoded path', () => {
const content = readSkillUnion('codex');
expect(content).toContain('command -v codex');
expect(content).not.toContain('/opt/homebrew/bin/codex');
// Defensive: catch any future regression that reintroduces `which codex`,
// which fails in environments where `which` isn't on PATH (some Windows
// shells, BusyBox-only containers). #1197.
expect(content).not.toContain('which codex');
});
test('codex/SKILL.md contains error handling for missing binary and auth', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('NOT_FOUND');
expect(content).toContain('codex login');
});
test('codex union uses mktemp for temp files', () => {
const content = readSkillUnion('codex');
expect(content).toContain('mktemp');
});
test('codex JSON stream parser uses portable Python discovery', () => {
// The JSONL parsers live in the challenge + consult mode sections; sweep
// the tmpl union and the rendered union so neither side drifts.
const sectionsDir = path.join(ROOT, 'codex', 'sections');
const tmplUnion = ['codex/SKILL.md.tmpl']
.map((rel) => fs.readFileSync(path.join(ROOT, rel), 'utf-8'))
.concat(
fs.readdirSync(sectionsDir).sort()
.filter((f) => f.endsWith('.md.tmpl'))
.map((f) => fs.readFileSync(path.join(sectionsDir, f), 'utf-8')),
)
.join('\n');
for (const content of [tmplUnion, readSkillUnion('codex')]) {
expect(content).toContain('PYTHON_CMD=$(command -v python3 2>/dev/null || command -v python 2>/dev/null || true)');
expect(content).toContain('PYTHONUNBUFFERED=1 "$PYTHON_CMD" -u -c');
expect(content).not.toContain('PYTHONUNBUFFERED=1 python3 -u -c');
}
});
test('adversarial review in /review always runs both passes', () => {
// Carved skill: the Step 4.8 adversarial body lives in sections/adversarial.md.
const content = readSkillUnion('review');
expect(content).toContain('Adversarial review (always-on)');
// Always-on: both Claude and Codex adversarial
expect(content).toContain('Claude adversarial subagent (always runs)');
expect(content).toContain('Codex adversarial challenge (runs whenever');
// Claude adversarial subagent dispatch
expect(content).toContain('Agent tool');
expect(content).toContain('FIXABLE');
expect(content).toContain('INVESTIGATE');
// Probe-based availability via the shared codexPreflight() (install + auth)
expect(content).toContain('CODEX_MODE');
expect(content).toContain('command -v codex'); // install check kept literal
// codex_reviews=disabled gates Codex passes only; Claude adversarial still runs
expectMentions(content, [['only', 'disabled', 'passes']], 'content');
// Review log
expect(content).toContain('adversarial-review');
expect(content).toContain('run-with-timeout --effort high ');
expect(content).toContain('ADVERSARIAL REVIEW SYNTHESIS');
// Large diff structured review still gated
expectMentions(content, [['only', 'structured', 'review']], 'content');
expect(content).toContain('200');
});
test('adversarial review in /ship runs the passes the review plan dispatches', () => {
const content = readShipUnion();
expect(content).toContain("Step 9's last full-pass plan governs this step");
expect(content).toContain('## Step 11: Adversarial review\n');
expect(content).toContain('adversarial-review');
expect(content).toContain('run-with-timeout --effort high ');
expect(content).toContain('Investigate and fix');
expect(content).toContain('Claude adversarial subagent (on `DISPATCH: adversarial-native`)');
});
test('scope drift detection in /review and /ship', () => {
const reviewContent = readSkillUnion('review');
const shipContent = readShipUnion();
// Both should contain scope drift from the shared resolver
for (const content of [reviewContent, shipContent]) {
expect(content).toContain('Scope Check:');
expect(content).toContain('DRIFT DETECTED');
expect(content).toContain('SCOPE CREEP');
expect(content).toContain('MISSING REQUIREMENTS');
expect(content).toContain('stated intent');
}
});
test('codex-host ship/review preserve adversarial review with a Claude outside voice', () => {
for (const skill of ['ship', 'review']) {
const content = readCodexSkillUnion(skill);
expect(content).not.toMatch(/codex\s+(?:exec|review)\s/);
expect(content).toContain('gstack-claude-code');
expect(content).toContain('codex_reviews');
expect(content).toContain('adversarial-review');
expect(content).toContain('Investigate and fix');
}
});
test('codex integration in /plan-eng-review offers plan critique', () => {
// Carved skill: the Codex outside-voice plan critique lives in
// sections/review-sections.md — read the skeleton+sections union. (The
// skeleton alone used to match "Codex" only via an inline-bash comment
// that the gstack-skill-start consolidation removed.)
const content = readSkillUnion('plan-eng-review');
expect(content).toContain('Codex');
expect(content).toContain('codex exec');
});
// D5 regression guard: the Codex outside voice is default-on, not opt-in. A future
// gen-skill-docs change must not silently reintroduce the "Want an outside voice?"
// AskUserQuestion. The CODEX_PLAN_REVIEW content renders into each skill's
// sections/review-sections.md (the skeleton points at it). plan-design-review uses
// DESIGN_OUTSIDE_VOICES, not CODEX_PLAN_REVIEW, so it is excluded here.
test('plan reviews run the Codex outside voice default-on (no opt-in question)', () => {
for (const skill of ['plan-eng-review', 'plan-ceo-review', 'plan-devex-review']) {
const content = fs.readFileSync(
path.join(ROOT, skill, 'sections', 'review-sections.md'), 'utf-8');
expect(content).not.toContain('Want an outside voice');
expect(content).toContain('CODEX_MODE');
expect(content).toContain('command -v codex'); // preflight install check (e2e relies on it)
}
});
test('/document-release includes the default-on Codex documentation review', () => {
// The doc-review renders into the carved release-body section (kept out of the
// always-loaded skeleton to respect the skeleton-byte budget).
const content = fs.readFileSync(
path.join(ROOT, 'document-release', 'sections', 'release-body.md'), 'utf-8');
expect(content).toContain('Codex Documentation Review (default-on)');
expect(content).toContain('CODEX_MODE');
expect(content).toContain('codex-doc-review');
});
test('codex-host document-release runs Claude Code and keeps the historical log identifier', () => {
const content = readCodexSkillUnion('document-release');
expect(content).toContain('Claude Code');
expect(content).toContain('gstack-claude-code');
expect(content).toContain('codex-doc-review');
expect(content).not.toMatch(/codex\s+(?:exec|review)\s/);
});
test('codex review invocations avoid the prompt plus --base argument shape', () => {
// The real invariant is "never pass a positional [PROMPT] together with a
// scope flag" — the CLI rejects that combination at argv parse time
// (#1428, #1479). Two different shapes satisfy it, and these files have
// diverged on which one they use:
//
// scoped — `codex review --base <base>` with NO prompt argument. The
// scope comes from the CLI, which is the only thing that actually sets
// it. This is what all three files now use.
// broken — prompt-only `codex review "<text>"` describing the diff
// range in prose. This parses, but the CLI falls back to *uncommitted
// working-tree* scope, so the review silently covers the wrong changes.
//
// The old assertion banned the substring `--base <base> -c '...'`, which
// the correct scoped form also contains — it could not tell the two apart,
// so it effectively banned the fix.
for (const rel of ['codex/SKILL.md', 'review/SKILL.md', 'ship/SKILL.md']) {
// ship's AND review's codex commands moved into sections/adversarial.md;
// codex's own scoped invocation lives in sections/review-mode.md (T9 carve).
const content = rel === 'ship/SKILL.md' ? readShipUnion()
: rel === 'review/SKILL.md' ? readSkillUnion('review')
: readSkillUnion('codex');
expect(content).toMatch(/codex\s+review\s+--base\b/);
const offending: string[] = [];
for (const line of content.split('\n')) {
if (line.includes('`codex review`')) continue;
const match = line.match(/(?:^|[;&|]\s*|\s)codex\s+review\b(.*)$/);
if (!match) continue;
const rest = match[1];
if (!/--base\b|--commit\b|--uncommitted\b/.test(rest)) continue;
const beforeFlag = rest.split(/--base\b|--commit\b|--uncommitted\b/)[0].trim();
// A quoted string or variable expansion before the scope flag is the bug.
if (/^["'$]|^--\s*["']/.test(beforeFlag)) offending.push(`${rel}: ${line.trim()}`);
}
expect(offending).toEqual([]);
}
});
test('codex review prompts always carry the filesystem boundary (#1503/#1522 regression)', () => {
// Pre-#1209, the bare `codex review --base` path stripped the filesystem
// boundary instruction, letting Codex spend tokens reading skill files.
// #1209's prompt rewrite restored the boundary by routing every default
// call through a prompt — but routing through a prompt is what breaks the
// diff scope, so codex/ no longer does that. What this test pins is the
// boundary TEXT, which must still be present for the paths that do take a
// prompt (`codex exec` for challenge, consult, and custom review focus).
// Do NOT "restore" the boundary by putting a prompt argument back on a
// scoped `codex review` call: that combination fails to parse, and
// dropping the scope flag to make it parse silently reviews the wrong diff.
const boundaryLine = /do not read or execute any files under ~\/\.claude\/, ~\/\.agents\/, \.claude\/skills\/, or agents\//i;
for (const rel of ['codex/SKILL.md', 'review/SKILL.md', 'ship/SKILL.md']) {
// ship's AND review's codex/adversarial boundary lines moved into sections/adversarial.md.
const content = rel === 'ship/SKILL.md' ? readShipUnion()
: rel === 'review/SKILL.md' ? readSkillUnion('review')
: fs.readFileSync(path.join(ROOT, rel), 'utf-8');
expect(content).toMatch(boundaryLine);
}
});
test('/review persists a review-log entry for ship readiness', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
expect(content).toContain('"skill":"review"');
expect(content).toContain('"issues_found":N');
expect(content).toContain('Persist Eng Review result');
});
test('Review Readiness Dashboard includes Adversarial Review row', () => {
const content = readShipUnion();
expect(content).toContain('Adversarial');
expect(content).toContain('codex-review');
});
});
// --- Trigger phrase validation ---
describe('Skill trigger phrases', () => {
// Skills that must have "Use when" trigger phrases in their description.
// Excluded: root gstack (browser tool), gstack-upgrade (gstack-specific),
// humanizer (text tool)
const SKILLS_REQUIRING_TRIGGERS = [
'qa', 'qa-only', 'ship', 'review', 'investigate', 'office-hours',
'plan-ceo-review', 'plan-eng-review', 'plan-design-review',
'design-review', 'design-consultation', 'retro', 'document-release',
'codex', 'browse', 'setup-browser-cookies', 'scrape',
];
for (const skill of SKILLS_REQUIRING_TRIGGERS) {
test(`${skill}/SKILL.md.tmpl has "Use when" trigger phrases`, () => {
const skillPath = path.join(ROOT, skill, 'SKILL.md.tmpl');
if (!fs.existsSync(skillPath)) return;
const content = fs.readFileSync(skillPath, 'utf-8');
// The catalog trim drops trigger prose from the Claude SKILL.md, so check
// the template description: --catalog-mode=full and the other hosts
// still route on it.
expect(content).toMatch(/Use when/i);
});
}
// Only the four core skills (the proactive: core default) may invite unasked
// suggestions; a non-core description that says "Proactively suggest" reaches
// every host that keeps full descriptions, under core too (#3446).
const SKILLS_REQUIRING_PROACTIVE = ['qa', 'ship', 'review', 'office-hours'];
for (const skill of SKILLS_REQUIRING_PROACTIVE) {
test(`${skill}/SKILL.md.tmpl has proactive routing phrase`, () => {
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md.tmpl'), 'utf-8');
// Same catalog-trim consideration — check the template description.
expect(content).toMatch(/Proactively (suggest|invoke)/i);
});
}
test('no non-core skill description invites proactive suggestions (#3446)', () => {
const offenders = fs.readdirSync(ROOT)
.filter(dir => !SKILLS_REQUIRING_PROACTIVE.includes(dir) && fs.existsSync(path.join(ROOT, dir, 'SKILL.md.tmpl')))
.filter(dir => {
const tmpl = fs.readFileSync(path.join(ROOT, dir, 'SKILL.md.tmpl'), 'utf-8');
return /Proactively (suggest|invoke)/i.test(tmpl.slice(0, tmpl.indexOf('\n---', 4)));
});
expect(offenders).toEqual([]);
});
});
// ─── Private-path leak detector ──────────────────────────────
//
// Catches accidental references to maintainer-private files in skill output.
// Adapted from the McGluut fork's skill-contract-audit.ts (we don't take the
// whole script — these are the unique checks not already covered by
// test/gen-skill-docs.test.ts:1668-2074 .claude/skills leakage tests).
describe('Private-path leak detection', () => {
const PRIVATE_PATTERNS: Array<{ pattern: RegExp; label: string }> = [
{ pattern: /coordination-board\.md/i, label: 'coordination-board.md' },
{ pattern: /SEEKING_LOG\.md/, label: 'SEEKING_LOG.md' },
{ pattern: /RATIONAL_SUBJECT\.md/, label: 'RATIONAL_SUBJECT.md' },
{ pattern: /VALUE_SIGNAL_LOOP\.md/, label: 'VALUE_SIGNAL_LOOP.md' },
{ pattern: /C:\\\\LLM Playground\\\\go/i, label: 'C:\\LLM Playground\\go' },
];
// Walk every SKILL.md and SKILL.md.tmpl in the repo (excluding node_modules,
// generated host outputs, and .git).
function discoverSkillSurface(): string[] {
const results: string[] = [];
function walk(dir: string) {
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
if (entry.name.startsWith('.') && entry.name !== '.agents') continue;
if (entry.name === 'node_modules' || entry.name === 'dist') continue;
const full = path.join(dir, entry.name);
if (entry.isDirectory()) {
walk(full);
} else if (entry.name === 'SKILL.md' || entry.name === 'SKILL.md.tmpl') {
results.push(full);
}
}
}
walk(ROOT);
return results;
}
test('no SKILL.md or SKILL.md.tmpl references private maintainer files', () => {
const files = discoverSkillSurface();
expect(files.length).toBeGreaterThan(0);
const leaks: string[] = [];
for (const file of files) {
const content = fs.readFileSync(file, 'utf-8');
for (const { pattern, label } of PRIVATE_PATTERNS) {
if (pattern.test(content)) {
leaks.push(`${path.relative(ROOT, file)} mentions ${label}`);
}
}
}
expect(leaks).toEqual([]);
});
});
// ─── Doc-inventory cross-check ───────────────────────────────
//
// Every skill directory (with a SKILL.md.tmpl) must appear in both AGENTS.md
// and docs/skills.md. Catches the inventory drift codex flagged (/debug
// → /investigate; missing /autoplan, /context-save, /plan-devex-review, etc.).
describe('Doc inventory cross-check', () => {
// Skills that don't get user-invocation lines in agent-facing docs.
// - 'qa-only' is a sub-mode of /qa with shared docs.
// - The 5 listed below are infrastructure (model overlays, shipped binary,
// hosts) that don't show up in the user-facing skill table.
const DOC_INVENTORY_EXCLUDE = new Set([
// Infra / non-skills
'agents', 'connect-chrome', 'contrib', 'hosts',
'lib', 'model-overlays', 'openclaw', 'supabase', 'scripts', 'test',
]);
function discoverSkillDirs(): string[] {
const dirs: string[] = [];
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
if (!entry.isDirectory()) continue;
if (entry.name.startsWith('.')) continue;
if (DOC_INVENTORY_EXCLUDE.has(entry.name)) continue;
const tmplPath = path.join(ROOT, entry.name, 'SKILL.md.tmpl');
if (fs.existsSync(tmplPath)) dirs.push(entry.name);
}
return dirs.sort();
}
test('every skill is documented in AGENTS.md', () => {
const agents = fs.readFileSync(path.join(ROOT, 'AGENTS.md'), 'utf-8');
const missing: string[] = [];
for (const skill of discoverSkillDirs()) {
// Match `/skill-name` as a token boundary.
if (!new RegExp(`/${skill}\\b`).test(agents)) missing.push(skill);
}
expect(missing).toEqual([]);
});
test('every skill is documented in docs/skills.md', () => {
const docs = fs.readFileSync(path.join(ROOT, 'docs', 'skills.md'), 'utf-8');
const missing: string[] = [];
for (const skill of discoverSkillDirs()) {
if (!new RegExp(`/${skill}\\b`).test(docs)) missing.push(skill);
}
expect(missing).toEqual([]);
});
});
// ─── Codex Skill Validation ──────────────────────────────────
describe('Codex skill validation', () => {
// .agents/ is gitignored (v0.11.2.0) — read from the module-level out-dir
// render (CODEX_OUT) instead of regenerating the live tree in place.
const AGENTS_DIR = path.join(CODEX_OUT, '.agents', 'skills');
// Discover all shared skills with templates.
// Host-exclusive outside-voice skills are intentionally omitted here:
// - /codex is unavailable on Codex
// - /claude-code is unavailable on Claude Code
const CLAUDE_SKILLS_WITH_TEMPLATES = (() => {
const skills: string[] = [];
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
if (entry.name === 'codex') continue; // Unavailable on Codex
if (entry.name === 'claude-code') continue; // Unavailable on Claude Code
if (fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) {
skills.push(entry.name);
}
}
return skills;
})();
test('shared skills have both Claude and Codex variants', () => {
for (const skillDir of CLAUDE_SKILLS_WITH_TEMPLATES) {
// Claude variant
const claudeMd = path.join(ROOT, skillDir, 'SKILL.md');
expect(fs.existsSync(claudeMd)).toBe(true);
// Codex variant
const codexName = skillDir.startsWith('gstack-') ? skillDir : `gstack-${skillDir}`;
const codexMd = path.join(AGENTS_DIR, codexName, 'SKILL.md');
expect(fs.existsSync(codexMd)).toBe(true);
}
// Root template has both too
expect(fs.existsSync(path.join(ROOT, 'SKILL.md'))).toBe(true);
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack', 'SKILL.md'))).toBe(true);
});
test('/codex skill has a Claude variant and no Codex variant', () => {
// Claude variant should exist
expect(fs.existsSync(path.join(ROOT, 'codex', 'SKILL.md'))).toBe(true);
// Codex variant must NOT exist
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-codex', 'SKILL.md'))).toBe(false);
});
test('Codex skill names follow gstack-{name} convention', () => {
const codexDirs = fs.readdirSync(AGENTS_DIR);
for (const dir of codexDirs) {
// Every directory should start with gstack
expect(dir.startsWith('gstack')).toBe(true);
// Root is just 'gstack', others are 'gstack-{name}'
if (dir !== 'gstack') {
expect(dir.startsWith('gstack-')).toBe(true);
}
}
});
test('$B commands in Codex SKILL.md files are valid browse commands', () => {
const codexDirs = fs.readdirSync(AGENTS_DIR);
for (const dir of codexDirs) {
const skillMd = path.join(AGENTS_DIR, dir, 'SKILL.md');
if (!fs.existsSync(skillMd)) continue;
const content = fs.readFileSync(skillMd, 'utf-8');
// Only validate if the skill contains $B commands
if (!content.includes('$B ')) continue;
const result = validateSkill(skillMd);
expect(result.invalid).toHaveLength(0);
}
});
});
// --- Repo mode and test failure triage validation ---
describe('Repo mode preamble validation', () => {
test('generated SKILL.md preamble contains REPO_MODE output', () => {
// Token-reduction Phase 1: the inline `gstack-repo-mode` call moved into
// bin/gstack-skill-start. The render pins the script invocation; the
// script pins the REPO_MODE echo + the gstack-repo-mode call.
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
expect(content).toContain('bin/gstack-skill-start');
const script = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-skill-start'), 'utf-8');
expect(script).toContain('REPO_MODE:');
expect(script).toContain('gstack-repo-mode');
});
test('tier 3+ skills contain See Something Say Something section', () => {
// Root SKILL.md is tier 1 (no Repo Mode). Check a tier 3 skill instead.
const content = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
expect(content).toContain('See Something, Say Something');
expect(content).toContain('REPO_MODE');
expect(content).toContain('solo');
expect(content).toContain('collaborative');
});
});
describe('Test failure triage in ship skill', () => {
test('ship/SKILL.md contains Test Failure Ownership Triage', () => {
const content = readShipUnion();
expect(content).toContain('Test Failure Ownership Triage');
});
test('ship/SKILL.md triage uses git diff for classification', () => {
const content = readShipUnion();
expect(content).toContain('git diff origin/<base>...HEAD --name-only');
});
test('ship/SKILL.md triage has solo and collaborative paths', () => {
const content = readShipUnion();
expect(content).toContain('REPO_MODE');
expect(content).toContain('solo');
expect(content).toContain('collaborative');
expect(content).toContain('Investigate and fix now');
expect(content).toContain('Add as P0 TODO');
});
test('ship/SKILL.md triage has GitHub issue assignment for collaborative mode', () => {
const content = readShipUnion();
expect(content).toContain('gh issue create');
expect(content).toContain('--assignee');
});
test('{{TEST_FAILURE_TRIAGE}} placeholder is fully resolved in ship/SKILL.md', () => {
const content = readShipUnion();
expect(content).not.toContain('{{TEST_FAILURE_TRIAGE}}');
});
test('ship/SKILL.md uses in-branch language for stop condition', () => {
const content = readShipUnion();
expectMentions(content, [['in-branch', 'unfixed', 'stop']], 'content');
});
});
describe('Failure routing: verified cause → fix, unknown cause → bounded triage (#3157, #3158)', () => {
const rule = generateFailureRouting();
const unknownCause = rule.slice(rule.indexOf('**Cause unknown:** ') + '**Cause unknown:** '.length);
test('the router sends bug reports through the failure rule, not straight to /investigate', () => {
const router = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
expect(router).toContain(rule);
const bugRule = router.split('\n').find(line => line.startsWith('- User reports a bug'));
expect(bugRule).toContain('**Failures**');
expect(bugRule).not.toMatch(/→ invoke `\/investigate`$/);
});
test('the rule keeps the root-cause bar on both paths', () => {
expect(rule).toMatch(/\*\*Cause supplied\*\*[^\n]*one read-only check[^\n]*If it does not, invoke `\/investigate` before any further tracing or edit/);
expect(unknownCause).toMatch(/^make one read-only triage pass[^\n]*Do not reproduce, test hypotheses or search wider[^\n]*still unclear, invoke `\/investigate` before any broader diagnosis or edit\.$/);
});
test('the routing gstack offers for CLAUDE.md, and this repo\'s own, check the cause before /investigate', () => {
const bugRule = readShippedSkillRouting().rules.find(line => line.startsWith('- Bugs/errors'));
expect(bugRule).toMatch(/→ invoke \/investigate, unless one read-only check confirms the cause/);
expect(fs.readFileSync(path.join(ROOT, 'CLAUDE.md'), 'utf-8').split('\n')).toContain(bugRule!);
});
test('/ship fixes a pre-existing failure through the same bounded triage', () => {
const tests = fs.readFileSync(path.join(ROOT, 'ship', 'sections', 'tests.md'), 'utf-8');
const fixNow = tests.slice(tests.indexOf('**If "Investigate and fix now":**'), tests.indexOf('**If "Add as P0 TODO":**'));
expect(fixNow).toContain('- One read-only root-cause pass, then `/investigate` if unclear.');
expect(fixNow).not.toContain('/investigate mindset');
});
});
describe('no compiled binaries in git', () => {
// Tracked files enumerated once and reused by both assertions. git ls-files -z
// + split is ~ms; the previous xargs-per-file shell loops blew past 5s on CI.
const trackedFiles: string[] = require('child_process')
.execSync('git ls-files -z', { cwd: ROOT, encoding: 'utf-8', timeout: 30_000 })
.split('\0')
.filter(Boolean);
test('git tracks no Mach-O or ELF binaries', () => {
// Only mode 100755 (executable) files can be binaries we care about. Pre-filter
// via git ls-files -s to avoid running `file` on every text file.
const lsOut: string = require('child_process').execSync('git ls-files -s', {
cwd: ROOT,
encoding: 'utf-8',
timeout: 30_000,
});
const executableFiles = lsOut
.split('\n')
.filter(Boolean)
.map((line: string) => {
const parts = line.split(/\s+/);
return { mode: parts[0], file: line.split('\t')[1] };
})
.filter((e: { mode: string; file: string }) => e.mode === '100755')
.map((e: { mode: string; file: string }) => e.file);
if (executableFiles.length === 0) return;
// Batch-invoke `file --mime-type` across all executable files at once.
const result: string = require('child_process')
.execSync(`file --mime-type -- ${executableFiles.map((f: string) => `'${f.replace(/'/g, "'\\''")}'`).join(' ')}`, {
cwd: ROOT,
encoding: 'utf-8',
timeout: 30_000,
})
.trim();
const binaries = result
.split('\n')
.filter((l: string) =>
/application\/(x-mach-binary|x-executable|x-pie-executable|x-sharedlib)/.test(l)
)
.map((l: string) => l.split(':')[0].trim());
expect(binaries).toEqual([]);
});
test('warns about tracked files larger than 2MB', () => {
// Large fixtures can be legitimate test infrastructure. Keep visibility on
// repository size without blocking those fixtures from living in git.
// Known-good fixtures are exempted from the warning to keep CI logs clean.
const MAX_BYTES = 2 * 1024 * 1024;
const knownLargeFixtures = new Set<string>([
// Currently empty — add repo-relative paths of intentionally-committed
// large fixtures here with a reason.
]);
const oversized = trackedFiles.flatMap((f: string) => {
if (knownLargeFixtures.has(f)) return [];
const full = path.join(ROOT, f);
try {
const size = fs.statSync(full).size;
return size > MAX_BYTES ? [{ file: f, size }] : [];
} catch {
return [];
}
});
if (oversized.length > 0) {
const formatted = oversized
.map(({ file, size }: { file: string; size: number }) => {
const mib = (size / (1024 * 1024)).toFixed(1);
return `${file} (${mib} MiB)`;
})
.join(', ');
console.warn(`[size-warning] tracked files over 2 MiB: ${formatted}`);
}
expect(Array.isArray(oversized)).toBe(true);
});
});
// ─── Browser-skills validation ──────────────────────────────────
//
// Browser-skills are bundled in <gstack-root>/browser-skills/<name>/. Each
// must have a SKILL.md whose frontmatter satisfies the contract enforced by
// browse/src/browser-skills.ts:parseSkillFile (host required, args + triggers
// parseable as the right shape). This test catches malformed bundled skills
// at CI time, before they ship.
describe('Bundled browser-skills frontmatter contract', () => {
const browserSkillsRoot = path.join(ROOT, 'browser-skills');
function listBundledSkillDirs(): string[] {
if (!fs.existsSync(browserSkillsRoot)) return [];
return fs.readdirSync(browserSkillsRoot)
.filter(name => !name.startsWith('.'))
.map(name => path.join(browserSkillsRoot, name))
.filter(dir => {
try { return fs.statSync(dir).isDirectory(); } catch { return false; }
});
}
test('each bundled skill has a SKILL.md', () => {
for (const dir of listBundledSkillDirs()) {
const skillFile = path.join(dir, 'SKILL.md');
expect(fs.existsSync(skillFile)).toBe(true);
}
});
test('each bundled skill SKILL.md frontmatter parses with required fields', async () => {
const { parseSkillFile } = await import('../browse/src/browser-skills');
for (const dir of listBundledSkillDirs()) {
const name = path.basename(dir);
const content = fs.readFileSync(path.join(dir, 'SKILL.md'), 'utf-8');
// parseSkillFile throws on missing required fields; we just want to
// make sure none of our shipped skills tripwire it.
const { frontmatter } = parseSkillFile(content, { skillName: name });
expect(frontmatter.name).toBe(name);
expect(typeof frontmatter.host).toBe('string');
expect(frontmatter.host.length).toBeGreaterThan(0);
expect(Array.isArray(frontmatter.triggers)).toBe(true);
expect(Array.isArray(frontmatter.args)).toBe(true);
}
});
test('each bundled skill has a script.ts', () => {
for (const dir of listBundledSkillDirs()) {
expect(fs.existsSync(path.join(dir, 'script.ts'))).toBe(true);
}
});
test('each bundled skill ships a sibling SDK at _lib/browse-client.ts', () => {
for (const dir of listBundledSkillDirs()) {
expect(fs.existsSync(path.join(dir, '_lib', 'browse-client.ts'))).toBe(true);
}
});
test('each bundled skill has a script.test.ts', () => {
for (const dir of listBundledSkillDirs()) {
expect(fs.existsSync(path.join(dir, 'script.test.ts'))).toBe(true);
}
});
test("each bundled skill's _lib/browse-client.ts matches the canonical SDK", () => {
// If the canonical SDK changes, the bundled copy must be updated. This
// test enforces that — the _lib copy should be byte-identical.
const canonical = fs.readFileSync(path.join(ROOT, 'browse', 'src', 'browse-client.ts'), 'utf-8');
for (const dir of listBundledSkillDirs()) {
const sibling = fs.readFileSync(path.join(dir, '_lib', 'browse-client.ts'), 'utf-8');
expect(sibling).toBe(canonical);
}
});
test('script.ts imports browse from ./_lib/browse-client', () => {
for (const dir of listBundledSkillDirs()) {
const content = fs.readFileSync(path.join(dir, 'script.ts'), 'utf-8');
expect(content).toMatch(/from\s+['"]\.\/_lib\/browse-client['"]/);
}
});
});
// Token-reduction Phase 5: the four ios skills demoted from preamble-tier 3
// to 2 — they never consume the tier-3 sections (repo-mode ownership, search
// before building) but DO fire AskUserQuestion, which tier >=2 provides.
describe('ios tier demotion (Phase 5)', () => {
const iosSkills = ['ios-fix', 'ios-clean', 'ios-sync', 'ios-design-review'];
for (const skill of iosSkills) {
test(`${skill} render drops tier-3 sections, keeps AUQ format`, () => {
const content = readSkillStartup(path.join(ROOT, skill, 'SKILL.md'));
expect(content).not.toContain('## Repo Ownership');
expect(content).not.toContain('## Search Before Building');
expect(content).toContain('## AskUserQuestion Format');
});
}
});