Files
gstack/test/skill-validation.test.ts
Garry Tan e8893a18b1 v1.20.0.0 feat: browser-skills runtime + gbrain-support carryover (#1233)
* feat(gbrain-sync): queue primitives + writer shims

Adds bin/gstack-brain-enqueue (atomic append to sync queue) and
bin/gstack-jsonl-merge (git merge driver, ts-sort with SHA-256 fallback).
Wires one backgrounded enqueue call into learnings-log, timeline-log,
review-log, and developer-profile --migrate. question-log and
question-preferences stay local per Codex v2 decision.

gstack-config gains gbrain_sync_mode (off/artifacts-only/full) and
gbrain_sync_mode_prompted keys, plus GSTACK_HOME env alignment so
tests don't leak into real ~/.gstack/config.yaml.

* feat(gbrain-sync): --once drain + secret scan + push

bin/gstack-brain-sync is the core sync binary. Subcommands: --once
(drain queue, allowlist-filter, privacy-class-filter, secret-scan
staged diff, commit with template, push with fetch+merge retry),
--status, --skip-file <path>, --drop-queue --yes, --discover-new
(cursor-based detection of artifact writes that skip the shim).

Secret regex families: AWS keys, GitHub tokens (ghp_/gho_/ghu_/ghs_/
ghr_/github_pat_), OpenAI sk-, PEM blocks, JWTs, bearer-token-in-JSON.
On hit: unstage, preserve queue, print remediation hint (--skip-file
or edit), exit clean. No daemon — invoked by preamble at skill
boundaries.

* feat(gbrain-sync): init, restore, uninstall, consumer registry

bin/gstack-brain-init: idempotent first-run. git init ~/.gstack/,
.gitignore=*, canonical .brain-allowlist + .brain-privacy-map.json,
pre-commit secret-scan hook (defense-in-depth), merge driver registration
via git config, gh repo create --private OR arbitrary --remote <url>,
initial push, ~/.gstack-brain-remote.txt for new-machine discovery,
GBrain consumer registration via HTTP POST.

bin/gstack-brain-restore: safe new-machine bootstrap. Refuses clobber
of existing allowlisted files, clones to staging, rsync-copies tracked
files, re-registers merge drivers (required — not cloned from remote),
rehydrates consumers.json, prompts for per-consumer tokens.

bin/gstack-brain-uninstall: clean off-ramp. Removes .git + .brain-*
files + consumers.json + config keys. Preserves user data (learnings,
plans, retros, profile). Optional --delete-remote for GitHub repos.

bin/gstack-brain-consumer + bin/gstack-brain-reader (symlink alias):
registry management. Internal 'consumer' term; user-facing 'reader'
per DX review decision.

* feat(gbrain-sync): preamble block — privacy gate + boundary sync

scripts/resolvers/preamble/generate-brain-sync-block.ts emits bash that
runs at every skill invocation:
- Detects ~/.gstack-brain-remote.txt on machines without local .git
  and surfaces a restore-available hint (does NOT auto-run restore).
- Runs gstack-brain-sync --once at skill start to drain any pending
  writes (and at skill end via prose instruction).
- Once-per-day auto-pull (cached via .brain-last-pull) for append-only
  JSONL files.
- Emits BRAIN_SYNC: status line every skill run.

Also emits prose for the host LLM to fire the one-time privacy
stop-gate (full / artifacts-only / off) when gbrain is detected and
gbrain_sync_mode_prompted is false. Wired into preamble.ts composition.

* test(gbrain-sync): 27-test consolidated suite

test/brain-sync.test.ts covers:
- Config: validation, defaults, GSTACK_HOME env isolation
- Enqueue: no-op gates, skip list, concurrent atomicity, JSON escape
- JSONL merge driver: 3-way + ts-sort + SHA-256 fallback
- Init + sync: canonical file creation, merge driver registration,
  push-reject + fetch+merge retry path
- Init refuses different remote (idempotency)
- Cross-machine restore round-trip (machine A write → machine B sees)
- Secret scan across all 6 regex families (AWS, GH, OpenAI, PEM, JWT,
  bearer-JSON). --skip-file unblock remediation
- Uninstall removes sync config, preserves user data
- --discover-new idempotence via mtime+size cursor

Behaviors verified via integration smokes during implementation. Known
follow-up: bun-test 5s default timeout needs 30s wrapper for
spawnSync-heavy tests.

* docs(gbrain-sync): user guide + error lookup + README section

docs/gbrain-sync.md: setup walkthrough, privacy modes, cross-machine
workflow, secret protection, two-machine conflict handling, uninstall,
troubleshooting reference.

docs/gbrain-sync-errors.md: problem/cause/fix index for every
user-visible error. Patterned on Rust's error docs + Stripe's API
error reference.

README.md: 'Cross-machine memory with GBrain sync' section near the
top (discovery moment), plus docs-table entry.

* chore: bump version and changelog (v1.7.0.0)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: regenerate SKILL.md files for gbrain-sync preamble block

Re-runs bun run gen:skill-docs after adding generateBrainSyncBlock
to scripts/resolvers/preamble.ts in a2aa8a07. CI check-freshness
caught the drift. All 36 SKILL.md files regenerated with the new
skill-start bash block + privacy-gate prose + skill-end sync
instructions baked in.

* fix(test): session-awareness reads AskUserQuestion Format from a Tier 2+ SKILL.md

The test was reading ROOT/SKILL.md (browse skill, Tier 1) which never
contained '## AskUserQuestion Format' — that section is only emitted
for Tier 2+ skills by scripts/resolvers/preamble.ts. As a result the
agent was prompted with an empty format guide and only emitted
'RECOMMENDATION' intermittently, making the test flaky.

Pre-existing on main (same ROOT/SKILL.md shape there) — surfaced now
because the agent run didn't hit the RECOMMENDATION/recommend/option a
fallback strings in this particular attempt.

Fix: read from office-hours/SKILL.md (Tier 3, always has the section)
with a fallback that scans for the first top-level skill dir whose
SKILL.md contains the header. Future template moves won't break this
test again.

* feat(browse): domain-skills storage + state machine

New module browse/src/domain-skills.ts implements the per-site notes
the agent writes for itself, persisted as type:"domain" rows alongside
/learn's per-project learnings.

Three scopes layered: per-project default, global by explicit promotion.
Project-active shadows global for the same host.

State machine (T6 — codex outside-voice):
  quarantined --3 uses w/o flag--> active(project) --promote--> global
        ^                                |
        +----- classifier flag during use

- Append-only JSONL with O_APPEND for atomic small writes
- Tolerant parser drops partial trailing line on read
- Tombstone for deletes (compactor cleans up later)
- Version log per (host, scope) enables rollback
- Hostname derived from active tab top-level origin (T3 confused-deputy fix)
- writeSkill rejects classifier_score >= 0.85 with structured error

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(browse): domain-skills storage + state machine

14 tests covering:
- T3 hostname normalization (lowercase, www. strip, port/path/query strip,
  subdomain-exact preserved)
- T4 scope shadowing (per-project active shadows global for same host)
- T5 persistence (version monotonicity, tolerant parser drops partial line)
- T6 state machine (quarantined → active after N=3 uses, classifier-flag
  blocks promotion, save-time score >= 0.85 rejected)
- Rollback by version log (restore prior body, advance version counter)
- Tombstone deletion (read returns null after delete)

All 14 pass in 27ms via bun test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browse): $B domain-skill subcommands

Wire the domain-skills storage layer into the browse CLI as a META command:

  $B domain-skill save              save body from stdin or --from-file
                                     (host derived from active tab — T3)
  $B domain-skill list              list all skills visible to current project
  $B domain-skill show <host>       print skill body
  $B domain-skill edit <host>       open in $EDITOR
  $B domain-skill promote-to-global <host>  cross-project promotion (T4)
  $B domain-skill rollback <host> [--global]  restore prior version
  $B domain-skill rm <host> [--global]        tombstone

Save path runs L1-L3 content filters from content-security.ts (importable
in compiled binary, unlike L4 ML classifier — see CLAUDE.md). The L4
classifier scan happens in sidebar-agent at prompt-injection load time.

Output is structured (problem + cause + suggested-action) per DX D7.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browse): $B cdp escape hatch — deny-default allowlist + two-tier mutex

Codex T2: flip CDP posture to deny-default. Allowed methods enumerated in
cdp-allowlist.ts with (scope: tab|browser, output: trusted|untrusted,
justification) per entry.

Initial allowlist (~25 methods) covers:
- Accessibility tree extraction (read-only)
- DOM/CSS inspection (read-only)
- Performance metrics
- Tracing
- Emulation viewport/UA override
- Page screenshot/PDF capture (output is binary, no marker injection vector)
- Network.enable/disable (no bodies/cookies — those are exfil surfaces)
- Runtime.getProperties (NO evaluate/callFunctionOn — those would be RCE)

Page.navigate is INTENTIONALLY NOT allowed; agents use $B goto which
goes through the URL blocklist.

Codex T7: two-tier mutex. tab-scoped methods take per-tab lock; browser-
scoped take global lock that blocks all tab locks. 5s acquire timeout
yields CDPMutexAcquireTimeout (no silent hangs). All lock acquires use
try/finally so errors don't leak the lock.

Path A from spike: uses Playwright's newCDPSession() per page. No second
WebSocket, no need for --remote-debugging-port. CDPSession is cached
per page in a WeakMap and cleared on page close.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(browse): CDP allowlist + two-tier mutex

13 tests:
- Allowlist linter: every entry has 4 required fields, no duplicates,
  justification length > 20 chars
- Deny-list verification: dangerous methods (Runtime.evaluate, Page.navigate,
  Network.getResponseBody, Browser.close, Target.attachToTarget, etc.) are
  NOT allowed (Codex T2 categories 4-7)
- Per-tab mutex serializes ops on same tab
- Per-tab mutex allows parallel ops across different tabs
- Global lock blocks tab locks; tab locks block global lock
- Acquire timeout yields CDPMutexAcquireTimeout (no silent hang)
- Timeout error names the tab id and the timeout budget

Also extends Network.disable justification to satisfy linter.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browse): telemetry signals + project-slug helper

Lightweight telemetry per DX D9: piggybacks on ~/.gstack/analytics/ pattern.
Hostname + aggregate counters only, no body content. GSTACK_TELEMETRY_OFF=1
silences. Fire-and-forget — never blocks calling path.

Signals fired so far:
- domain_skill_saved {host, scope, state, bytes}
- domain_skill_save_blocked {host, reason}

(domain_skill_fired and cdp_method_* fired in subsequent commits.)

Also extracts project-slug resolution into project-slug.ts so server.ts
and domain-skill-commands.ts share one cached lookup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browse): sidebar prompt-context injection + CDP telemetry

server.ts spawnClaude now:
- Imports per-project domain skill matching the active tab's hostname
  via readDomainSkill()
- Wraps the body in UNTRUSTED EXTERNAL CONTENT envelope (so the L4
  classifier in sidebar-agent sees it at load time per Eng D4)
- Appends as <domain-skill source="..." host="..." version="..."> block
- Fires domain_skill_fired telemetry (host, source, version)
- Calls recordSkillUse fire-and-forget so the auto-promote-after-N=3
  state machine advances on each successful prompt injection

System prompt also gets a one-liner introducing $B domain-skill commands
to agents (DX D4 start-of-task discoverability hint).

cdp-bridge.ts fires:
- cdp_method_denied (drives next allow-list growth)
- cdp_method_lock_acquire_ms (P50/P99 quantile observability)
- cdp_method_called (allowed methods)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(browse): telemetry module

3 tests covering:
- logTelemetry writes JSONL with ts injected
- GSTACK_TELEMETRY_OFF=1 silences all events
- logTelemetry never throws on disk failures

Uses GSTACK_HOME env var to redirect writes to a tmp dir; the telemetry
module reads HOME lazily so test mutations take effect.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: domain-skills reference + error lookup table

docs/domain-skills.md mirrors the layered shape of docs/gbrain-sync.md
(DX D8): how agents use it, state machine, storage layout, security model
(L1-L3 + L4 layered defense), error reference table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(readme): browser-harness-js plug + domain-skills section

New "Domain skills + raw CDP escape hatch" section under "The sprint"
covering both v1.8.0.0 features. Plugs browser-use/browser-harness-js
as the no-rails alternative for users who want raw CDP without gstack's
security stack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v1.8.0.0)

Branch-scoped bump on top of merged 1.7.0.0 base. CHANGELOG entry covers
the full v1.8.0.0 scope: $B domain-skill, $B cdp escape hatch, two-tier
mutex, telemetry signals, sidebar prompt-context injection. Includes
Codex outside-voice trail (7 of 20 findings resolved, 12 mooted by T1
scope drop).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* todos: 7 follow-ups from v1.8.0.0 review trail

P1: Self-authoring $B commands with out-of-process worker isolation
    (Codex T1 deferred from v1.8.0.0 — needs real isolation design)
P2: Migrate /learn to SQLite (Codex T5 long-term primitive fix)
P2: Remove plan-mode handshake from /plan-devex-review (skill bug)
P3: GBrain skillpack publishing for domain-skills
P3: Replay/record demonstrated flows to domain-skills
P3: $B commands review batch-mode UX (alternative to inline approval)
P3: Heuristic command-gap watcher (DX D4 alternative C)

Each entry has the standard What/Why/Pros/Cons/Context/Effort/Priority/
Depends-on shape so anyone picking these up later has full context.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(browse): lazy GSTACK_HOME resolution in domain-skills

Module-level constants (GLOBAL_FILE, derived path) were evaluated at
module-load and cached. When E2E and unit tests run in the same Bun
test pass and set GSTACK_HOME differently, the second test sees the
first test's path. Switch to lazy gstackHome() / globalFile() / projectFile()
helpers so process.env mutations take effect.

Mirrors the pattern already used in telemetry.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(browse): E2E gate-tier tests for domain-skills + CDP

domain-skills-e2e.test.ts (4 tests):
- save derives host from active tab top-level origin (T3)
- save lands quarantined; list surfaces it
- readSkill returns null until 3 uses without flag promote to active (T6)
- save without an active page errors with structured guidance

cdp-e2e.test.ts (8 tests):
- Accessibility.getFullAXTree returns wrapped JSON (allowed, untrusted-output)
- Performance.getMetrics returns plain JSON (allowed, trusted-output)
- Runtime.evaluate DENIED with structured guidance (T2 RCE block)
- Page.navigate DENIED (must use $B goto for blocklist routing)
- Network.getResponseBody DENIED (exfil block)
- malformed JSON params surfaces clear error
- non Domain.method format surfaces clear error
- $B cdp help returns help text

Both files boot a real Chromium via BrowserManager.launch() and exercise
the dispatch handlers end-to-end. Total 12 E2E tests in <2s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: regenerate SKILL.md files with new $B commands

bun run gen:skill-docs picks up the domain-skill and cdp META_COMMANDS
entries added in commands.ts. Both top-level SKILL.md and browse/SKILL.md
now list the new commands in their Meta and Inspection tables.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(fixtures): regenerate ship SKILL.md golden baselines for v1.7.0.0

Pre-existing failures inherited from garrytan/gbrain-support: the GBrain
Sync preamble block (added in v1.7.0.0) appears in regenerated SKILL.md
output but the golden baselines in test/fixtures/golden/ were never
updated. Three failures fixed:

  golden-file regression > Claude ship skill matches golden baseline
  golden-file regression > Codex ship skill matches golden baseline
  golden-file regression > Factory ship skill matches golden baseline

Goldens regenerated by copying the current ship/SKILL.md, codex
.agents/skills/gstack-ship/SKILL.md, and .factory/skills/gstack-ship/SKILL.md
files. Diff is the v1.7.0.0 GBrain Sync preamble block + privacy stop-gate
(no behavioral changes — just preamble text).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(brain-sync): bearer-token regex catches values with leading space

Pre-existing bug from v1.7.0.0: the bearer-token-json secret pattern
required values matching [A-Za-z0-9_./+=-]{16,}, which rejected the
"Bearer <token>" form because the literal space after "Bearer" wasn't
in the character class. Real Authorization headers use "Bearer <token>"
syntax, and the test fixture
  '"authorization":"Bearer abcdef1234567890abcdef1234567890"'
sat unscanned despite being a leak-class secret.

One-character fix: add space to the value character class. Test
'gstack-brain-sync secret scan > blocks bearer-json' now passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(brain-sync): GSTACK_HOME isolation test compares mtime, not content

Pre-existing flaky test: the GSTACK_HOME-overrides-real-config test asserted
the real ~/.gstack/config.yaml does NOT contain "gbrain_sync_mode: full"
after the test. That fails for any user whose real config legitimately has
that key set from prior usage — the test's invariant is "the command did
not modify the real file," not "the real file lacks any specific value."

Switch to mtime + content snapshot: capture both BEFORE running the command,
then verify both are unchanged after. Also add a positive assertion that
the tmpHome config DID get the new key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(skill-validation): exempt deliberate large fixtures from 2MB limit

Pre-existing failure: the "git tracks no files larger than 2MB" test
caught browse/test/fixtures/security-bench-haiku-responses.json (28.8MB
of replay data committed in v1.6.4.0 for security benchmark gate tests).

The test exists to catch accidentally-committed binaries (Mach-O dist
binaries, etc), not to forbid all large files. Add an explicit
LARGE_FIXTURE_EXEMPTIONS allowlist so deliberate replay fixtures pass
the gate while accidental binaries still fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(skill-token): mint scoped tokens per skill spawn

Wraps token-registry.createToken/revokeToken with skill-specific
clientId encoding (skill:<name>:<spawn-id>) and read+write defaults.
Skill scripts get a per-spawn capability token bound to browser-driving
commands; the daemon root token never leaves the harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browse-client): SDK for browser-skill scripts

Thin wrapper over POST /command with bearer auth. Resolves daemon
port + token from GSTACK_PORT + GSTACK_SKILL_TOKEN env vars first
(set by $B skill run when spawning), falls back to .gstack/browse.json
for standalone debug runs.

Convenience methods cover the read+write surface skills typically need:
goto, click, fill, text, html, snapshot, links, forms, accessibility,
attrs, media, data, scroll, press, type, select, wait, hover, screenshot.
Low-level command(cmd, args) escape hatch for anything else.

This is the canonical SDK source. Each browser-skill ships a sibling
copy at <skill>/_lib/browse-client.ts so each skill is fully portable
and version-pinned.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browser-skills): 3-tier storage helpers

listBrowserSkills() walks project > global > bundled (first-wins),
parses SKILL.md frontmatter, no INDEX.json. readBrowserSkill() does
the same for a single name. tombstoneBrowserSkill() moves a skill
into .tombstones/<name>-<ts>/ for recoverability.

Frontmatter parser handles the subset browser-skills need: scalars
(host, description, trusted, version, source), string lists
(triggers), and arg-mapping lists ([{name, description}, ...]).
Quoted values handle colons; trusted defaults to false.

Bundled tier path is auto-detected from the binary install location;
project tier comes from git rev-parse; global is ~/.gstack/. All tier
paths are overridable for hermetic tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browser-skills): \$B skill list/show/run/test/rm subcommands

handleSkillCommand dispatches to per-subcommand handlers; spawnSkill is
the load-bearing function that:

  1. Mints a per-spawn scoped token (read+write only) bound to the
     skill name + spawn-id.
  2. Builds the spawn env:
     - trusted: passes process.env minus GSTACK_TOKEN (defense in depth).
     - untrusted: minimal allowlist (LANG, LC_ALL, TERM, TZ) + locked
       PATH; explicitly drops anything matching TOKEN/KEY/SECRET/etc.
       Also drops AWS_/AZURE_/GCP_/GOOGLE_APPLICATION_/ANTHROPIC_/OPENAI_/
       GITHUB_/GH_/SSH_/GPG_/NPM_TOKEN/PYPI_ patterns.
   3. Always injects GSTACK_PORT + GSTACK_SKILL_TOKEN last (cannot be
     overridden by parent env).
  4. Spawns bun run script.ts -- <args> with cwd=skillDir, captures
     stdout (1MB cap), stderr, and timeout-kills past the deadline.
  5. Revokes the token in finally{}, always.

list output prints the resolved tier inline so "why did it run that
one?" never becomes a debugging mystery (Codex finding #4 mitigation).

server.ts threads the listen port to meta-commands via MetaCommandOpts.daemonPort.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browser-skills): bundled hackernews-frontpage reference skill

Smallest interesting browser-skill: scrapes HN front page, returns
30 stories as JSON. No auth, stable HTML, fully fixture-tested.

Files:
  SKILL.md                          frontmatter + prose
  script.ts                         exports parseStoriesFromHtml(html)
                                    main: goto + html + parse + JSON.stringify
  _lib/browse-client.ts             vendored copy of the SDK
  fixtures/hn-2026-04-26.html       captured front page (5 stories)
  script.test.ts                    13 assertions against the fixture

The parser is a pure function over HTML so script.test.ts runs
without a daemon (just imports parseStoriesFromHtml and asserts).

This exercises every Phase 1 component end-to-end:
  - browse-client SDK (script imports browse from ./_lib/)
  - 3-tier lookup (hackernews-frontpage lives in the bundled tier)
  - scoped tokens (read+write is enough for goto + html)
  - spawn lifecycle (\$B skill run hackernews-frontpage)
  - file-fixture testing (\$B skill test hackernews-frontpage)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(skill-validation): cover bundled browser-skills

Adds 7 assertions per bundled skill at <root>/browser-skills/<name>/:
  - SKILL.md exists
  - frontmatter parses with required fields (name/host/triggers/args)
  - script.ts exists
  - _lib/browse-client.ts exists and matches the canonical SDK byte-for-byte
  - script.test.ts exists
  - script.ts imports browse from ./_lib/browse-client

The byte-identical SDK check enforces the version-pinning contract:
when the canonical SDK at browse/src/browse-client.ts changes, every
bundled skill's _lib/ copy must be re-synced or this test fails.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(designs): add BROWSER_SKILLS_V1 design doc

Captures the 13 locked decisions, two-axis trust model (daemon-side
scoped tokens + process-side env access), 3-tier lookup, file
layout, and full responses to all 8 Codex outside-voice findings.
Includes Phase 2-4 sketches for future branches.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(todos): replace self-authoring-\$B P1 with browser-skills phases

Phase 1 of the browser-skills design shipped on this branch (sidesteps
the in-daemon isolation problem the original P1 was blocked on). The
new entries enumerate the work that remains:

  P1: Phase 2 (/scrape + /automate skill templates)
  P2: Phase 3 (resolver injection at session start)
  P2: Phase 4 (eval infra + fixture staleness + OS sandbox)

Cross-references docs/designs/BROWSER_SKILLS_V1.md for the full
architecture and the 8 Codex review findings + responses.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* release: v1.9.0.0 — browser-skills runtime

VERSION 1.8.0.0 → 1.9.0.0. CHANGELOG entry leads with what humans
can do today (hand-write deterministic browser scripts, run them in
200ms via \$B skill run). Notes explicitly that agent authoring
lands in next release; no fabricated perf numbers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(browser-skills-e2e): exercise dispatch with bundled hackernews-frontpage

Covers the full \$B skill list/show/test pipeline against the real
bundled reference skill (defaultTierPaths picks up <repo>/browser-skills/).
Verifies frontmatter shape, the three-tier walk surfaces the bundled
entry, and \$B skill test successfully runs the bundled script.test.ts
in a child bun process.

\$B skill run end-to-end against the live network is intentionally NOT
covered here (would be flaky against news.ycombinator.com); the spawn
lifecycle is exercised in browser-skill-commands.test.ts using inline
synthetic skills.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: regen SKILL.md to surface the skill META command

bun run gen:skill-docs picked up the new \`skill\` command from
COMMAND_DESCRIPTIONS in browse/src/commands.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* release: bump v1.9.0.0 → v1.13.0.0

Main shipped through v1.11.1.0 while this branch was in flight; v1.12.x
is presumed claimed by another in-flight branch. Use v1.13.0.0 as the
next available slot.

Updated VERSION, package.json, and the CHANGELOG header. Entry body
unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* release: bump v1.13.0.0 → v1.16.0.0

Main shipped v1.13.0.0 (claude outside-voice skill), v1.14.0.0
(sidebar REPL), and v1.15.0.0 (slim preamble + plan-mode E2E)
while this branch was in flight. Use v1.16.0.0 as the next
available slot.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(browse-skills): atomic write helper for /skillify (D3)

stageSkill writes a candidate skill into ~/.gstack/.tmp/skillify-<spawnId>/
with restrictive perms. commitSkill does an atomic fs.renameSync into the
final tier path with realpath/lstat discipline (refuses symlinked staging
dirs, refuses to clobber existing skills). discardStaged is the cleanup
path for test failures and approval rejections, idempotent and bounded
to the per-spawn wrapper. validateSkillName enforces lowercase/digits/
dashes only, no path-escape characters.

Implements the D3 contract from the v1.19.0.0 plan review: never a
half-written skill on disk. Test fail or approval reject = rm -rf the
temp dir, no tombstone for never-approved skills.

Closes Codex finding #5 (atomic skill packaging) for Phase 2a.

34 unit assertions covering: stage validation, file-path escape rejection,
permission check, atomic rename, clobber refusal, symlink refusal, project
tier unresolved, idempotent discard, end-to-end happy + simulated test
failure + approval reject paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(scrape): /scrape <intent> skill template

One entry point for pulling page data. Three paths under the hood:

1. Match — agent reads $B skill list, semantically matches the user's
   intent against each skill's triggers + description + host. Confident
   match = $B skill run <name> in ~200ms.
2. Prototype — no match, drive the page with $B goto/text/html/links etc.
   Return JSON, append a one-line "say /skillify" nudge.
3. Mutating refusal — verbs like submit/click/fill route to /automate
   (Phase 2b P0); /scrape is read-only by contract.

Match decision lives in the agent, not the daemon. No new code in
browse/src/, no expanded daemon command surface, no new prompt-injection
blast radius.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(skillify): /skillify codifies last /scrape into permanent skill

The productivity multiplier. /scrape discovers the flow; /skillify writes
it as deterministic Playwright-via-browse-client code so the next /scrape
on the same intent runs in ~200ms.

11-step flow with three locked contracts from the v1.19.0.0 plan review:

D1 — Provenance guard. Walk back ≤10 agent turns for a clearly-bounded
/scrape result. Refuse with one specific message if cold. No silent
synthesis from chat fragments.

D2 — Synthesis input slice. Extract ONLY the final-attempt $B calls that
produced the JSON the user accepted, plus the user's intent string. Drop
failed selectors, drop unrelated chat, drop earlier-session content.
Closes Codex finding #6 by picking option (b) from the design doc:
re-prompt from agent's own context, not a structured recorder.

D3 — Atomic write. Stage to ~/.gstack/.tmp/skillify-<spawnId>/, run
$B skill test against the temp dir, only rename into the final tier path
on test pass + user approval. Test fail or approval reject = rm -rf the
temp dir entirely.

Default tier: global (~/.gstack/browser-skills/<name>/). --project flag
overrides to per-project. Generated test must include at least one ★★
assertion (parsed JSON has expected shape + non-empty key fields), not a
smoke ★ assertion.

Bun runtime distribution (Codex finding #7) carries over to Phase 4.
Documented in the skill's Limits section.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(browser-skills): gate-tier E2E for /scrape + /skillify (D4)

Five scenarios cover the productivity loop and the contracts locked
during the v1.19.0.0 plan review:

  scrape-match-path           — intent matching bundled hackernews-frontpage
                                routes via $B skill run, no prototype phase
  scrape-prototype-path       — no matching skill, drives $B against a local
                                file:// fixture, returns JSON, suggests
                                /skillify
  skillify-happy-path         — /scrape then /skillify; skill written to
                                ~/.gstack/browser-skills/<name>/ with the
                                full file tree; SKILL.md prose body must
                                not contain conversation fragments (D2)
  skillify-provenance-refusal — cold /skillify with no prior /scrape refuses
                                with the D1 message; nothing on disk (D1)
  skillify-approval-reject    — /scrape then /skillify but reject in the
                                approval gate; temp dir is removed, nothing
                                at the final tier path (D3)

All five gate-tier (~$0.50-$1.50 each, ~$5 total per CI run). Set EVALS=1
to enable. Uses local file:// fixtures so prototype + skillify scenarios
run deterministically without network.

Touchfiles registers all 5 entries with proper deps on scrape/**,
skillify/**, browse/src/browser-skill-write.ts, and the Phase 1 runtime
modules. The match-path test depends on the bundled hackernews-frontpage
skill so its touchfile includes browser-skills/hackernews-frontpage/**.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(browser-skills): TODOS Phase 2a + design doc D1-D4 decisions

TODOS.md:
- Narrows existing P1 (was "/scrape and /automate") to "/scrape and
  /skillify" — the /scrape + /skillify wedge ships in this branch.
  Codex finding #6 (synthesis) removed from Cons (resolved by D2);
  finding #7 (Bun runtime) stays as the open carry-over.
- Adds new ## P0 above PACING_UPDATES_V0 for the /automate follow-up.
  Same skillify pattern as /scrape, different trust profile (per-step
  confirmation gate when running non-codified). Reuses /skillify and
  the D3 helper as-is. Effort M.

BROWSER_SKILLS_V1.md:
- Phase table re-organized into 1, 2a, 2b, 3, 4. Phase 1 + Phase 2a
  consolidate into v1.19.0.0 ship (the v1.16.0.0 branch-internal
  bump never landed on main).
- New "Phase 2a" sub-section captures the four decisions locked
  during /plan-eng-review:
    D1 — provenance guard (≤10 turn walk-back, refuse if cold)
    D2 — synthesis input slice (final-attempt $B calls only,
         closes Codex finding #6)
    D3 — atomic write discipline (temp-dir-then-rename via new
         browse/src/browser-skill-write.ts helper)
    D4 — full test scope (5 gate E2E + 1 unit + smoke)
- New "Phase 2b" sketch for /automate: same skillify machinery,
  per-mutating-step confirmation gate, deferred to next branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* release: v1.16.0.0 -> v1.19.0.0 — browser-skills Phase 1 + 2a

Consolidates the v1.16.0.0 branch-internal bump (Phase 1 runtime, never
landed on main) with Phase 2a (/scrape + /skillify + atomic-write helper)
into one v1.19.0.0 ship per CLAUDE.md "Never orphan branch-internal
versions" rule.

Headline: Browser-skills land end-to-end. /scrape <intent> first call
drives the page; second call runs the codified script in 200ms.

The unified CHANGELOG entry covers:
- Phase 1 runtime: $B skill list/show/run/test/rm, scoped tokens,
  3-tier storage, bundled hackernews-frontpage reference.
- Phase 2a: /scrape + /skillify gstack skills, browser-skill-write.ts
  atomic helper, 5 gate-tier E2E + 34 unit assertions.

Numbers table updated: 5 new modules (+browser-skill-write), 2 new
gstack skills, 6 of 8 Codex outside-voice findings resolved (synthesis
#6 closed by D2; Bun runtime #7 + OS sandbox #1 stay deferred to Phase 4).

/automate (Phase 2b) is split out as P0 in TODOS for the next branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(commands): tighten descriptions for LLM-judge baseline pinning

The skill-llm-eval test "baseline score pinning" failed CI on three
retry attempts: judge gave command_reference.actionability=3, baseline
demands ≥4. Judge cited 8 specific gaps in COMMAND_DESCRIPTIONS.

This commit closes 7 of 8 by tightening the descriptions:

- press: documents that key names are case-sensitive Playwright keys,
  shows modifier syntax (Shift+Enter, Control+A), links the full key
  list. Removes the "is this case-sensitive?" guesswork.
- is: documents that <sel> accepts either a CSS selector OR an @ref
  token from a prior snapshot, and that property values are case-
  sensitive.
- scroll: documents that there is no --by/--to amount option, points
  at `js window.scrollTo(0, N)` for pixel-precise scrolling.
- js / eval: clarifies that both run in the same JS sandbox, the
  difference is just inline expr (js) vs file (eval).
- storage: clarifies sessionStorage is read-only via this command,
  points at `js sessionStorage.setItem(...)` for the write path.
- chain: walks through how to invoke (pipe a JSON array of arrays to
  $B chain), confirms it stops at the first error.
- cdp: explains how to discover allowed methods (read cdp-allowlist.ts)
  + shows a concrete example invocation.
- domain-skill: explains that the "classifier flag" is set automatically
  by the L4 prompt-injection scan (agents do not set it manually);
  enumerates the full lifecycle verbs.

The 8th gap (storage set syntax conflict) is also resolved as part of
the storage rewrite.

Two pipe-character bugs caught by the existing
`no command description contains pipe character` guard at
`test/gen-skill-docs.test.ts:595`: the chain example originally used
`echo '[...]' | $B chain` (literal pipe) and the cdp description used
`tab|browser` / `trusted|untrusted` (also literal pipes). Both rewritten
to keep markdown table cells intact.

Verification: 696/0 pass on skill-validation + gen-skill-docs after
regen across all hosts. The CI llm-judge eval will re-run against the
new SKILL.md and should hit actionability ≥4 reliably.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(browser): rewrite BROWSER.md as complete reference

Full rewrite covering the gstack browser surface as of v1.19.0.0. Up from
488 to 1,299 lines, 26 top-level sections.

Adds previously-undocumented subsystems:

- The productivity loop: /scrape + /skillify with D1 (provenance guard),
  D2 (final-attempt-only synthesis), D3 (atomic-write discipline) contracts.
- Browser-skills runtime: anatomy, three-tier storage, scoped tokens, trust
  model (capability + env axes), sibling SDK distribution, atomic-write
  helper, bundled hackernews-frontpage reference.
- Domain-skills: per-site agent notes with quarantined → active → global
  state machine and the L4-classifier auto-promotion gate.
- Pair-agent: dual-listener architecture, 26-command tunnel allowlist,
  canDispatchOverTunnel pure gate, three token types (root, setup key,
  scoped), denial log path + salt model.
- Security stack L1-L6: layer table, thresholds (BLOCK/WARN/LOG_ONLY/
  SOLO_CONTENT_BLOCK), ensemble rule, classifier model paths, env knobs.
- Side Panel deep dive: Terminal pane (Claude PTY) as the primary surface
  with Activity/Refs/Inspector as debug overlays, WS auth via
  Sec-WebSocket-Protocol, gstackInjectToTerminal cross-pane plumbing.
- CDP escape hatch: $B cdp deny-default allowlist, $B inspect CSS inspector,
  $B ux-audit page structure extraction.
- Meta commands previously undocumented: tabs/frames/state/watch/inbox/
  tab-each, with usage and storage paths.
- Authentication: three token types with lifetimes, SSE session cookie,
  PTY session cookie, token registry behavior.
- Full source map: 30+ file inventory of browse/src/ vs the old 11-file
  list.

Preserves from before: architecture diagram, daemon lifecycle, snapshot
ref staleness, screenshot modes, goto file:// vs load-html semantics,
batch endpoint, JS await wrapping, env vars, performance numbers vs MCP,
Playwright acknowledgments, dev guide.

Cross-links to ARCHITECTURE.md, CLAUDE.md, docs/REMOTE_BROWSER_ACCESS.md,
docs/designs/BROWSER_SKILLS_V1.md, scrape/SKILL.md, skillify/SKILL.md,
TODOS.md so anyone landing on BROWSER.md can navigate to the load-bearing
companion docs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(server): tab-ownership gate keys on tabPolicy, not isWrite

Browser-skill spawns hit `403: Tab not owned by your agent` on every
first run because the gate at server.ts:639 fired for any non-root
write, regardless of the token's tabPolicy. The bundled
hackernews-frontpage reference skill failed identically. Every
/skillify-generated skill failed identically. The user's natural
tabs have no claimed owner — by design — so any skill driving
them via `goto` (a write) was 403'd.

The intent in skill-token.ts:79 was always correct: `tabPolicy: 'shared'`
with the comment "skill scripts may switch tabs as needed." The
enforcement just ignored it.

Two surgical changes:

browser-manager.ts:checkTabAccess — gate now keys on options.ownOnly
only. Shared-policy tokens (skill spawns, default scoped clients) get
permissive access — root-equivalent for the tab gate. Own-only tokens
(pair-agent over the ngrok tunnel) still require ownership for every
read and write. isWrite stays in the signature for callers that want
to log or branch elsewhere; it no longer gates the decision.

server.ts:639 — gate predicate narrowed from
  (WRITE_COMMANDS.has(command) || tokenInfo.tabPolicy === 'own-only')
to just
  tokenInfo.tabPolicy === 'own-only'
The 'newtab' exemption stays. Shared tokens skip the gate entirely;
own-only tokens still hit it. Comment block above the gate updated to
document the new predicate intent.

Pair-agent isolation is intact. Tunnel tokens still default to
tabPolicy: 'own-only', still must `newtab` first to get a tab they
can drive, still can't dispatch any of the 23 commands outside the
tunnel allowlist.

The capability gate (scope checks) and rate limits already constrain
what local scoped clients can do; tab ownership was never a security
boundary for them — only for pair-agent. This release makes the
enforcement match the original design intent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(server): lock the shared-vs-own-only tab gate contract

The pre-fix tests at tab-isolation.test.ts:43,57 encoded the broken
behavior as the contract — they specifically asserted "scoped agent
cannot write to unowned tab," which was the exact failure mode that
broke browser-skills. They passed because they tested the wrong
invariant.

This commit replaces those tests with explicit shared-vs-own-only
coverage that documents what each policy actually means:

- Shared scoped agents (skill spawns, default scoped clients) can
  read AND write any tab — unowned, their own, or another agent's.
  The capability is gated by scope checks + rate limits, not by tab
  ownership.
- Own-only scoped agents (pair-agent over tunnel) cannot read OR
  write any tab they don't own. Pre-fix this case was conflated with
  shared writes; now it's explicit.

9 unit assertions on checkTabAccess, up from 6. Each test names
the policy axis it's covering so a future refactor can't quietly
flip the contract.

Adds source-shape regression test 10a in server-auth.test.ts:
"tab gate predicate is own-only-scoped, not write-scoped." The
gate's `if (...)` line MUST contain `tabPolicy === 'own-only'` and
MUST NOT contain `WRITE_COMMANDS.has(command) ||`. If a future
refactor re-introduces the write-scoped gate, this fails immediately
in free-tier `bun test`.

Updates the marker for the existing newtab-excluded test to match
the new comment block ("Tab ownership check (own-only tokens /
pair-agent isolation)").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* release: v1.19.0.0 -> v1.20.0.0 — fix tab-ownership footgun

Patch release on top of v1.19.0.0. The shipping headline of v1.19.0.0
(/scrape + /skillify productivity loop) was broken on first run in any
session where the daemon already had a tab. Bundled
hackernews-frontpage failed identically. Every /skillify-generated
skill failed identically.

The fix narrows the tab-ownership gate from "any non-root write" to
"tabPolicy === 'own-only' only." Pair-agent isolation (the v1.6.0.0
threat model) is intact; local skill spawns get their original
behavior back.

VERSION: 1.19.0.0 -> 1.20.0.0
package.json version: synced.

CHANGELOG entry leads with the user-visible impact: the productivity
loop works again, no half-second-stalls of confused 403s. Includes
before/after metrics on the bundled reference skill and the broken-
contract pre-fix tests that hid the regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(claude): sharpen CHANGELOG rule — diff between main and ship

Codifies what was already implicit in the existing "Never orphan
branch-internal versions" + "Only document what shipped between main
and this change" sections, but with sharper language and concrete
NEVER examples.

The rule: a CHANGELOG entry is the diff between main and the shipping
branch — what users get when they upgrade. NOT how the branch got
there. Branch-internal version bumps, mid-branch bug fixes, plan
review outcomes, and patch narratives all belong in PR descriptions
and commit messages, not in CHANGELOG.

Adds explicit examples of phrasing to NEVER use:
  - "v1.X had a bug that v1.Y fixes" (mentions a branch-internal version)
  - "The shipping headline of v1.X was broken because..." (apologizes
    for never-released state)
  - "Pre-fix tests encoded the broken behavior" (contributor's victory
    lap, not user benefit)
  - "Two surgical edits, both in the dispatch path" (micro-narrative
    of the patch)

The constructive replacement: describe the released system as a
property, not as a fix. "Browser-skills run end-to-end with the
expected tab-access semantics." If a property is worth calling out,
document it in the trust-model section, not as a "we fixed X" callout.

Pairs with feedback_no_shame_changelog and
feedback_changelog_harden_against_critics memories — entries should
read as a flex even to a hostile screenshotter, never admit prior
breakage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(changelog): consolidate v1.20.0.0 as the diff vs main

Rewrites the v1.20.0.0 entry to describe what users get when they
upgrade from main (v1.17.0.0) to this release: browser-skills
end-to-end. Drops all branch-internal narrative — Phase 1 / Phase 2a
labels, the v1.8.0.0 P1 history paragraph, the test-counts-by-phase
split, and the patch micro-narrative for the tab-policy semantics.

The previously-separate v1.19.0.0 entry (a branch-internal version
that never landed on main) collapses into v1.20.0.0 per the
"Never orphan branch-internal versions" rule.

Tab-access policies are now documented as a property of the trust
model: `'shared'` (skill spawns) is permissive, `'own-only'`
(pair-agent over the tunnel) is strict. No "fix" framing, no
mention of an intermediate state where it was broken.

Adds the BROWSER.md rewrite and the new tab-isolation +
server-auth source-shape regression tests to the itemized changes.

The reverse-chronological order remains: v1.20.0.0 → v1.17.0.0 →
v1.16.0.0 → v1.15.0.0 → ... Gaps (v1.18, v1.19) are fine — those
were branch-internal version numbers that never landed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-28 20:08:04 -07:00

1765 lines
70 KiB
TypeScript

import { describe, test, expect } from 'bun:test';
import { validateSkill, extractRemoteSlugPatterns, extractWeightsFromTable } from './helpers/skill-parser';
import { ALL_COMMANDS, COMMAND_DESCRIPTIONS, READ_COMMANDS, WRITE_COMMANDS, META_COMMANDS } from '../browse/src/commands';
import { SNAPSHOT_FLAGS } from '../browse/src/snapshot';
import * as fs from 'fs';
import * as path from 'path';
const ROOT = path.resolve(import.meta.dir, '..');
describe('SKILL.md command validation', () => {
test('all $B commands in SKILL.md are valid browse commands', () => {
const result = validateSkill(path.join(ROOT, 'SKILL.md'));
expect(result.invalid).toHaveLength(0);
expect(result.valid.length).toBeGreaterThan(0);
});
test('all snapshot flags in SKILL.md are valid', () => {
const result = validateSkill(path.join(ROOT, 'SKILL.md'));
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in browse/SKILL.md are valid browse commands', () => {
const result = validateSkill(path.join(ROOT, 'browse', 'SKILL.md'));
expect(result.invalid).toHaveLength(0);
expect(result.valid.length).toBeGreaterThan(0);
});
test('all snapshot flags in browse/SKILL.md are valid', () => {
const result = validateSkill(path.join(ROOT, 'browse', 'SKILL.md'));
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in qa/SKILL.md are valid browse commands', () => {
const qaSkill = path.join(ROOT, 'qa', 'SKILL.md');
if (!fs.existsSync(qaSkill)) return; // skip if missing
const result = validateSkill(qaSkill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in qa/SKILL.md are valid', () => {
const qaSkill = path.join(ROOT, 'qa', 'SKILL.md');
if (!fs.existsSync(qaSkill)) return;
const result = validateSkill(qaSkill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in qa-only/SKILL.md are valid browse commands', () => {
const qaOnlySkill = path.join(ROOT, 'qa-only', 'SKILL.md');
if (!fs.existsSync(qaOnlySkill)) return;
const result = validateSkill(qaOnlySkill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in qa-only/SKILL.md are valid', () => {
const qaOnlySkill = path.join(ROOT, 'qa-only', 'SKILL.md');
if (!fs.existsSync(qaOnlySkill)) return;
const result = validateSkill(qaOnlySkill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in plan-design-review/SKILL.md are valid browse commands', () => {
const skill = path.join(ROOT, 'plan-design-review', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in plan-design-review/SKILL.md are valid', () => {
const skill = path.join(ROOT, 'plan-design-review', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in design-review/SKILL.md are valid browse commands', () => {
const skill = path.join(ROOT, 'design-review', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in design-review/SKILL.md are valid', () => {
const skill = path.join(ROOT, 'design-review', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in design-consultation/SKILL.md are valid browse commands', () => {
const skill = path.join(ROOT, 'design-consultation', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in design-consultation/SKILL.md are valid', () => {
const skill = path.join(ROOT, 'design-consultation', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
test('all $B commands in autoplan/SKILL.md are valid browse commands', () => {
const skill = path.join(ROOT, 'autoplan', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.invalid).toHaveLength(0);
});
test('all snapshot flags in autoplan/SKILL.md are valid', () => {
const skill = path.join(ROOT, 'autoplan', 'SKILL.md');
if (!fs.existsSync(skill)) return;
const result = validateSkill(skill);
expect(result.snapshotFlagErrors).toHaveLength(0);
});
});
describe('Command registry consistency', () => {
test('COMMAND_DESCRIPTIONS covers all commands in sets', () => {
const allCmds = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
const descKeys = new Set(Object.keys(COMMAND_DESCRIPTIONS));
for (const cmd of allCmds) {
expect(descKeys.has(cmd)).toBe(true);
}
});
test('COMMAND_DESCRIPTIONS has no extra commands not in sets', () => {
const allCmds = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
for (const key of Object.keys(COMMAND_DESCRIPTIONS)) {
expect(allCmds.has(key)).toBe(true);
}
});
test('ALL_COMMANDS matches union of all sets', () => {
const union = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
expect(ALL_COMMANDS.size).toBe(union.size);
for (const cmd of union) {
expect(ALL_COMMANDS.has(cmd)).toBe(true);
}
});
test('SNAPSHOT_FLAGS option keys are valid SnapshotOptions fields', () => {
const validKeys = new Set([
'interactive', 'compact', 'depth', 'selector',
'diff', 'annotate', 'outputPath', 'cursorInteractive',
'heatmap',
]);
for (const flag of SNAPSHOT_FLAGS) {
expect(validKeys.has(flag.optionKey)).toBe(true);
}
});
});
describe('Usage string consistency', () => {
// Normalize a usage string to its structural skeleton for comparison.
// Replaces <param-names> with <>, [optional] with [], strips parenthetical hints.
// This catches format mismatches (e.g., <name>:<value> vs <name> <value>)
// without tripping on abbreviation differences (e.g., <sel> vs <selector>).
function skeleton(usage: string): string {
return usage
.replace(/\(.*?\)/g, '') // strip parenthetical hints like (e.g., Enter, Tab)
.replace(/<[^>]*>/g, '<>') // normalize <param-name> → <>
.replace(/\[[^\]]*\]/g, '[]') // normalize [optional] → []
.replace(/\s+/g, ' ') // collapse whitespace
.trim();
}
// Cross-check Usage: patterns in implementation against COMMAND_DESCRIPTIONS
test('implementation Usage: structural format matches COMMAND_DESCRIPTIONS', () => {
const implFiles = [
path.join(ROOT, 'browse', 'src', 'write-commands.ts'),
path.join(ROOT, 'browse', 'src', 'read-commands.ts'),
path.join(ROOT, 'browse', 'src', 'meta-commands.ts'),
];
// Extract "Usage: browse <pattern>" from throw new Error(...) calls
const usagePattern = /throw new Error\(['"`]Usage:\s*browse\s+(.+?)['"`]\)/g;
const implUsages = new Map<string, string>();
for (const file of implFiles) {
const content = fs.readFileSync(file, 'utf-8');
let match;
while ((match = usagePattern.exec(content)) !== null) {
const usage = match[1].split('\\n')[0].trim();
const cmd = usage.split(/\s/)[0];
implUsages.set(cmd, usage);
}
}
// Compare structural skeletons
const mismatches: string[] = [];
for (const [cmd, implUsage] of implUsages) {
const desc = COMMAND_DESCRIPTIONS[cmd];
if (!desc) continue;
if (!desc.usage) continue;
const descSkel = skeleton(desc.usage);
const implSkel = skeleton(implUsage);
if (descSkel !== implSkel) {
mismatches.push(`${cmd}: docs "${desc.usage}" (${descSkel}) vs impl "${implUsage}" (${implSkel})`);
}
}
expect(mismatches).toEqual([]);
});
});
describe('Generated SKILL.md freshness', () => {
test('no unresolved {{placeholders}} in generated SKILL.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
const unresolved = content.match(/\{\{\w+\}\}/g);
expect(unresolved).toBeNull();
});
test('no unresolved {{placeholders}} in generated browse/SKILL.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
const unresolved = content.match(/\{\{\w+\}\}/g);
expect(unresolved).toBeNull();
});
test('generated SKILL.md has AUTO-GENERATED header', () => {
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
expect(content).toContain('AUTO-GENERATED');
});
});
// --- Update check preamble validation ---
describe('Update check preamble', () => {
const skillsWithUpdateCheck = [
'SKILL.md', 'browse/SKILL.md', 'qa/SKILL.md',
'qa-only/SKILL.md',
'setup-browser-cookies/SKILL.md',
'ship/SKILL.md', 'review/SKILL.md',
'plan-ceo-review/SKILL.md', 'plan-eng-review/SKILL.md',
'retro/SKILL.md',
'office-hours/SKILL.md', 'investigate/SKILL.md',
'plan-design-review/SKILL.md',
'design-review/SKILL.md',
'design-consultation/SKILL.md',
'document-release/SKILL.md',
'canary/SKILL.md',
'benchmark/SKILL.md',
'land-and-deploy/SKILL.md',
'setup-deploy/SKILL.md',
'cso/SKILL.md',
];
for (const skill of skillsWithUpdateCheck) {
test(`${skill} update check line ends with || true`, () => {
const content = fs.readFileSync(path.join(ROOT, skill), 'utf-8');
// The second line of the bash block must end with || true
// to avoid exit code 1 when _UPD is empty (up to date)
const match = content.match(/\[ -n "\$_UPD" \].*$/m);
expect(match).not.toBeNull();
expect(match![0]).toContain('|| true');
});
}
test('all skills with update check are generated from .tmpl', () => {
for (const skill of skillsWithUpdateCheck) {
const tmplPath = path.join(ROOT, skill + '.tmpl');
expect(fs.existsSync(tmplPath)).toBe(true);
}
});
test('update check bash block exits 0 when up to date', () => {
// Simulate the exact preamble command from SKILL.md
const result = Bun.spawnSync(['bash', '-c',
'_UPD=$(echo "" || true); [ -n "$_UPD" ] && echo "$_UPD" || true'
], { stdout: 'pipe', stderr: 'pipe' });
expect(result.exitCode).toBe(0);
});
test('update check bash block exits 0 when upgrade available', () => {
const result = Bun.spawnSync(['bash', '-c',
'_UPD=$(echo "UPGRADE_AVAILABLE 0.3.3 0.4.0" || true); [ -n "$_UPD" ] && echo "$_UPD" || true'
], { stdout: 'pipe', stderr: 'pipe' });
expect(result.exitCode).toBe(0);
expect(result.stdout.toString().trim()).toBe('UPGRADE_AVAILABLE 0.3.3 0.4.0');
});
});
// --- Part 7: Cross-skill path consistency (A1) ---
describe('Cross-skill path consistency', () => {
test('REMOTE_SLUG derivation pattern is identical across files that use it', () => {
const patterns = extractRemoteSlugPatterns(ROOT, ['qa', 'review']);
const allPatterns: string[] = [];
for (const [, filePatterns] of patterns) {
allPatterns.push(...filePatterns);
}
// Should find at least 2 occurrences (qa/SKILL.md + review/greptile-triage.md)
expect(allPatterns.length).toBeGreaterThanOrEqual(2);
// All occurrences must be character-for-character identical
const unique = new Set(allPatterns);
if (unique.size > 1) {
const variants = Array.from(unique);
throw new Error(
`REMOTE_SLUG pattern differs across files:\n` +
variants.map((v, i) => ` ${i + 1}: ${v}`).join('\n')
);
}
});
test('all greptile-history write references specify both per-project and global paths', () => {
const filesToCheck = [
'review/SKILL.md',
'ship/SKILL.md',
'review/greptile-triage.md',
];
for (const file of filesToCheck) {
const filePath = path.join(ROOT, file);
if (!fs.existsSync(filePath)) continue;
const content = fs.readFileSync(filePath, 'utf-8');
const hasBoth = (content.includes('per-project') && content.includes('global')) ||
(content.includes('$REMOTE_SLUG/greptile-history') && content.includes('~/.gstack/greptile-history'));
expect(hasBoth).toBe(true);
}
});
test('greptile-triage.md contains both project and global history paths', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'greptile-triage.md'), 'utf-8');
expect(content).toContain('$REMOTE_SLUG/greptile-history.md');
expect(content).toContain('~/.gstack/greptile-history.md');
});
test('retro/SKILL.md reads global greptile-history (not per-project)', () => {
const content = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
expect(content).toContain('~/.gstack/greptile-history.md');
// Should NOT reference per-project path for reads
expect(content).not.toContain('$REMOTE_SLUG/greptile-history.md');
});
});
// --- Part 7: QA skill structure validation (A2) ---
describe('QA skill structure validation', () => {
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
test('qa/SKILL.md has all 11 phases', () => {
const phases = [
'Phase 1', 'Initialize',
'Phase 2', 'Authenticate',
'Phase 3', 'Orient',
'Phase 4', 'Explore',
'Phase 5', 'Document',
'Phase 6', 'Wrap Up',
'Phase 7', 'Triage',
'Phase 8', 'Fix Loop',
'Phase 9', 'Final QA',
'Phase 10', 'Report',
'Phase 11', 'TODOS',
];
for (const phase of phases) {
expect(qaContent).toContain(phase);
}
});
test('has all four QA modes defined', () => {
const modes = [
'Diff-aware',
'Full',
'Quick',
'Regression',
];
for (const mode of modes) {
expect(qaContent).toContain(mode);
}
// Mode triggers/flags
expect(qaContent).toContain('--quick');
expect(qaContent).toContain('--regression');
});
test('has all three tiers defined', () => {
const tiers = ['Quick', 'Standard', 'Exhaustive'];
for (const tier of tiers) {
expect(qaContent).toContain(tier);
}
});
test('health score weights sum to 100%', () => {
const weights = extractWeightsFromTable(qaContent);
expect(weights.size).toBeGreaterThan(0);
let sum = 0;
for (const pct of weights.values()) {
sum += pct;
}
expect(sum).toBe(100);
});
test('health score has all 8 categories', () => {
const weights = extractWeightsFromTable(qaContent);
const expectedCategories = [
'Console', 'Links', 'Visual', 'Functional',
'UX', 'Performance', 'Content', 'Accessibility',
];
for (const cat of expectedCategories) {
expect(weights.has(cat)).toBe(true);
}
expect(weights.size).toBe(8);
});
test('has four mode definitions (Diff-aware/Full/Quick/Regression)', () => {
expect(qaContent).toContain('### Diff-aware');
expect(qaContent).toContain('### Full');
expect(qaContent).toContain('### Quick');
expect(qaContent).toContain('### Regression');
});
test('output structure references report directory layout', () => {
expect(qaContent).toContain('qa-report-');
expect(qaContent).toContain('baseline.json');
expect(qaContent).toContain('screenshots/');
expect(qaContent).toContain('.gstack/qa-reports/');
});
});
// --- Part 7: Greptile history format consistency (A3) ---
describe('Greptile history format consistency', () => {
test('greptile-triage.md defines the canonical history format', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'greptile-triage.md'), 'utf-8');
expect(content).toContain('<YYYY-MM-DD>');
expect(content).toContain('<owner/repo>');
expect(content).toContain('<type');
expect(content).toContain('<file-pattern>');
expect(content).toContain('<category>');
});
test('review/SKILL.md and ship/SKILL.md both reference greptile-triage.md for write details', () => {
const reviewContent = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
const shipContent = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(reviewContent.toLowerCase()).toContain('greptile-triage.md');
expect(shipContent.toLowerCase()).toContain('greptile-triage.md');
});
test('greptile-triage.md defines all 9 valid categories', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'greptile-triage.md'), 'utf-8');
const categories = [
'race-condition', 'null-check', 'error-handling', 'style',
'type-safety', 'security', 'performance', 'correctness', 'other',
];
for (const cat of categories) {
expect(content).toContain(cat);
}
});
});
// --- Hardcoded branch name detection in templates ---
describe('No hardcoded branch names in SKILL templates', () => {
const tmplFiles = [
'ship/SKILL.md.tmpl',
'review/SKILL.md.tmpl',
'qa/SKILL.md.tmpl',
'plan-ceo-review/SKILL.md.tmpl',
'retro/SKILL.md.tmpl',
'document-release/SKILL.md.tmpl',
'plan-eng-review/SKILL.md.tmpl',
'plan-design-review/SKILL.md.tmpl',
'codex/SKILL.md.tmpl',
];
// Patterns that indicate hardcoded 'main' in git commands
const gitMainPatterns = [
/\bgit\s+diff\s+(?:origin\/)?main\b/,
/\bgit\s+log\s+(?:origin\/)?main\b/,
/\bgit\s+fetch\s+origin\s+main\b/,
/\bgit\s+merge\s+origin\/main\b/,
/\borigin\/main\b/,
];
// Lines that are allowed to mention 'main' (fallback logic, prose)
const allowlist = [
/fall\s*back\s+to\s+`main`/i,
/fall\s*back\s+to\s+`?main`?/i,
/typically\s+`?main`?/i,
/If\s+on\s+`main`/i, // old pattern — should not exist
];
for (const tmplFile of tmplFiles) {
test(`${tmplFile} has no hardcoded 'main' in git commands`, () => {
const filePath = path.join(ROOT, tmplFile);
if (!fs.existsSync(filePath)) return;
const lines = fs.readFileSync(filePath, 'utf-8').split('\n');
const violations: string[] = [];
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
const isAllowlisted = allowlist.some(p => p.test(line));
if (isAllowlisted) continue;
for (const pattern of gitMainPatterns) {
if (pattern.test(line)) {
violations.push(`Line ${i + 1}: ${line.trim()}`);
break;
}
}
}
if (violations.length > 0) {
throw new Error(
`${tmplFile} has hardcoded 'main' in git commands:\n` +
violations.map(v => ` ${v}`).join('\n')
);
}
});
}
});
// --- Part 7b: TODOS-format.md reference consistency ---
describe('TODOS-format.md reference consistency', () => {
test('review/TODOS-format.md exists and defines canonical format', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'TODOS-format.md'), 'utf-8');
expect(content).toContain('**What:**');
expect(content).toContain('**Why:**');
expect(content).toContain('**Priority:**');
expect(content).toContain('**Effort:**');
expect(content).toContain('## Completed');
});
test('skills that write TODOs reference TODOS-format.md', () => {
const shipContent = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
const ceoPlanContent = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
const engPlanContent = fs.readFileSync(path.join(ROOT, 'plan-eng-review', 'SKILL.md'), 'utf-8');
expect(shipContent).toContain('TODOS-format.md');
expect(ceoPlanContent).toContain('TODOS-format.md');
expect(engPlanContent).toContain('TODOS-format.md');
});
});
// --- v0.4.1 feature coverage: RECOMMENDATION format, session awareness, enum completeness ---
describe('v0.4.1 preamble features', () => {
// Tier 1 skills have core preamble only (no AskUserQuestion format)
const tier1Skills = ['SKILL.md', 'browse/SKILL.md', 'setup-browser-cookies/SKILL.md', 'benchmark/SKILL.md'];
// Tier 2+ skills have AskUserQuestion format with RECOMMENDATION
const tier2PlusSkills = [
'qa/SKILL.md', 'qa-only/SKILL.md',
'ship/SKILL.md', 'review/SKILL.md',
'plan-ceo-review/SKILL.md', 'plan-eng-review/SKILL.md',
'retro/SKILL.md',
'office-hours/SKILL.md', 'investigate/SKILL.md',
'plan-design-review/SKILL.md',
'design-review/SKILL.md',
'design-consultation/SKILL.md',
'document-release/SKILL.md',
'canary/SKILL.md',
'land-and-deploy/SKILL.md',
'setup-deploy/SKILL.md',
'cso/SKILL.md',
];
const skillsWithPreamble = [...tier1Skills, ...tier2PlusSkills];
for (const skill of tier2PlusSkills) {
test(`${skill} contains AskUserQuestion Pros/Cons format`, () => {
const content = fs.readFileSync(path.join(ROOT, skill), 'utf-8');
// v1.7.0.0 Pros/Cons format tokens. The preamble resolver
// (generate-ask-user-format.ts) injects all of these into every
// tier-2+ skill. Drop any of them and the test catches it on the
// next `bun test` run.
expect(content).toContain('AskUserQuestion');
expect(content).toContain('Pros / cons:');
expect(content).toContain('Recommendation: <choice>');
expect(content).toContain('Net:');
expect(content).toContain('ELI10');
expect(content).toContain('Stakes if we pick wrong:');
// Concrete format markers must be documented in the resolver text
expect(content).toMatch(/✅/);
expect(content).toMatch(/❌/);
});
}
for (const skill of skillsWithPreamble) {
test(`${skill} contains session awareness`, () => {
const content = fs.readFileSync(path.join(ROOT, skill), 'utf-8');
expect(content).toContain('_SESSIONS');
});
}
for (const skill of skillsWithPreamble) {
test(`${skill} contains escalation protocol`, () => {
const content = fs.readFileSync(path.join(ROOT, skill), 'utf-8');
expect(content).toContain('DONE_WITH_CONCERNS');
expect(content).toContain('BLOCKED');
expect(content).toContain('NEEDS_CONTEXT');
});
}
});
// --- Structural tests for new skills ---
describe('office-hours skill structure', () => {
const content = fs.readFileSync(path.join(ROOT, 'office-hours', 'SKILL.md'), 'utf-8');
// Original structural assertions
for (const section of ['Phase 1', 'Phase 2', 'Phase 3', 'Phase 4', 'Phase 5', 'Phase 6',
'Design Doc', 'Supersedes', 'APPROVED', 'Premise Challenge',
'Alternatives', 'Smart-skip']) {
test(`contains ${section}`, () => expect(content).toContain(section));
}
// Dual-mode structure
for (const section of ['Startup mode', 'Builder mode']) {
test(`contains ${section}`, () => expect(content).toContain(section));
}
// Mode detection question
test('contains explicit mode detection question', () => {
expect(content).toContain("what's your goal");
});
// Six forcing questions (startup mode)
for (const question of ['Demand Reality', 'Status Quo', 'Desperate Specificity',
'Narrowest Wedge', 'Observation & Surprise', 'Future-Fit']) {
test(`contains forcing question: ${question}`, () => expect(content).toContain(question));
}
// Builder mode questions
test('contains builder brainstorming questions', () => {
expect(content).toContain('coolest version');
expect(content).toContain('delightful');
});
// Intrapreneurship adaptation
test('contains intrapreneurship adaptation', () => {
expect(content).toContain('Intrapreneurship');
});
// YC founder discovery engine
test('contains YC apply CTA with ref tracking', () => {
expect(content).toContain('ycombinator.com/apply?ref=gstack');
});
test('contains "What I noticed" design doc section', () => {
expect(content).toContain('What I noticed about how you think');
});
test('contains golden age framing', () => {
expect(content).toContain('golden age');
});
test('contains Garry Tan personal plea', () => {
expect(content).toContain('Garry Tan, the creator of GStack');
});
test('contains founder signal synthesis phase', () => {
expect(content).toContain('Founder Signal Synthesis');
});
test('contains three-tier decision rubric', () => {
expect(content).toContain('Top tier');
expect(content).toContain('Middle tier');
expect(content).toContain('Base tier');
});
test('contains anti-slop examples', () => {
expect(content).toContain('GOOD:');
expect(content).toContain('BAD:');
});
test('contains "One more thing" transition beat', () => {
expect(content).toContain('One more thing');
});
// Operating principles per mode
test('contains startup operating principles', () => {
expect(content).toContain('Specificity is the only currency');
});
test('contains builder operating principles', () => {
expect(content).toContain('Delight is the currency');
});
// Spec Review Loop (Phase 5.5)
test('contains spec review loop', () => {
expect(content).toContain('Spec Review Loop');
});
test('contains adversarial review dimensions', () => {
for (const dim of ['Completeness', 'Consistency', 'Clarity', 'Scope', 'Feasibility']) {
expect(content).toContain(dim);
}
});
test('contains subagent dispatch instruction', () => {
expect(content).toMatch(/Agent.*tool|subagent/i);
});
test('contains max 3 iterations', () => {
expect(content).toMatch(/3.*iteration|maximum.*3/i);
});
test('contains quality score', () => {
expect(content).toContain('quality score');
});
test('contains spec review metrics path', () => {
expect(content).toContain('spec-review.jsonl');
});
test('contains convergence guard', () => {
expect(content).toMatch(/convergence/i);
});
// Visual Sketch (Phase 4.5)
test('contains visual sketch section', () => {
expect(content).toContain('Visual Sketch');
});
test('contains wireframe generation', () => {
expect(content).toMatch(/wireframe|sketch/i);
});
test('contains DESIGN.md awareness', () => {
expect(content).toContain('DESIGN.md');
});
test('contains browse rendering', () => {
expect(content).toContain('$B goto');
expect(content).toContain('$B screenshot');
});
test('contains rough aesthetic instruction', () => {
expect(content).toMatch(/rough|hand-drawn/i);
});
});
describe('investigate skill structure', () => {
const content = fs.readFileSync(path.join(ROOT, 'investigate', 'SKILL.md'), 'utf-8');
for (const section of ['Iron Law', 'Root Cause', 'Pattern Analysis', 'Hypothesis',
'DEBUG REPORT', '3-strike', 'BLOCKED']) {
test(`contains ${section}`, () => expect(content).toContain(section));
}
});
// Contributor mode was removed in v0.13.10.0 — replaced by operational self-improvement.
// Tests for contributor mode preamble structure are no longer applicable.
describe('Enum & Value Completeness in review checklist', () => {
const checklist = fs.readFileSync(path.join(ROOT, 'review', 'checklist.md'), 'utf-8');
test('checklist has Enum & Value Completeness section', () => {
expect(checklist).toContain('Enum & Value Completeness');
});
test('Enum & Value Completeness is classified as CRITICAL', () => {
// It should appear under Pass 1 — CRITICAL, not Pass 2
const pass1Start = checklist.indexOf('### Pass 1');
const pass2Start = checklist.indexOf('### Pass 2');
const enumStart = checklist.indexOf('Enum & Value Completeness');
expect(enumStart).toBeGreaterThan(pass1Start);
expect(enumStart).toBeLessThan(pass2Start);
});
test('Enum & Value Completeness mentions tracing through consumers', () => {
expect(checklist).toContain('Trace it through every consumer');
expect(checklist).toContain('case');
expect(checklist).toContain('allowlist');
});
test('Enum & Value Completeness is in the severity classification as CRITICAL', () => {
const gateSection = checklist.slice(checklist.indexOf('## Severity Classification'));
// The ASCII art has CRITICAL on the left and INFORMATIONAL on the right
// Enum & Value Completeness should appear on a line with the CRITICAL tree (├─ or └─)
const enumLine = gateSection.split('\n').find(l => l.includes('Enum & Value Completeness'));
expect(enumLine).toBeDefined();
// It's on the left (CRITICAL) side — starts with ├─ or └─
expect(enumLine!.trimStart().startsWith('├─') || enumLine!.trimStart().startsWith('└─')).toBe(true);
});
test('Fix-First Heuristic exists in checklist and is referenced by review + ship', () => {
expect(checklist).toContain('## Fix-First Heuristic');
expect(checklist).toContain('AUTO-FIX');
expect(checklist).toContain('ASK');
const reviewSkill = fs.readFileSync(path.join(ROOT, 'review/SKILL.md'), 'utf-8');
const shipSkill = fs.readFileSync(path.join(ROOT, 'ship/SKILL.md'), 'utf-8');
expect(reviewSkill).toContain('AUTO-FIX');
expect(reviewSkill).toContain('[AUTO-FIXED]');
expect(shipSkill).toContain('AUTO-FIX');
expect(shipSkill).toContain('[AUTO-FIXED]');
});
});
// --- Completeness Principle spot-check ---
describe('Completeness Principle in generated SKILL.md files', () => {
const skillsWithPreamble = [
'qa/SKILL.md',
'qa-only/SKILL.md',
'ship/SKILL.md', 'review/SKILL.md',
'plan-ceo-review/SKILL.md', 'plan-eng-review/SKILL.md',
'retro/SKILL.md',
'plan-design-review/SKILL.md',
'design-review/SKILL.md',
'design-consultation/SKILL.md',
'document-release/SKILL.md',
'cso/SKILL.md', ];
for (const skill of skillsWithPreamble) {
test(`${skill} contains Completeness Principle section`, () => {
const content = fs.readFileSync(path.join(ROOT, skill), 'utf-8');
expect(content).toContain('Completeness Principle');
expect(content).toContain('Boil the Lake');
});
}
test('Completeness Principle keeps compact scoring guidance in tier 2+ skills', () => {
const content = fs.readFileSync(path.join(ROOT, 'cso', 'SKILL.md'), 'utf-8');
expect(content).toContain('Completeness: X/10');
expect(content).toContain('10 = all edge cases');
expect(content).toContain('Note: options differ in kind, not coverage');
expect(content).toContain('Do not fabricate scores');
});
});
// --- Part 7: Planted-bug fixture validation (A4) ---
describe('Planted-bug fixture validation', () => {
test('qa-eval ground truth has exactly 5 planted bugs', () => {
const groundTruth = JSON.parse(
fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'qa-eval-ground-truth.json'), 'utf-8')
);
expect(groundTruth.bugs).toHaveLength(5);
expect(groundTruth.total_bugs).toBe(5);
});
test('qa-eval-spa ground truth has exactly 5 planted bugs', () => {
const groundTruth = JSON.parse(
fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'qa-eval-spa-ground-truth.json'), 'utf-8')
);
expect(groundTruth.bugs).toHaveLength(5);
expect(groundTruth.total_bugs).toBe(5);
});
test('qa-eval-checkout ground truth has exactly 5 planted bugs', () => {
const groundTruth = JSON.parse(
fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'qa-eval-checkout-ground-truth.json'), 'utf-8')
);
expect(groundTruth.bugs).toHaveLength(5);
expect(groundTruth.total_bugs).toBe(5);
});
test('qa-eval.html contains the planted bugs', () => {
const html = fs.readFileSync(path.join(ROOT, 'browse', 'test', 'fixtures', 'qa-eval.html'), 'utf-8');
// BUG 1: broken link
expect(html).toContain('/nonexistent-404-page');
// BUG 2: disabled submit
expect(html).toContain('disabled');
// BUG 3: overflow
expect(html).toContain('overflow: hidden');
// BUG 4: missing alt
expect(html).toMatch(/<img[^>]*src="\/logo\.png"[^>]*>/);
expect(html).not.toMatch(/<img[^>]*src="\/logo\.png"[^>]*alt=/);
// BUG 5: console error
expect(html).toContain("Cannot read properties of undefined");
});
test('review-eval-vuln.rb contains expected vulnerability patterns', () => {
const content = fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-vuln.rb'), 'utf-8');
expect(content).toContain('params[:id]');
expect(content).toContain('update_column');
});
});
// --- CEO review mode validation ---
describe('CEO review mode validation', () => {
const content = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
test('has all four CEO review modes defined', () => {
const modes = ['SCOPE EXPANSION', 'SELECTIVE EXPANSION', 'HOLD SCOPE', 'SCOPE REDUCTION'];
for (const mode of modes) {
expect(content).toContain(mode);
}
});
test('has CEO plan persistence step', () => {
expect(content).toContain('ceo-plans');
expect(content).toContain('status: ACTIVE');
});
test('has docs/designs promotion section', () => {
expect(content).toContain('docs/designs');
expect(content).toContain('PROMOTED');
});
test('mode quick reference has four columns', () => {
expect(content).toContain('EXPANSION');
expect(content).toContain('SELECTIVE');
expect(content).toContain('HOLD SCOPE');
expect(content).toContain('REDUCTION');
});
// Skill chaining (benefits-from)
test('contains prerequisite skill offer for office-hours', () => {
expect(content).toContain('Prerequisite Skill Offer');
expect(content).toContain('/office-hours');
});
test('contains mid-session detection', () => {
expect(content).toContain('Mid-session detection');
expect(content).toMatch(/still figuring out|seems lost/i);
});
// Spec review on CEO plans
test('contains spec review loop for CEO plan documents', () => {
expect(content).toContain('Spec Review Loop');
});
});
// --- gstack-slug helper ---
describe('gstack-slug', () => {
const SLUG_BIN = path.join(ROOT, 'bin', 'gstack-slug');
test('binary exists and is executable', () => {
expect(fs.existsSync(SLUG_BIN)).toBe(true);
const stat = fs.statSync(SLUG_BIN);
expect(stat.mode & 0o111).toBeGreaterThan(0);
});
test('outputs SLUG and BRANCH lines in a git repo', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' });
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
expect(output).toContain('SLUG=');
expect(output).toContain('BRANCH=');
});
test('SLUG does not contain forward slashes', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' });
const slug = result.stdout.toString().match(/SLUG=(.*)/)?.[1] ?? '';
expect(slug).not.toContain('/');
expect(slug.length).toBeGreaterThan(0);
});
test('BRANCH does not contain forward slashes', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' });
const branch = result.stdout.toString().match(/BRANCH=(.*)/)?.[1] ?? '';
expect(branch).not.toContain('/');
expect(branch.length).toBeGreaterThan(0);
});
test('output is eval-compatible (KEY=VALUE format)', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' });
const lines = result.stdout.toString().trim().split('\n');
expect(lines.length).toBe(2);
expect(lines[0]).toMatch(/^SLUG=.+/);
expect(lines[1]).toMatch(/^BRANCH=.+/);
});
test('output values contain only safe characters (no shell metacharacters)', () => {
const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' });
const slug = result.stdout.toString().match(/SLUG=(.*)/)?.[1] ?? '';
const branch = result.stdout.toString().match(/BRANCH=(.*)/)?.[1] ?? '';
// Only alphanumeric, dot, dash, underscore are allowed (#133)
expect(slug).toMatch(/^[a-zA-Z0-9._-]+$/);
expect(branch).toMatch(/^[a-zA-Z0-9._-]+$/);
});
test('eval sets variables under bash with set -euo pipefail', () => {
const result = Bun.spawnSync(
['bash', '-c', 'set -euo pipefail; eval "$(./bin/gstack-slug 2>/dev/null)"; echo "SLUG=$SLUG"; echo "BRANCH=$BRANCH"'],
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }
);
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
expect(output).toMatch(/^SLUG=.+/m);
expect(output).toMatch(/^BRANCH=.+/m);
});
test('no templates or bin scripts use source process substitution for gstack-slug', () => {
const result = Bun.spawnSync(
['grep', '-r', 'source <(.*gstack-slug', '--include=*.tmpl', '--include=gstack-review-*', '.'],
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }
);
// grep returns exit code 1 when no matches found — that's what we want
expect(result.stdout.toString().trim()).toBe('');
});
});
// --- Test Bootstrap validation ---
describe('Test Bootstrap ({{TEST_BOOTSTRAP}}) integration', () => {
test('TEST_BOOTSTRAP resolver produces valid content', () => {
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(qaContent).toContain('Test Framework Bootstrap');
expect(qaContent).toContain('RUNTIME:ruby');
expect(qaContent).toContain('RUNTIME:node');
expect(qaContent).toContain('RUNTIME:python');
expect(qaContent).toContain('no-test-bootstrap');
expect(qaContent).toContain('BOOTSTRAP_DECLINED');
});
test('TEST_BOOTSTRAP appears in qa/SKILL.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('Test Framework Bootstrap');
expect(content).toContain('TESTING.md');
expect(content).toContain('CLAUDE.md');
});
test('TEST_BOOTSTRAP appears in ship/SKILL.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Test Framework Bootstrap');
expect(content).toContain('Step 4');
});
test('TEST_BOOTSTRAP appears in design-review/SKILL.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
expect(content).toContain('Test Framework Bootstrap');
});
test('TEST_BOOTSTRAP does NOT appear in qa-only/SKILL.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa-only', 'SKILL.md'), 'utf-8');
expect(content).not.toContain('Test Framework Bootstrap');
// But should have the recommendation note
expect(content).toContain('No test framework detected');
expect(content).toContain('Run `/qa` to bootstrap');
});
test('bootstrap includes framework knowledge table', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('vitest');
expect(content).toContain('minitest');
expect(content).toContain('pytest');
expect(content).toContain('cargo test');
expect(content).toContain('phpunit');
expect(content).toContain('ExUnit');
});
test('bootstrap includes CI/CD pipeline generation', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('.github/workflows/test.yml');
expect(content).toContain('GitHub Actions');
});
test('bootstrap includes first real tests step', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('First real tests');
expect(content).toContain('git log --since=30.days');
expect(content).toContain('Prioritize by risk');
});
test('bootstrap includes vibe coding philosophy', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('vibe coding');
expect(content).toContain('100% test coverage');
});
test('WebSearch is in allowed-tools for qa, ship, design-review', () => {
const qa = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
const ship = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
const qaDesign = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
expect(qa).toContain('WebSearch');
expect(ship).toContain('WebSearch');
expect(qaDesign).toContain('WebSearch');
});
});
// --- Phase 8e.5 regression test validation ---
describe('Phase 8e.5 regression test generation', () => {
test('qa/SKILL.md contains Phase 8e.5', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('8e.5. Regression Test');
expect(content).toContain('test(qa): regression test');
expect(content).toContain('WTF-likelihood exclusion');
});
test('qa/SKILL.md Rule 13 is amended for regression tests', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('Only modify tests when generating regression tests in Phase 8e.5');
expect(content).not.toContain('Never modify tests or CI configuration');
});
test('design-review has CSS-aware Phase 8e.5 variant', () => {
const content = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
expect(content).toContain('8e.5. Regression Test (design-review variant)');
expect(content).toContain('CSS-only');
expect(content).toContain('test(design): regression test');
});
test('regression test includes full attribution comment format', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('// Regression: ISSUE-NNN');
expect(content).toContain('// Found by /qa on');
expect(content).toContain('// Report: .gstack/qa-reports/');
});
test('regression test uses auto-incrementing names', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
expect(content).toContain('auto-incrementing');
expect(content).toContain('max number + 1');
});
});
// --- Step 3.4 coverage audit validation ---
describe('Step 3.4 test coverage audit', () => {
test('ship/SKILL.md contains Step 7', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Step 7: Test Coverage Audit');
// The coverage diagram collapses code-path and user-flow counts onto one
// summary line. Verify that summary is present (labels are stable).
expect(content).toContain('Code paths:');
});
test('Step 3.4 includes quality scoring rubric', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('★★★');
expect(content).toContain('★★');
expect(content).toContain('edge cases AND error paths');
expect(content).toContain('happy path only');
});
test('Step 3.4 includes before/after test count', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Count test files before');
expect(content).toContain('Count test files after');
});
test('ship PR body includes Test Coverage section', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('## Test Coverage');
});
test('ship rules include test generation rule', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Step 7 generates coverage tests');
expect(content).toContain('Never commit failing tests');
});
test('Step 3.4 includes vibe coding philosophy', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('vibe coding becomes yolo coding');
});
test('Step 3.4 traces actual codepaths, not just syntax', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Trace every codepath');
expect(content).toContain('Trace data flow');
expect(content).toContain('Diagram the execution');
});
test('Step 3.4 maps user flows and interaction edge cases', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Map user flows');
expect(content).toContain('Interaction edge cases');
expect(content).toContain('Double-click');
expect(content).toContain('Navigate away');
expect(content).toContain('Error states the user can see');
expect(content).toContain('Empty/zero/boundary states');
});
test('Step 3.4 diagram includes user-flow coverage summary', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
// The diagram was compressed from separate CODE PATH COVERAGE / USER FLOW
// COVERAGE section headers into a single summary line. Assert on the
// labels that still appear on that summary line.
expect(content).toContain('Code paths:');
expect(content).toContain('User flows:');
});
});
// --- Ship step numbering regression guard ---
describe('ship step numbering', () => {
// Allowed sub-steps that are resolver-generated and intentionally nested:
// 8.1 (Plan Verification), 8.2 (Scope Drift), 9.1 (Review Army), 9.2 (Findings Merge),
// 9.3 (Cross-review dedup), 15.0 (WIP squash — continuous checkpoint), 15.1 (Bisectable commits).
const ALLOWED_SUBSTEPS = new Set(['8.1', '8.2', '9.1', '9.2', '9.3', '15.0', '15.1']);
test('ship/SKILL.md.tmpl contains no unexpected fractional step numbers', () => {
const tmpl = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md.tmpl'), 'utf-8');
// Match "Step X.Y" where X.Y is a decimal step reference (e.g., "Step 3.47", "Step 8.1")
const matches = Array.from(tmpl.matchAll(/Step (\d+\.\d+)/g));
const violations = matches
.map((m) => m[1])
.filter((n) => !ALLOWED_SUBSTEPS.has(n));
if (violations.length > 0) {
const unique = Array.from(new Set(violations)).sort();
throw new Error(
`ship/SKILL.md.tmpl contains fractional step numbers that are not in the allowed sub-step list.\n` +
` Found: ${unique.join(', ')}\n` +
` Allowed sub-steps: ${Array.from(ALLOWED_SUBSTEPS).sort().join(', ')}\n` +
` Fix: use clean integer step numbers (1-20), or add to ALLOWED_SUBSTEPS if intentional.`
);
}
});
test('ship/SKILL.md main headings use clean integer step numbers', () => {
const skill = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
// Headings like "## Step 7: Test Coverage Audit" — NOT sub-steps like "## Step 8.1:"
const headings = Array.from(skill.matchAll(/^## Step (\d+(?:\.\d+)?):/gm)).map(
(m) => m[1]
);
const fractional = headings.filter((n) => n.includes('.'));
const unexpected = fractional.filter((n) => !ALLOWED_SUBSTEPS.has(n));
expect(unexpected).toEqual([]);
});
test('review/SKILL.md step numbers unchanged (regression guard for resolver conditionals)', () => {
const skill = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
// /review uses its own fractional numbering: 1.5, 2.5, 4.5, 5.5, 5.6, 5.7, 5.8
// If the ship-side renumber accidentally touched the review-side of resolver conditionals,
// these would vanish. This test catches that.
expect(skill).toContain('## Step 1.5: Scope Drift Detection');
expect(skill).toContain('## Step 4.5: Review Army');
expect(skill).toContain('## Step 5.7: Adversarial review');
});
});
// --- Retro test health validation ---
describe('Retro test health tracking', () => {
test('retro/SKILL.md has test health data gathering commands', () => {
const content = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
expect(content).toContain('# 10. Test file count');
expect(content).toContain('# 11. Regression test commits');
expect(content).toContain('# 12. Test files changed');
});
test('retro/SKILL.md has Test Health metrics row', () => {
const content = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
expect(content).toContain('Test Health');
expect(content).toContain('regression tests');
});
test('retro/SKILL.md has Test Health narrative section', () => {
const content = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
expect(content).toContain('### Test Health');
expect(content).toContain('Total test files');
expect(content).toContain('vibe coding safe');
});
test('retro JSON schema includes test_health field', () => {
const content = fs.readFileSync(path.join(ROOT, 'retro', 'SKILL.md'), 'utf-8');
expect(content).toContain('test_health');
expect(content).toContain('total_test_files');
expect(content).toContain('regression_test_commits');
});
});
// --- QA report template regression tests section ---
describe('QA report template', () => {
test('qa-report-template.md has Regression Tests section', () => {
const content = fs.readFileSync(path.join(ROOT, 'qa', 'templates', 'qa-report-template.md'), 'utf-8');
expect(content).toContain('## Regression Tests');
expect(content).toContain('committed / deferred / skipped');
expect(content).toContain('### Deferred Tests');
expect(content).toContain('**Precondition:**');
});
});
// --- Codex skill validation ---
describe('Codex skill', () => {
test('codex/SKILL.md exists and has correct frontmatter', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('name: codex');
expect(content).toContain('version: 1.0.0');
expect(content).toContain('allowed-tools:');
});
test('codex/SKILL.md contains all three modes', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('Step 2A: Review Mode');
expect(content).toContain('Step 2B: Challenge');
expect(content).toContain('Step 2C: Consult Mode');
});
test('codex/SKILL.md contains gate verdict logic', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('[P1]');
expect(content).toContain('GATE: PASS');
expect(content).toContain('GATE: FAIL');
});
test('codex/SKILL.md contains session continuity', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('codex-session-id');
expect(content).toContain('codex exec resume');
});
test('codex/SKILL.md contains cost tracking', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('tokens used');
expect(content).toContain('Est. cost');
});
test('codex/SKILL.md contains cross-model comparison', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('CROSS-MODEL ANALYSIS');
expect(content).toContain('Agreement rate');
});
test('codex/SKILL.md contains review log persistence', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('codex-review');
expect(content).toContain('gstack-review-log');
});
test('codex/SKILL.md uses which for binary discovery, not hardcoded path', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('which codex');
expect(content).not.toContain('/opt/homebrew/bin/codex');
});
test('codex/SKILL.md contains error handling for missing binary and auth', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('NOT_FOUND');
expect(content).toContain('codex login');
});
test('codex/SKILL.md uses mktemp for temp files', () => {
const content = fs.readFileSync(path.join(ROOT, 'codex', 'SKILL.md'), 'utf-8');
expect(content).toContain('mktemp');
});
test('adversarial review in /review always runs both passes', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
expect(content).toContain('Adversarial review (always-on)');
// Always-on: both Claude and Codex adversarial
expect(content).toContain('Claude adversarial subagent (always runs)');
expect(content).toContain('Codex adversarial challenge (always runs when available)');
// Claude adversarial subagent dispatch
expect(content).toContain('Agent tool');
expect(content).toContain('FIXABLE');
expect(content).toContain('INVESTIGATE');
// Codex availability check
expect(content).toContain('CODEX_NOT_AVAILABLE');
// OLD_CFG only gates Codex, not Claude
expect(content).toContain('skip Codex passes only');
// Review log
expect(content).toContain('adversarial-review');
expect(content).toContain('reasoning_effort="high"');
expect(content).toContain('ADVERSARIAL REVIEW SYNTHESIS');
// Large diff structured review still gated
expect(content).toContain('Codex structured review (large diffs only');
expect(content).toContain('200');
});
test('adversarial review in /ship always runs both passes', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Adversarial review (always-on)');
expect(content).toContain('adversarial-review');
expect(content).toContain('reasoning_effort="high"');
expect(content).toContain('Investigate and fix');
expect(content).toContain('Claude adversarial subagent (always runs)');
});
test('scope drift detection in /review and /ship', () => {
const reviewContent = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
const shipContent = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
// Both should contain scope drift from the shared resolver
for (const content of [reviewContent, shipContent]) {
expect(content).toContain('Scope Check:');
expect(content).toContain('DRIFT DETECTED');
expect(content).toContain('SCOPE CREEP');
expect(content).toContain('MISSING REQUIREMENTS');
expect(content).toContain('stated intent');
}
});
test('codex-host ship/review do NOT contain adversarial review step', () => {
// .agents/ is gitignored — generate on demand
Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex'], {
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
});
const shipContent = fs.readFileSync(path.join(ROOT, '.agents', 'skills', 'gstack-ship', 'SKILL.md'), 'utf-8');
expect(shipContent).not.toContain('codex review --base');
expect(shipContent).not.toContain('CODEX_REVIEWS');
const reviewContent = fs.readFileSync(path.join(ROOT, '.agents', 'skills', 'gstack-review', 'SKILL.md'), 'utf-8');
expect(reviewContent).not.toContain('codex review --base');
expect(reviewContent).not.toContain('codex_reviews');
expect(reviewContent).not.toContain('CODEX_REVIEWS');
expect(reviewContent).not.toContain('adversarial-review');
expect(reviewContent).not.toContain('Investigate and fix');
});
test('codex integration in /plan-eng-review offers plan critique', () => {
const content = fs.readFileSync(path.join(ROOT, 'plan-eng-review', 'SKILL.md'), 'utf-8');
expect(content).toContain('Codex');
expect(content).toContain('codex exec');
});
test('/review persists a review-log entry for ship readiness', () => {
const content = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
expect(content).toContain('"skill":"review"');
expect(content).toContain('"issues_found":N');
expect(content).toContain('Persist Eng Review result');
});
test('Review Readiness Dashboard includes Adversarial Review row', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Adversarial');
expect(content).toContain('codex-review');
});
});
// --- Trigger phrase validation ---
describe('Skill trigger phrases', () => {
// Skills that must have "Use when" trigger phrases in their description.
// Excluded: root gstack (browser tool), gstack-upgrade (gstack-specific),
// humanizer (text tool)
const SKILLS_REQUIRING_TRIGGERS = [
'qa', 'qa-only', 'ship', 'review', 'investigate', 'office-hours',
'plan-ceo-review', 'plan-eng-review', 'plan-design-review',
'design-review', 'design-consultation', 'retro', 'document-release',
'codex', 'browse', 'setup-browser-cookies',
];
for (const skill of SKILLS_REQUIRING_TRIGGERS) {
test(`${skill}/SKILL.md has "Use when" trigger phrases`, () => {
const skillPath = path.join(ROOT, skill, 'SKILL.md');
if (!fs.existsSync(skillPath)) return;
const content = fs.readFileSync(skillPath, 'utf-8');
// Extract description from frontmatter
const frontmatterEnd = content.indexOf('---', 4);
const frontmatter = content.slice(0, frontmatterEnd);
expect(frontmatter).toMatch(/Use when/i);
});
}
// Skills with proactive triggers should have "Proactively suggest" in description
const SKILLS_REQUIRING_PROACTIVE = [
'qa', 'qa-only', 'ship', 'review', 'investigate', 'office-hours',
'plan-ceo-review', 'plan-eng-review', 'plan-design-review',
'design-review', 'design-consultation', 'retro', 'document-release',
];
for (const skill of SKILLS_REQUIRING_PROACTIVE) {
test(`${skill}/SKILL.md has proactive routing phrase`, () => {
const skillPath = path.join(ROOT, skill, 'SKILL.md');
if (!fs.existsSync(skillPath)) return;
const content = fs.readFileSync(skillPath, 'utf-8');
const frontmatterEnd = content.indexOf('---', 4);
const frontmatter = content.slice(0, frontmatterEnd);
expect(frontmatter).toMatch(/Proactively (suggest|invoke)/i);
});
}
});
// ─── Codex Skill Validation ──────────────────────────────────
describe('Codex skill validation', () => {
const AGENTS_DIR = path.join(ROOT, '.agents', 'skills');
// .agents/ is gitignored (v0.11.2.0) — generate on demand for tests
Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex'], {
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
});
// Discover all shared skills with templates.
// Host-exclusive outside-voice skills are intentionally omitted here:
// - /codex is Claude-only
// - /claude is external-host-only
const CLAUDE_SKILLS_WITH_TEMPLATES = (() => {
const skills: string[] = [];
for (const entry of fs.readdirSync(ROOT, { withFileTypes: true })) {
if (!entry.isDirectory() || entry.name.startsWith('.') || entry.name === 'node_modules') continue;
if (entry.name === 'codex') continue; // Claude-only skill
if (entry.name === 'claude') continue; // External-host-only skill
if (fs.existsSync(path.join(ROOT, entry.name, 'SKILL.md.tmpl'))) {
skills.push(entry.name);
}
}
return skills;
})();
test('all skills (except /codex) have both Claude and Codex variants', () => {
for (const skillDir of CLAUDE_SKILLS_WITH_TEMPLATES) {
// Claude variant
const claudeMd = path.join(ROOT, skillDir, 'SKILL.md');
expect(fs.existsSync(claudeMd)).toBe(true);
// Codex variant
const codexName = skillDir.startsWith('gstack-') ? skillDir : `gstack-${skillDir}`;
const codexMd = path.join(AGENTS_DIR, codexName, 'SKILL.md');
expect(fs.existsSync(codexMd)).toBe(true);
}
// Root template has both too
expect(fs.existsSync(path.join(ROOT, 'SKILL.md'))).toBe(true);
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack', 'SKILL.md'))).toBe(true);
});
test('/codex skill is Claude-only — no Codex variant', () => {
// Claude variant should exist
expect(fs.existsSync(path.join(ROOT, 'codex', 'SKILL.md'))).toBe(true);
// Codex variant must NOT exist
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-codex', 'SKILL.md'))).toBe(false);
});
test('/claude skill is external-host-only — no Claude-host variant', () => {
// Claude host should not get an outside-voice skill that shells into Claude.
expect(fs.existsSync(path.join(ROOT, 'claude', 'SKILL.md'))).toBe(false);
// Codex/external hosts should get the generated wrapper.
expect(fs.existsSync(path.join(AGENTS_DIR, 'gstack-claude', 'SKILL.md'))).toBe(true);
});
test('Codex skill names follow gstack-{name} convention', () => {
const codexDirs = fs.readdirSync(AGENTS_DIR);
for (const dir of codexDirs) {
// Every directory should start with gstack
expect(dir.startsWith('gstack')).toBe(true);
// Root is just 'gstack', others are 'gstack-{name}'
if (dir !== 'gstack') {
expect(dir.startsWith('gstack-')).toBe(true);
}
}
});
test('$B commands in Codex SKILL.md files are valid browse commands', () => {
const codexDirs = fs.readdirSync(AGENTS_DIR);
for (const dir of codexDirs) {
const skillMd = path.join(AGENTS_DIR, dir, 'SKILL.md');
if (!fs.existsSync(skillMd)) continue;
const content = fs.readFileSync(skillMd, 'utf-8');
// Only validate if the skill contains $B commands
if (!content.includes('$B ')) continue;
const result = validateSkill(skillMd);
expect(result.invalid).toHaveLength(0);
}
});
});
// --- Repo mode and test failure triage validation ---
describe('Repo mode preamble validation', () => {
test('generated SKILL.md preamble contains REPO_MODE output', () => {
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
expect(content).toContain('REPO_MODE:');
expect(content).toContain('gstack-repo-mode');
});
test('tier 3+ skills contain See Something Say Something section', () => {
// Root SKILL.md is tier 1 (no Repo Mode). Check a tier 3 skill instead.
const content = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
expect(content).toContain('See Something, Say Something');
expect(content).toContain('REPO_MODE');
expect(content).toContain('solo');
expect(content).toContain('collaborative');
});
});
describe('Test failure triage in ship skill', () => {
test('ship/SKILL.md contains Test Failure Ownership Triage', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Test Failure Ownership Triage');
});
test('ship/SKILL.md triage uses git diff for classification', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('git diff origin/<base>...HEAD --name-only');
});
test('ship/SKILL.md triage has solo and collaborative paths', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('REPO_MODE');
expect(content).toContain('solo');
expect(content).toContain('collaborative');
expect(content).toContain('Investigate and fix now');
expect(content).toContain('Add as P0 TODO');
});
test('ship/SKILL.md triage has GitHub issue assignment for collaborative mode', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('gh issue create');
expect(content).toContain('--assignee');
});
test('{{TEST_FAILURE_TRIAGE}} placeholder is fully resolved in ship/SKILL.md', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).not.toContain('{{TEST_FAILURE_TRIAGE}}');
});
test('ship/SKILL.md uses in-branch language for stop condition', () => {
const content = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('In-branch test failures');
});
});
describe('no compiled binaries in git', () => {
// Tracked files enumerated once and reused by both assertions. git ls-files -z
// + split is ~ms; the previous xargs-per-file shell loops blew past 5s on CI.
const trackedFiles: string[] = require('child_process')
.execSync('git ls-files -z', { cwd: ROOT, encoding: 'utf-8' })
.split('\0')
.filter(Boolean);
test('git tracks no Mach-O or ELF binaries', () => {
// Only mode 100755 (executable) files can be binaries we care about. Pre-filter
// via git ls-files -s to avoid running `file` on every text file.
const lsOut: string = require('child_process').execSync('git ls-files -s', {
cwd: ROOT,
encoding: 'utf-8',
});
const executableFiles = lsOut
.split('\n')
.filter(Boolean)
.map((line: string) => {
const parts = line.split(/\s+/);
return { mode: parts[0], file: line.split('\t')[1] };
})
.filter((e: { mode: string; file: string }) => e.mode === '100755')
.map((e: { mode: string; file: string }) => e.file);
if (executableFiles.length === 0) return;
// Batch-invoke `file --mime-type` across all executable files at once.
const result: string = require('child_process')
.execSync(`file --mime-type -- ${executableFiles.map((f: string) => `'${f.replace(/'/g, "'\\''")}'`).join(' ')}`, {
cwd: ROOT,
encoding: 'utf-8',
})
.trim();
const binaries = result
.split('\n')
.filter((l: string) =>
/application\/(x-mach-binary|x-executable|x-pie-executable|x-sharedlib)/.test(l)
)
.map((l: string) => l.split(':')[0].trim());
expect(binaries).toEqual([]);
});
test('warns about tracked files larger than 2MB', () => {
// Large fixtures can be legitimate test infrastructure. Keep visibility on
// repository size without blocking those fixtures from living in git.
// Known-good fixtures are exempted from the warning to keep CI logs clean.
const MAX_BYTES = 2 * 1024 * 1024;
const knownLargeFixtures = new Set([
// Deterministic replay fixture for BrowseSafe-Bench. The live bench is
// expensive; this file is intentionally committed so the gate is free.
'browse/test/fixtures/security-bench-haiku-responses.json',
]);
const oversized = trackedFiles.flatMap((f: string) => {
if (knownLargeFixtures.has(f)) return [];
const full = path.join(ROOT, f);
try {
const size = fs.statSync(full).size;
return size > MAX_BYTES ? [{ file: f, size }] : [];
} catch {
return [];
}
});
if (oversized.length > 0) {
const formatted = oversized
.map(({ file, size }: { file: string; size: number }) => {
const mib = (size / (1024 * 1024)).toFixed(1);
return `${file} (${mib} MiB)`;
})
.join(', ');
console.warn(`[size-warning] tracked files over 2 MiB: ${formatted}`);
}
expect(Array.isArray(oversized)).toBe(true);
});
});
// `sidebar agent (#584)` describe block was here. sidebar-agent.ts and
// the entire chat-queue path were ripped in favor of the interactive
// claude PTY (terminal-agent.ts); these assertions had no target file.
// Terminal-pane invariants are covered by browse/test/sidebar-tabs.test.ts
// and browse/test/terminal-agent.test.ts.
// ─── Browser-skills validation ──────────────────────────────────
//
// Browser-skills are bundled in <gstack-root>/browser-skills/<name>/. Each
// must have a SKILL.md whose frontmatter satisfies the contract enforced by
// browse/src/browser-skills.ts:parseSkillFile (host required, args + triggers
// parseable as the right shape). This test catches malformed bundled skills
// at CI time, before they ship.
describe('Bundled browser-skills frontmatter contract', () => {
const browserSkillsRoot = path.join(ROOT, 'browser-skills');
function listBundledSkillDirs(): string[] {
if (!fs.existsSync(browserSkillsRoot)) return [];
return fs.readdirSync(browserSkillsRoot)
.filter(name => !name.startsWith('.'))
.map(name => path.join(browserSkillsRoot, name))
.filter(dir => {
try { return fs.statSync(dir).isDirectory(); } catch { return false; }
});
}
test('each bundled skill has a SKILL.md', () => {
for (const dir of listBundledSkillDirs()) {
const skillFile = path.join(dir, 'SKILL.md');
expect(fs.existsSync(skillFile)).toBe(true);
}
});
test('each bundled skill SKILL.md frontmatter parses with required fields', async () => {
const { parseSkillFile } = await import('../browse/src/browser-skills');
for (const dir of listBundledSkillDirs()) {
const name = path.basename(dir);
const content = fs.readFileSync(path.join(dir, 'SKILL.md'), 'utf-8');
// parseSkillFile throws on missing required fields; we just want to
// make sure none of our shipped skills tripwire it.
const { frontmatter } = parseSkillFile(content, { skillName: name });
expect(frontmatter.name).toBe(name);
expect(typeof frontmatter.host).toBe('string');
expect(frontmatter.host.length).toBeGreaterThan(0);
expect(Array.isArray(frontmatter.triggers)).toBe(true);
expect(Array.isArray(frontmatter.args)).toBe(true);
}
});
test('each bundled skill has a script.ts', () => {
for (const dir of listBundledSkillDirs()) {
expect(fs.existsSync(path.join(dir, 'script.ts'))).toBe(true);
}
});
test('each bundled skill ships a sibling SDK at _lib/browse-client.ts', () => {
for (const dir of listBundledSkillDirs()) {
expect(fs.existsSync(path.join(dir, '_lib', 'browse-client.ts'))).toBe(true);
}
});
test('each bundled skill has a script.test.ts', () => {
for (const dir of listBundledSkillDirs()) {
expect(fs.existsSync(path.join(dir, 'script.test.ts'))).toBe(true);
}
});
test("each bundled skill's _lib/browse-client.ts matches the canonical SDK", () => {
// If the canonical SDK changes, the bundled copy must be updated. This
// test enforces that — the _lib copy should be byte-identical.
const canonical = fs.readFileSync(path.join(ROOT, 'browse', 'src', 'browse-client.ts'), 'utf-8');
for (const dir of listBundledSkillDirs()) {
const sibling = fs.readFileSync(path.join(dir, '_lib', 'browse-client.ts'), 'utf-8');
expect(sibling).toBe(canonical);
}
});
test('script.ts imports browse from ./_lib/browse-client', () => {
for (const dir of listBundledSkillDirs()) {
const content = fs.readFileSync(path.join(dir, 'script.ts'), 'utf-8');
expect(content).toMatch(/from\s+['"]\.\/_lib\/browse-client['"]/);
}
});
});