CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-05-02 03:35:09 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
2014557e7f096817c5fdd4b965c4c10eb40d0eb7
gstack/test/helpers/touchfiles.ts
T
Garry Tan 2014557e7f v1.12.0.0 feat: /setup-gbrain — coding-agent onboarding for gbrain (#1183) 
* feat(setup-gbrain): add gstack-gbrain-repo-policy bin helper

Per-remote trust-tier store for the forthcoming /setup-gbrain skill.
Tiers are the D3 triad (read-write / read-only / deny), keyed by a
normalized remote URL so ssh-shorthand and https variants collapse to
the same entry. The file carries _schema_version: 2 (D2-eng); legacy
`allow` values from pre-D3 experiments auto-migrate to `read-write`
on first read, idempotent, with a one-shot log line.

Pure bash + jq to match the existing gstack-brain-* family. Atomic
writes via tmpfile + rename. Policy file mode 0600. Corrupt files
quarantine to .corrupt-<ts> and start fresh.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(setup-gbrain): unit tests for gstack-gbrain-repo-policy

24 tests covering normalize (ssh/https/shorthand/uppercase collapse to
one key), set/get round-trip, all three D3 tiers accepted, invalid
tiers rejected, file mode 0600, _schema_version field written on fresh
files, legacy allow migration (including idempotence and preservation
of non-allow entries), corrupt-JSON quarantine + fresh-file recovery,
list output sorting, and get-without-arg auto-detect against a git
repo with no origin.

All tests green against a per-test tmpdir GSTACK_HOME so nothing
leaks into the real ~/.gstack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(setup-gbrain): add gstack-gbrain-detect state reporter

Pure-introspection JSON emitter for the /setup-gbrain skill's
start-up branching. Reports: gbrain presence + version on PATH,
~/.gbrain/config.json existence + engine, `gbrain doctor --json`
health (wrapped in timeout 5s to match the /health D6 pattern),
gstack-brain-sync mode via gstack-config, and ~/.gstack/.git
presence for the memory-sync feature.

Never modifies state. Always emits valid JSON even when every check
is false. Handles malformed ~/.gbrain/config.json without crashing
— gbrain_engine is null in that case, not an error.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(setup-gbrain): add gstack-gbrain-install with D5 detect-first + D19 PATH-shadow guard

Clones gbrain at a pinned commit (v0.18.2) and registers it via
`bun link`. Before any clone:

  D5 detect-first — probes ~/git/gbrain, ~/gbrain, and the install
  target for a valid pre-existing clone (package.json with name
  "gbrain" and bin.gbrain set). If one is found, `bun link` runs
  there instead of cloning a second copy. Prevents the day-one
  duplicate-install footgun on the skill author's own machine.

After install:

  D19 PATH-shadow guard — reads the install-dir's package.json
  version, compares to `gbrain --version` on PATH. On mismatch:
  exits 3, prints every gbrain binary on PATH via `type -a`, and
  gives a remediation menu. Setup skills refuse broken environments
  instead of warning and continuing.

Prereq checks (bun, git, https://github.com reachability) fail fast
with install hints. --dry-run and --validate-only flags let the
skill probe the plan without touching state; tests use them to
cover D5 and D19 without exercising real bun link.

Pin is a load-bearing version: setup-gbrain v1 verified against
gbrain v0.18.2. Updating requires re-running Pre-Impl Gate 1 to
verify gbrain's CLI + config shapes haven't drifted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(setup-gbrain): unit tests for gstack-gbrain-detect + install

15 tests covering: detect emits valid JSON when nothing configured,
reports gstack_brain_git on GSTACK_HOME/.git presence, reads
~/.gbrain/config.json engine, tolerates malformed config, detects
a mocked gbrain binary on PATH with version parsing.

For install: D5 detect-first uses ~/git/gbrain fixtures under a
sandboxed HOME, verifies fall-through to fresh clone when no valid
clone exists, rejects invalid package.json shapes. D19 PATH-shadow
validation uses a fake gbrain on a minimal SAFE_PATH to simulate
version mismatch, same-version-pass, v-prefix tolerance, missing
binary on PATH, and missing version field in package.json.

--validate-only mode in the install bin makes the D19 check unit-
testable without running real bun link (which touches ~/.bun/bin).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(setup-gbrain): add gstack-gbrain-lib.sh with read_secret_to_env (D3-eng)

Shared secret-read helper for PAT (D11) and pooler URL paste (D16).
One implementation of the hardest-to-get-right pattern: stty -echo +
SIGINT/TERM/EXIT trap that restores terminal mode, read into a named
env var, optional redacted preview.

Validates the target var name against [A-Z_][A-Z0-9_]* to prevent
bash name-injection via `read -r "$varname"`. When stdin is not a TTY
(CI, piped tests) the stty branches skip cleanly — piped input doesn't
echo anyway. Exports the var after read so subprocesses inherit it;
callers own the `unset` at handoff time.

Sourced, not executed — no +x bit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(setup-gbrain): add gstack-gbrain-supabase-verify structural URL check

Zero-network validator for Supabase Session Pooler URLs before handing
them to `gbrain init`. Canonical shape verified per gbrain init.ts:266:
  postgresql://postgres.<ref>:<password>@aws-0-<region>.pooler.supabase.com:6543/postgres

Rejects direct-connection URLs (db.*.supabase.co:5432) with a distinct
exit code 3 and clear IPv6-failure remediation — that's the most common
paste mistake users make, so it earns its own UX path rather than a
generic "bad URL" error.

Never echoes the URL (contains a password) in error messages; tests
verify a distinct seed password never appears in stderr on any reject
path. Accepts URL from argv[1] or stdin ("-" or no arg).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(setup-gbrain): unit tests for supabase-verify + lib.sh secret helper

22 tests. verify: accepts canonical pooler URL (argv + stdin modes),
rejects direct-connection URL with exit 3, rejects wrong scheme, wrong
port, empty password, missing userinfo, plain 'postgres' user (catches
direct-URL paste errors), wrong host, empty URL. Case-insensitive host
match. Explicit negative: error messages never echo the URL password.

lib.sh read_secret_to_env: reads piped stdin into the named env var,
exports to subprocesses, redacted-preview emits masked form on stderr
with the seed password absent, rejects invalid var names (lowercase,
leading digit, hyphens), rejects missing/unknown flags, secret value
never appears on stdout.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(setup-gbrain): add gstack-gbrain-supabase-provision Management API wrapper

Four subcommands: list-orgs, create, wait, pooler-url. Built against
the verified Supabase Management API shape (Pre-Impl Gate 1):

  - POST /v1/projects with {name, db_pass, organization_slug, region}
    — not the original plan's /v1/organizations/{ref}/projects
  - No `plan` field; subscription tier is org-level per the OpenAPI
    description ("Subscription Plan is now set on organization level
    and is ignored in this request")
  - GET /v1/projects/{ref}/config/database/pooler for pooler config
    — not /config/database

Secrets discipline: SUPABASE_ACCESS_TOKEN (PAT) and DB_PASS read from
env only, never from argv (D8 grep test enforces this). `set +x` at
the top as a defensive default so debug tracing never leaks secrets.
Management API hostname hardcoded to SUPABASE_API_BASE env override —
no user-controlled URL portion (SSRF guard).

HTTP error paths: 401/403 → exit 3 (auth), 402 → 4 (quota), 409 → 5
(conflict), 429 + 5xx → exponential-backoff retry up to 3 attempts,
then exit 8. Wait subcommand polls every 5s until ACTIVE_HEALTHY
with a configurable timeout; terminal states (INIT_FAILED, REMOVED,
etc.) exit 7 immediately with a clear message. Timeout emits the
--resume-provision hint so the skill can recover.

Pooler-url constructs the URL locally from db_user/host/port/name +
DB_PASS rather than trusting the API response's connection_string
field, which is templated with [PASSWORD] rather than the real value.
Handles both object and array response shapes, preferring session
pool_mode when Supabase returns multiple pooler configs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(setup-gbrain): unit tests for gstack-gbrain-supabase-provision via mock API

22 tests covering D21 HTTP error suite (401/403/402/409/429/5xx) and
happy paths for all four subcommands. Every test spins up a Bun.serve
mock server bound to SUPABASE_API_BASE so nothing hits the real API.

Uses Bun.spawn (async) rather than spawnSync because spawnSync blocks
the Bun event loop, which prevents Bun.serve mocks from responding —
calls would hit curl's own timeout instead of round-tripping.

Verifies: POST body contains organization_slug (not organization_id)
and no `plan` field, bearer-token auth header, retry-on-429 with
eventual success, exit-8 on persistent 5xx after max retries, wait
succeeds on ACTIVE_HEALTHY, exits 7 on INIT_FAILED, exits 6 with
--resume-provision hint on timeout, pooler-url builds URL locally
from db_user/host/port/name + DB_PASS (not response connection_string
template), handles array pooler responses.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(setup-gbrain): add SKILL.md.tmpl — user-facing skill prompt

Stitches together every slice built so far (repo-policy, detect,
install, lib.sh secret helper, supabase-verify, supabase-provision)
into a single interactive flow. Paths: Supabase existing-URL, Supabase
auto-provision (D7), Supabase manual, PGLite local, switch (PGLite ↔
Supabase via gbrain migrate wrapped in timeout 180s per D9).

Secrets discipline per D8/D10/D11: PAT + DB_PASS + pooler URL all
read via read_secret_to_env from lib.sh and handed to gbrain via
GBRAIN_DATABASE_URL env, never argv. PAT carries the full D11 scope
disclosure before collection and an explicit revocation reminder after
success. D12 SIGINT recovery prints the in-flight ref + resume command.

D18 MCP registration is scoped honestly to Claude Code — skips with
a manual-register hint when `claude` is not on PATH. D6 per-remote
trust-triad question (read-write/read-only/deny/skip-for-now) gates
repo import; the triad values compose with the D2-eng schema-version
policy file so future migrations stay deterministic.

Skill runs concurrent-run-locked via mkdir ~/.gstack/.setup-gbrain.lock.d
(atomic, same pattern as gstack-brain-sync). Telemetry (D4) payload
carries enumerated categorical values only — never URL, PAT, or any
postgresql:// substring.

--repo, --switch, --resume-provision, --cleanup-orphans shortcut modes
documented inline; the skill parses its own invocation args.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(health): integrate gbrain as D6 composite dimension

Adds a GBrain row to the /health dashboard rubric with weight 10%.
Three sub-signals rolled into one 0-10 score: doctor status (0.5),
sync queue depth (0.3), last-push age (0.2). Redistributes when
gbrain_sync_mode is off so the dimension stays fair.

Weights rebalance: typecheck 25→22, lint 20→18, test 30→28,
deadcode 15→13, shell 10→9, gbrain +10 — sums to 100.

gbrain doctor --json wrapped in timeout 5s so a hung gbrain never
stalls the /health dashboard. Dimension is omitted (not red) when
gbrain is not installed — running /health on a non-gbrain machine
shouldn't penalize that choice.

History-JSONL adds a `gbrain` field. Pre-D6 entries read as null for
trend comparison; new tracking starts from first post-D6 run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): add secret-sink-harness for negative-space leak testing (D21 #5)

Runs a subprocess with a seeded secret, captures every channel the
subprocess could leak through, and asserts the seed never appears.
Built per the D1-eng tightened contract: per-run tmp $HOME, four seed
match rules (exact + URL-decoded + first-12-char prefix + base64),
fd-level stdout/stderr capture via Bun.spawn, post-mortem walk of
every file written under $HOME, separate buckets for telemetry JSONL.

Reusable: any future skill that handles secrets can import
runWithSecretSink and run positive/negative controls against its own
bins. The harness itself is ~180 lines of TS with no external deps
beyond Bun + node:fs.

Out of scope for v1 (documented as follow-ups): subprocess env dump
(portable /proc reading), the user's real shell history (bins don't
modify it).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: secret-sink harness positive controls + real-bin negative controls

11 tests. Positive controls deliberately leak a seed in every covered
channel (stdout, stderr, a file under $HOME, the telemetry JSONL path,
base64-encoded, first-12-char prefix) and assert the harness catches
each one. Without these, a harness that silently under-reports would
look identical to a harness that works.

Negative controls run real setup-gbrain bins with distinctive seeds:
  - supabase-verify rejects a mysql:// URL and a direct-connection URL,
    password never appears in any captured channel
  - lib.sh read_secret_to_env reads piped stdin, emits only the length,
    seed value stays invisible
  - supabase-provision on an auth-failure path fails fast without
    leaking the PAT to any channel

Covers D21 #5 leak harness + uses it to validate D3-eng, D10, D11
discipline end-to-end on the already-shipped bins.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(setup-gbrain): add list-orphans + delete-project subcommands (D20)

Powers /setup-gbrain --cleanup-orphans. list-orphans filters the
authenticated user's Supabase projects by name prefix (default
"gbrain") and excludes the project the local ~/.gbrain/config.json
currently points at, so only unclaimed gbrain-shaped projects come
back. Active-ref detection parses the pooler URL's user portion
(postgres.<ref>:<pw>@...).

delete-project is a thin DELETE /v1/projects/{ref} wrapper with no
confirmation of its own — the skill's UI layer owns the per-project
confirm AskUserQuestion loop. Keeps responsibilities clean: the bin
manages HTTP; the skill manages user intent.

Both subcommands reuse the existing api_call retry+backoff and the
same PAT discipline (env only, never argv).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(setup-gbrain): list-orphans active-ref filtering + delete-project 404

6 new tests bringing the supabase-provision suite to 28:

list-orphans:
  - Filters to gbrain-prefixed projects, excludes the active-ref derived
    from ~/.gbrain/config.json's pooler URL
  - Treats all gbrain-prefixed projects as orphans when no config exists
    (first run on a new machine)
  - Respects custom --name-prefix for users who named their brain
    something else

delete-project:
  - Happy path sends DELETE /v1/projects/<ref> and returns {deleted_ref}
  - 404 surfaces cleanly (exit 2, "404" in stderr)
  - Missing <ref> positional rejected with exit 2

Uses per-test tmpdir HOME with a stubbed ~/.gbrain/config.json so
active-ref extraction runs against deterministic fixtures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate setup-gbrain SKILL.md after main merge

* chore: bump version and changelog (v1.12.0.0)

Ships /setup-gbrain and its supporting infrastructure end-to-end:
per-remote trust policy, installer with PATH-shadow guard, shared
secret-read helper, structural URL verifier, Supabase Management
API wrapper, /health GBrain dimension, secret-sink test harness.

100 new tests across 5 suites, all green. Three pre-existing test
failures noted as P0 in TODOS.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: add USING_GBRAIN_WITH_GSTACK.md + update README for /setup-gbrain

README changes:
- Rewrote the "Cross-machine memory with GBrain sync" section into
  "GBrain — persistent knowledge for your coding agent." Covers the
  three /setup-gbrain paths (Supabase existing URL, auto-provision,
  PGLite local), MCP registration, per-remote trust triad, and the
  (still-separate) memory sync feature.
- Added /setup-gbrain row to the skills table pointing at the full guide.
- Added /setup-gbrain to both skill-list install snippets.
- Added USING_GBRAIN_WITH_GSTACK.md to the Docs table.

New doc (USING_GBRAIN_WITH_GSTACK.md):
- All three setup paths with trust-surface caveats
- MCP registration details (and honest Claude-Code-v1 scoping)
- Per-remote trust triad semantics + how to change a policy
- Switching engines (PGLite ↔ Supabase) via --switch
- GStack memory sync + its relationship to the gbrain knowledge base
- /setup-gbrain --cleanup-orphans for orphan Supabase projects
- Full command + flag reference, every bin helper, every env var
- Security model: what's enforced in code, what's enforced by the leak
  harness, and the honest limits of v1
- Troubleshooting: PATH shadowing, direct-connection URL reject,
  auto-provision timeout, stale lock, policy file hand-edits,
  migrate hang
- Why-this-design section explaining the non-obvious choices

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(brain-sync): secret scanner now catches Bearer-prefixed auth tokens in JSON

The bearer-token-json regex value charset was [A-Za-z0-9_./+=-]{16,},
which does NOT permit spaces. Real HTTP auth headers embed the scheme
name with a literal space — "Bearer <token>" — so the value portion
actually starts with "Bearer " and the existing regex couldn't match.
Result: any JSON blob containing "authorization":"Bearer ..." would
slip past the scanner and sync to the user's private brain repo with
the bearer token inline.

Added optional (Bearer |Basic |Token )? prefix in front of the value
charset. Now matches the common auth-scheme forms without broadening
the matcher to tolerate arbitrary whitespace (which would false-positive
on lots of benign JSON).

Verified against 5 positive cases (bearer-in-json, clean bearer, apikey
no-prefix, token with Bearer, password no-prefix) + 3 negative cases
(too-short tokens, non-secret field names like username, random JSON).

This closes the P0 security regression first noticed during v1.12.0.0
/ship. brain-sync.test.ts now passes all 7 secret-scan fixtures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: mock-gh integration tests for gstack-brain-init auto-create path

8 tests covering the gh-repo-create happy path that had zero coverage
before. Existing brain-sync.test.ts always passes --remote <bare-url>
to bypass gh entirely, so the interactive default ("press Enter, we'll
run gh repo create for you") was shipping on trust.

Test strategy: write a bash stub for gh that records every call into
a file, then run gstack-brain-init with that stub on PATH. Assertions
verify: gh auth status is checked, gh repo create fires with the
computed gstack-brain-<user> default name + --private + --source
flags, fall-through to gh repo view when create reports already-exists,
user-provided URL bypasses gh entirely, gh-not-on-path and gh-not-authed
branches both prompt for URL, --remote flag short-circuits all gh
calls, conflicting-remote re-runs exit 1 with a clear message.

No real GitHub, no live auth. Gate tier — runs on every commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): privacy-gate AskUserQuestion fires from preamble (periodic tier)

Two periodic-tier E2E tests exercising the preamble's privacy gate
end-to-end via the Agent SDK + canUseTool. Previously uncovered:

- Positive: stages a fake gbrain on PATH + gbrain_sync_mode_prompted=false
  in config, runs a real skill, intercepts tool-use. Asserts the
  preamble fires a 3-option AskUserQuestion matching the canonical
  prose ("publish session memory" / "artifact" / "decline") and does
  NOT fire a second time in the same run (idempotency within session).

- Negative: same staging but prompted=true. Asserts the gate stays
  silent even with gbrain detected on the host.

Registered in test/helpers/touchfiles.ts as `brain-privacy-gate`
(periodic) with dependency tracking on generate-brain-sync-block.ts,
the three gstack-brain-* bins, gstack-config, and the Agent SDK runner.
Diff-based selection re-runs the E2E when any of those change.

Cost: ~$0.30-$0.50 per run. Only fires under EVALS=1 EVALS_TIER=periodic;
gate tier stays free.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update TODOS for bearer-json fix + new brain-sync test coverage

Moves the bearer-json secret-scan regression from the P0 "pre-existing
failures" block into the Completed section with full context on the
fix, the mock-gh tests, the E2E privacy-gate tests, and the touchfile
registration. Remaining P0s are the GSTACK_HOME config-isolation bug
and the stale Opus 4.7 overlay pacing assertion, both unrelated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): E2E privacy gate — ambient env + skill-file prompt

Two fixes to get the E2E actually running end-to-end (first attempt
failed at the SDK auth step, second at the assertion step):

1. Don't pass an explicit `env:` object to runAgentSdkTest. The SDK's
   auth pipeline misses ANTHROPIC_API_KEY when env is supplied as an
   object (verified against the plan-mode-no-op test, which passes no
   env and auths cleanly). Mutate process.env before the call instead,
   and restore the originals in finally so other tests don't inherit
   the ambient mutation.

2. The "Run /learn with no arguments" user prompt was too narrow — the
   model reduced it to a direct action and skipped the preamble
   privacy-gate directives entirely, so zero AskUserQuestions fired.
   Mirror the plan-mode-no-op pattern: point the model at the skill
   file on disk and ask it to follow every preamble directive. Bumped
   maxTurns from 6 to 10 to give the preamble room to execute.

Verified both tests pass under `EVALS=1 EVALS_TIER=periodic bun test
test/skill-e2e-brain-privacy-gate.test.ts` against a real ANTHROPIC_API_KEY.
Cost per run: ~$0.30-$0.50 per test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(CLAUDE.md): source ANTHROPIC/OPENAI keys from ~/.zshrc for paid evals

Conductor workspaces don't inherit the interactive shell env, so
both API keys are absent from the default process env even though
they're set in ~/.zshrc. Documents the source-from-zshrc pattern
(grep + eval, never echo the value) plus the Agent SDK gotcha: do
NOT pass env as an object to runAgentSdkTest — mutate process.env
ambiently and restore in finally.

Discovered this during the brain-privacy-gate E2E. First run failed
at SDK auth with 401; second failed because explicit env handoff
bypassed the SDK's own auth routing. Fix pattern now codified so
the next paid-eval session in a Conductor workspace doesn't hit the
same two dead ends.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 01:38:21 -07:00

602 lines
33 KiB
TypeScript
Raw Blame History

/**
* Diff-based test selection for E2E and LLM-judge evals.
*
* Each test declares which source files it depends on ("touchfiles").
* The test runner checks `git diff` and only runs tests whose
* dependencies were modified. Override with EVALS_ALL=1 to run everything.
*/
import { spawnSync } from 'child_process';
// --- Glob matching ---
/**
* Match a file path against a glob pattern.
* Supports:
* ** — match any number of path segments
* * — match within a single segment (no /)
*/
export function matchGlob(file: string, pattern: string): boolean {
const regexStr = pattern
.replace(/\./g, '\\.')
.replace(/\*\*/g, '{{GLOBSTAR}}')
.replace(/\*/g, '[^/]*')
.replace(/\{\{GLOBSTAR\}\}/g, '.*');
return new RegExp(`^${regexStr}$`).test(file);
}
// --- Touchfile maps ---
/**
* E2E test touchfiles — keyed by testName (the string passed to runSkillTest).
* Each test lists the file patterns that, if changed, require the test to run.
*/
export const E2E_TOUCHFILES: Record<string, string[]> = {
// Browse core (+ test-server dependency)
'browse-basic': ['browse/src/**', 'browse/test/test-server.ts'],
'browse-snapshot': ['browse/src/**', 'browse/test/test-server.ts'],
// SKILL.md setup + preamble (depend on ROOT SKILL.md + gen-skill-docs)
'skillmd-setup-discovery': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'skillmd-no-local-binary': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'skillmd-outside-git': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'session-awareness': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'operational-learning': ['scripts/resolvers/preamble.ts', 'bin/gstack-learnings-log'],
// QA (+ test-server dependency)
'qa-quick': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts'],
'qa-b6-static': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval.html', 'test/fixtures/qa-eval-ground-truth.json'],
'qa-b7-spa': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval-spa.html', 'test/fixtures/qa-eval-spa-ground-truth.json'],
'qa-b8-checkout': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval-checkout.html', 'test/fixtures/qa-eval-checkout-ground-truth.json'],
'qa-only-no-fix': ['qa-only/**', 'qa/templates/**'],
'qa-fix-loop': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts'],
'qa-bootstrap': ['qa/**', 'ship/**'],
// Review
'review-sql-injection': ['review/**', 'test/fixtures/review-eval-vuln.rb'],
'review-enum-completeness': ['review/**', 'test/fixtures/review-eval-enum*.rb'],
'review-base-branch': ['review/**'],
'review-design-lite': ['review/**', 'test/fixtures/review-eval-design-slop.*'],
// Review Army (specialist dispatch)
'review-army-migration-safety': ['review/**', 'scripts/resolvers/review-army.ts', 'bin/gstack-diff-scope'],
'review-army-perf-n-plus-one': ['review/**', 'scripts/resolvers/review-army.ts', 'bin/gstack-diff-scope'],
'review-army-delivery-audit': ['review/**', 'scripts/resolvers/review.ts', 'scripts/resolvers/review-army.ts'],
'review-army-quality-score': ['review/**', 'scripts/resolvers/review-army.ts'],
'review-army-json-findings': ['review/**', 'scripts/resolvers/review-army.ts'],
'review-army-red-team': ['review/**', 'scripts/resolvers/review-army.ts'],
'review-army-consensus': ['review/**', 'scripts/resolvers/review-army.ts'],
// Office Hours
'office-hours-spec-review': ['office-hours/**', 'scripts/gen-skill-docs.ts'],
'office-hours-forcing-energy': ['office-hours/**', 'scripts/resolvers/preamble.ts', 'test/fixtures/mode-posture/**', 'test/helpers/llm-judge.ts'],
'office-hours-builder-wildness': ['office-hours/**', 'scripts/resolvers/preamble.ts', 'test/fixtures/mode-posture/**', 'test/helpers/llm-judge.ts'],
// Plan reviews
'plan-ceo-review': ['plan-ceo-review/**'],
'plan-ceo-review-selective': ['plan-ceo-review/**'],
'plan-ceo-review-benefits': ['plan-ceo-review/**', 'scripts/gen-skill-docs.ts'],
'plan-ceo-review-expansion-energy': ['plan-ceo-review/**', 'scripts/resolvers/preamble.ts', 'test/fixtures/mode-posture/**', 'test/helpers/llm-judge.ts'],
'plan-eng-review': ['plan-eng-review/**'],
'plan-eng-review-artifact': ['plan-eng-review/**'],
'plan-review-report': ['plan-eng-review/**', 'scripts/gen-skill-docs.ts'],
// Plan-mode handshake (v1.10.2.0) — gate-tier safety regression tests.
// Each fires when any of: the interactive skill's template, the resolver,
// preamble composition, the Agent SDK harness, the question registry, or
// the one-way-door classifier changes.
'plan-ceo-review-plan-mode': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'scripts/question-registry.ts', 'scripts/one-way-doors.ts', 'test/helpers/agent-sdk-runner.ts'],
'plan-eng-review-plan-mode': ['plan-eng-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'scripts/question-registry.ts', 'scripts/one-way-doors.ts', 'test/helpers/agent-sdk-runner.ts'],
'plan-design-review-plan-mode-handshake': ['plan-design-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'scripts/question-registry.ts', 'scripts/one-way-doors.ts', 'test/helpers/agent-sdk-runner.ts'],
'plan-devex-review-plan-mode': ['plan-devex-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'scripts/question-registry.ts', 'scripts/one-way-doors.ts', 'test/helpers/agent-sdk-runner.ts'],
'plan-mode-no-op': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/agent-sdk-runner.ts'],
'e2e-harness-audit': ['plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'test/helpers/agent-sdk-runner.ts'],
'brain-privacy-gate': ['scripts/resolvers/preamble/generate-brain-sync-block.ts', 'scripts/resolvers/preamble.ts', 'bin/gstack-brain-sync', 'bin/gstack-brain-init', 'bin/gstack-config', 'test/helpers/agent-sdk-runner.ts'],
// AskUserQuestion format regression (RECOMMENDATION + Completeness: N/10)
// Fires when either template OR the two preamble resolvers change.
'plan-ceo-review-format-mode': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-ceo-review-format-approach': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-eng-review-format-coverage': ['plan-eng-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-eng-review-format-kind': ['plan-eng-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
// v1.7.0.0 Pros/Cons format cadence + format + negative-escape evals.
// Dependencies: same as format-mode + the 4 plan-review templates + overlay.
// All periodic-tier (non-deterministic Opus 4.7 behavior).
'plan-ceo-review-prosons-cadence': ['plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-review-prosons-format': ['plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-review-prosons-hardstop-neg': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-review-prosons-neutral-neg': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
// Expanded coverage (CT3) — 6 non-plan-review skills inherit Pros/Cons via preamble
'ship-prosons-format': ['ship/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'office-hours-prosons-format': ['office-hours/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'investigate-prosons-format': ['investigate/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'qa-prosons-format': ['qa/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'review-prosons-format': ['review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'design-review-prosons-format': ['design-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'document-release-prosons-format': ['document-release/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
// /plan-tune (v1 observational)
'plan-tune-inspect': ['plan-tune/**', 'scripts/question-registry.ts', 'scripts/psychographic-signals.ts', 'scripts/one-way-doors.ts', 'bin/gstack-question-log', 'bin/gstack-question-preference', 'bin/gstack-developer-profile'],
// Codex offering verification
'codex-offered-office-hours': ['office-hours/**', 'scripts/gen-skill-docs.ts'],
'codex-offered-ceo-review': ['plan-ceo-review/**', 'scripts/gen-skill-docs.ts'],
'codex-offered-design-review': ['plan-design-review/**', 'scripts/gen-skill-docs.ts'],
'codex-offered-eng-review': ['plan-eng-review/**', 'scripts/gen-skill-docs.ts'],
// Ship
'ship-base-branch': ['ship/**', 'bin/gstack-repo-mode'],
'ship-local-workflow': ['ship/**', 'scripts/gen-skill-docs.ts'],
'review-dashboard-via': ['ship/**', 'scripts/resolvers/review.ts', 'codex/**', 'autoplan/**', 'land-and-deploy/**'],
'ship-plan-completion': ['ship/**', 'scripts/gen-skill-docs.ts'],
'ship-plan-verification': ['ship/**', 'scripts/gen-skill-docs.ts'],
// Retro
'retro': ['retro/**'],
'retro-base-branch': ['retro/**'],
// Global discover
'global-discover': ['bin/gstack-global-discover.ts', 'test/global-discover.test.ts'],
// CSO
'cso-full-audit': ['cso/**'],
'cso-diff-mode': ['cso/**'],
'cso-infra-scope': ['cso/**'],
// Learnings
'learnings-show': ['learn/**', 'bin/gstack-learnings-search', 'bin/gstack-learnings-log', 'scripts/resolvers/learnings.ts'],
// Session Intelligence (timeline, context recovery, /context-save + /context-restore)
'timeline-event-flow': ['bin/gstack-timeline-log', 'bin/gstack-timeline-read'],
'context-recovery-artifacts': ['scripts/resolvers/preamble.ts', 'bin/gstack-timeline-log', 'bin/gstack-slug', 'learn/**'],
'context-save-writes-file': ['context-save/**', 'bin/gstack-slug'],
'context-restore-loads-latest': ['context-restore/**', 'bin/gstack-slug'],
// Context skills E2E (live-fire, Skill-tool routing path) — see
// test/skill-e2e-context-skills.test.ts. These are periodic-tier because
// each one spawns claude -p and costs ~$0.20-$0.40. Collectively they
// verify the thing the /checkpoint → /context-save rename was for.
'context-save-routing': ['context-save/**', 'scripts/resolvers/preamble.ts'],
'context-save-then-restore-roundtrip': ['context-save/**', 'context-restore/**', 'bin/gstack-slug'],
'context-restore-fragment-match': ['context-restore/**'],
'context-restore-empty-state': ['context-restore/**'],
'context-restore-list-delegates': ['context-restore/**'],
'context-restore-legacy-compat': ['context-restore/**'],
'context-save-list-current-branch': ['context-save/**'],
'context-save-list-all-branches': ['context-save/**'],
// Document-release
'document-release': ['document-release/**'],
// Codex (Claude E2E — tests /codex skill via Claude)
'codex-review': ['codex/**'],
// Codex E2E (tests skills via Codex CLI + worktree)
'codex-discover-skill': ['codex/**', '.agents/skills/**', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts'],
'codex-review-findings': ['review/**', '.agents/skills/gstack-review/**', 'codex/**', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts'],
// Gemini E2E — smoke test only (Gemini gets lost in worktrees on complex tasks)
'gemini-smoke': ['.agents/skills/**', 'test/helpers/gemini-session-runner.ts', 'lib/worktree.ts'],
// Coverage audit (shared fixture) + triage + gates
'ship-coverage-audit': ['ship/**', 'test/fixtures/coverage-audit-fixture.ts', 'bin/gstack-repo-mode'],
'review-coverage-audit': ['review/**', 'test/fixtures/coverage-audit-fixture.ts'],
'plan-eng-coverage-audit': ['plan-eng-review/**', 'test/fixtures/coverage-audit-fixture.ts'],
'ship-triage': ['ship/**', 'bin/gstack-repo-mode'],
// Plan completion audit + verification
'ship-plan-completion': ['ship/**', 'scripts/gen-skill-docs.ts'],
'ship-plan-verification': ['ship/**', 'qa-only/**', 'scripts/gen-skill-docs.ts'],
'ship-idempotency': ['ship/**', 'scripts/resolvers/utility.ts'],
'review-plan-completion': ['review/**', 'scripts/gen-skill-docs.ts'],
// Design
'design-consultation-core': ['design-consultation/**', 'scripts/gen-skill-docs.ts', 'test/helpers/llm-judge.ts'],
'design-consultation-existing': ['design-consultation/**', 'scripts/gen-skill-docs.ts'],
'design-consultation-research': ['design-consultation/**', 'scripts/gen-skill-docs.ts'],
'design-consultation-preview': ['design-consultation/**', 'scripts/gen-skill-docs.ts'],
'plan-design-review-plan-mode': ['plan-design-review/**', 'scripts/gen-skill-docs.ts'],
'plan-design-review-no-ui-scope': ['plan-design-review/**', 'scripts/gen-skill-docs.ts'],
'design-review-fix': ['design-review/**', 'browse/src/**', 'scripts/gen-skill-docs.ts'],
// Design Shotgun
'design-shotgun-path': ['design-shotgun/**', 'design/src/**', 'scripts/resolvers/design.ts'],
'design-shotgun-session': ['design-shotgun/**', 'scripts/resolvers/design.ts'],
'design-shotgun-full': ['design-shotgun/**', 'design/src/**', 'browse/src/**'],
// gstack-upgrade
'gstack-upgrade-happy-path': ['gstack-upgrade/**'],
// Deploy skills
'land-and-deploy-workflow': ['land-and-deploy/**', 'scripts/gen-skill-docs.ts'],
'land-and-deploy-first-run': ['land-and-deploy/**', 'scripts/gen-skill-docs.ts', 'bin/gstack-slug'],
'land-and-deploy-review-gate': ['land-and-deploy/**', 'bin/gstack-review-read'],
'canary-workflow': ['canary/**', 'browse/src/**'],
'benchmark-workflow': ['benchmark/**', 'browse/src/**'],
'setup-deploy-workflow': ['setup-deploy/**', 'scripts/gen-skill-docs.ts'],
// Sidebar agent
'sidebar-navigate': ['browse/src/server.ts', 'browse/src/sidebar-agent.ts', 'browse/src/sidebar-utils.ts', 'extension/**'],
'sidebar-url-accuracy': ['browse/src/server.ts', 'browse/src/sidebar-agent.ts', 'browse/src/sidebar-utils.ts', 'extension/background.js'],
'sidebar-css-interaction': ['browse/src/server.ts', 'browse/src/sidebar-agent.ts', 'browse/src/write-commands.ts', 'browse/src/read-commands.ts', 'browse/src/cdp-inspector.ts', 'extension/**'],
// Autoplan
'autoplan-core': ['autoplan/**', 'plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**'],
'autoplan-dual-voice': ['autoplan/**', 'codex/**', 'bin/gstack-codex-probe', 'scripts/resolvers/review.ts', 'scripts/resolvers/design.ts'],
// Multi-provider benchmark adapters — live API smoke against real claude/codex/gemini CLIs
'benchmark-providers-live': ['bin/gstack-model-benchmark', 'test/helpers/providers/**', 'test/helpers/benchmark-runner.ts', 'test/helpers/pricing.ts'],
// Skill routing — journey-stage tests (depend on ALL skill descriptions)
'journey-ideation': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-plan-eng': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-debug': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-qa': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-code-review': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-ship': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-docs': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-retro': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-design-system': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-visual-qa': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
// Opus 4.7 behavior evals — keys match testName: values in the test file.
// Routing sub-tests use template literal `routing-${c.name}` testNames,
// which the touchfile completeness scanner skips; they inherit selection
// from the file-level touchfile entry via GLOBAL_TOUCHFILES.
'fanout-arm-overlay-on':
['model-overlays/claude.md', 'model-overlays/opus-4-7.md', 'scripts/models.ts', 'scripts/resolvers/model-overlay.ts'],
'fanout-arm-overlay-off':
['model-overlays/claude.md', 'model-overlays/opus-4-7.md', 'scripts/models.ts', 'scripts/resolvers/model-overlay.ts'],
// Overlay efficacy harness (SDK) — measures whether overlay nudges change
// behavior under @anthropic-ai/claude-agent-sdk (closer to real Claude Code
// than `claude -p`). testNames in the file are template literals so the
// completeness scanner doesn't require them; these entries exist for
// diff-based selection accuracy.
'overlay-harness-opus-4-7-fanout-toy': [
'model-overlays/**',
'test/fixtures/overlay-nudges.ts',
'test/helpers/agent-sdk-runner.ts',
'scripts/resolvers/model-overlay.ts',
],
'overlay-harness-opus-4-7-fanout-realistic': [
'model-overlays/**',
'test/fixtures/overlay-nudges.ts',
'test/helpers/agent-sdk-runner.ts',
'scripts/resolvers/model-overlay.ts',
],
};
/**
* E2E test tiers — 'gate' blocks PRs, 'periodic' runs weekly/on-demand.
* Must have exactly the same keys as E2E_TOUCHFILES.
*/
export const E2E_TIERS: Record<string, 'gate' | 'periodic'> = {
// Browse core — gate (if browse breaks, everything breaks)
'browse-basic': 'gate',
'browse-snapshot': 'gate',
// SKILL.md setup — gate (if setup breaks, no skill works)
'skillmd-setup-discovery': 'gate',
'skillmd-no-local-binary': 'gate',
'skillmd-outside-git': 'gate',
'session-awareness': 'gate',
'operational-learning': 'gate',
// QA — gate for functional, periodic for quality/benchmarks
'qa-quick': 'gate',
'qa-b6-static': 'periodic',
'qa-b7-spa': 'periodic',
'qa-b8-checkout': 'periodic',
'qa-only-no-fix': 'gate', // CRITICAL guardrail: Edit tool forbidden
'qa-fix-loop': 'periodic',
'qa-bootstrap': 'gate',
// Review — gate for functional/guardrails, periodic for quality
'review-sql-injection': 'gate', // Security guardrail
'review-enum-completeness': 'gate',
'review-base-branch': 'gate',
'review-design-lite': 'periodic', // 4/7 threshold is subjective
'review-coverage-audit': 'gate',
'review-plan-completion': 'gate',
'review-dashboard-via': 'gate',
// Review Army — gate for core functionality, periodic for multi-specialist
'review-army-migration-safety': 'gate', // Specialist activation guardrail
'review-army-perf-n-plus-one': 'gate', // Specialist activation guardrail
'review-army-delivery-audit': 'gate', // Delivery integrity guardrail
'review-army-quality-score': 'gate', // Score computation
'review-army-json-findings': 'gate', // JSON schema compliance
'review-army-red-team': 'periodic', // Multi-agent coordination
'review-army-consensus': 'periodic', // Multi-specialist agreement
// Office Hours
'office-hours-spec-review': 'gate',
'office-hours-forcing-energy': 'gate', // V1.1 mode-posture regression gate (Sonnet generator)
'office-hours-builder-wildness': 'gate', // V1.1 mode-posture regression gate (Sonnet generator)
// Plan reviews — gate for cheap functional, periodic for Opus quality
'plan-ceo-review': 'periodic',
'plan-ceo-review-selective': 'periodic',
'plan-ceo-review-benefits': 'gate',
'plan-ceo-review-expansion-energy': 'gate', // V1.1 mode-posture regression gate (Opus generator, Sonnet judge)
'plan-eng-review': 'periodic',
'plan-eng-review-artifact': 'periodic',
'plan-eng-coverage-audit': 'gate',
'plan-review-report': 'gate',
// Plan-mode handshake — deterministic safety regression, gate-tier
'plan-ceo-review-plan-mode': 'gate',
'plan-eng-review-plan-mode': 'gate',
'plan-design-review-plan-mode-handshake': 'gate',
'plan-devex-review-plan-mode': 'gate',
'plan-mode-no-op': 'gate',
'e2e-harness-audit': 'gate',
// Privacy gate for gstack-brain-sync — periodic (non-deterministic LLM call,
// costs ~$0.30-$0.50 per run, not needed on every commit)
'brain-privacy-gate': 'periodic',
// AskUserQuestion format regression — periodic (Opus 4.7 non-deterministic benchmark)
'plan-ceo-review-format-mode': 'periodic',
'plan-ceo-review-format-approach': 'periodic',
'plan-eng-review-format-coverage': 'periodic',
'plan-eng-review-format-kind': 'periodic',
// v1.7.0.0 Pros/Cons format — cadence + negative-escape evals (all periodic)
'plan-ceo-review-prosons-cadence': 'periodic',
'plan-review-prosons-format': 'periodic',
'plan-review-prosons-hardstop-neg': 'periodic',
'plan-review-prosons-neutral-neg': 'periodic',
// CT3 expanded coverage — non-plan-review skills inheriting Pros/Cons (all periodic)
'ship-prosons-format': 'periodic',
'office-hours-prosons-format': 'periodic',
'investigate-prosons-format': 'periodic',
'qa-prosons-format': 'periodic',
'review-prosons-format': 'periodic',
'design-review-prosons-format': 'periodic',
'document-release-prosons-format': 'periodic',
// /plan-tune — gate (core v1 DX promise: plain-English intent routing)
'plan-tune-inspect': 'gate',
// Codex offering verification
'codex-offered-office-hours': 'gate',
'codex-offered-ceo-review': 'gate',
'codex-offered-design-review': 'gate',
'codex-offered-eng-review': 'gate',
// Session Intelligence — gate for data flow, periodic for agent integration
'timeline-event-flow': 'gate', // Binary data flow (no LLM needed)
'context-recovery-artifacts': 'gate', // Preamble reads seeded artifacts
'context-save-writes-file': 'gate', // /context-save writes a file
'context-restore-loads-latest': 'gate', // Cross-branch newest-by-filename restore
// Context skills live-fire — periodic (each test spawns claude -p, ~$0.20-$0.40)
'context-save-routing': 'periodic', // Proves /context-save routes via Skill tool
'context-save-then-restore-roundtrip': 'periodic', // Full cycle in one session
'context-restore-fragment-match': 'periodic', // /context-restore <fragment>
'context-restore-empty-state': 'periodic', // Graceful zero-saves message
'context-restore-list-delegates': 'periodic', // /context-restore list redirect
'context-restore-legacy-compat': 'periodic', // Pre-rename files still load
'context-save-list-current-branch': 'periodic', // Default branch filter
'context-save-list-all-branches': 'periodic', // --all flag
// Ship — gate (end-to-end ship path)
'ship-base-branch': 'gate',
'ship-local-workflow': 'gate',
'ship-coverage-audit': 'gate',
'ship-triage': 'gate',
'ship-plan-completion': 'gate',
'ship-plan-verification': 'gate',
'ship-idempotency': 'periodic',
// Retro — gate for cheap branch detection, periodic for full Opus retro
'retro': 'periodic',
'retro-base-branch': 'gate',
// Global discover
'global-discover': 'gate',
// CSO — gate for security guardrails, periodic for quality
'cso-full-audit': 'gate', // Hardcoded secrets detection
'cso-diff-mode': 'gate',
'cso-infra-scope': 'periodic',
// Learnings — gate (functional guardrail: seeded learnings must appear)
'learnings-show': 'gate',
// Document-release — gate (CHANGELOG guardrail)
'document-release': 'gate',
// Codex — periodic (Opus, requires codex CLI)
'codex-review': 'periodic',
// Multi-AI — periodic (require external CLIs)
'codex-discover-skill': 'periodic',
'codex-review-findings': 'periodic',
'gemini-smoke': 'periodic',
// Design — gate for cheap functional, periodic for Opus/quality
'design-consultation-core': 'periodic',
'design-consultation-existing': 'periodic',
'design-consultation-research': 'gate',
'design-consultation-preview': 'gate',
'plan-design-review-plan-mode': 'periodic',
'plan-design-review-no-ui-scope': 'gate',
'design-review-fix': 'periodic',
'design-shotgun-path': 'gate',
'design-shotgun-session': 'gate',
'design-shotgun-full': 'periodic',
// gstack-upgrade
'gstack-upgrade-happy-path': 'gate',
// Deploy skills
'land-and-deploy-workflow': 'gate',
'land-and-deploy-first-run': 'gate',
'land-and-deploy-review-gate': 'gate',
'canary-workflow': 'gate',
'benchmark-workflow': 'gate',
'setup-deploy-workflow': 'gate',
// Sidebar agent
'sidebar-navigate': 'periodic',
'sidebar-url-accuracy': 'periodic',
'sidebar-css-interaction': 'periodic',
// Autoplan — periodic (not yet implemented)
'autoplan-core': 'periodic',
'autoplan-dual-voice': 'periodic',
// Multi-provider benchmark — periodic (requires external CLIs + auth, paid)
'benchmark-providers-live': 'periodic',
// Skill routing — periodic (LLM routing is non-deterministic)
'journey-ideation': 'periodic',
'journey-plan-eng': 'periodic',
'journey-debug': 'periodic',
'journey-qa': 'periodic',
'journey-code-review': 'periodic',
'journey-ship': 'periodic',
'journey-docs': 'periodic',
'journey-retro': 'periodic',
'journey-design-system': 'periodic',
'journey-visual-qa': 'periodic',
// Opus 4.7 overlay evals — periodic (non-deterministic LLM behavior + Opus cost)
'fanout-arm-overlay-on': 'periodic',
'fanout-arm-overlay-off': 'periodic',
// Overlay efficacy harness (SDK, paid) — periodic only
'overlay-harness-opus-4-7-fanout-toy': 'periodic',
'overlay-harness-opus-4-7-fanout-realistic': 'periodic',
};
/**
* LLM-judge test touchfiles — keyed by test description string.
*/
export const LLM_JUDGE_TOUCHFILES: Record<string, string[]> = {
'command reference table': ['SKILL.md', 'SKILL.md.tmpl', 'browse/src/commands.ts'],
'snapshot flags reference': ['SKILL.md', 'SKILL.md.tmpl', 'browse/src/snapshot.ts'],
'browse/SKILL.md reference': ['browse/SKILL.md', 'browse/SKILL.md.tmpl', 'browse/src/**'],
'setup block': ['SKILL.md', 'SKILL.md.tmpl'],
'regression vs baseline': ['SKILL.md', 'SKILL.md.tmpl', 'browse/src/commands.ts', 'test/fixtures/eval-baselines.json'],
'qa/SKILL.md workflow': ['qa/SKILL.md', 'qa/SKILL.md.tmpl'],
'qa/SKILL.md health rubric': ['qa/SKILL.md', 'qa/SKILL.md.tmpl'],
'qa/SKILL.md anti-refusal': ['qa/SKILL.md', 'qa/SKILL.md.tmpl', 'qa-only/SKILL.md', 'qa-only/SKILL.md.tmpl'],
'cross-skill greptile consistency': ['review/SKILL.md', 'review/SKILL.md.tmpl', 'ship/SKILL.md', 'ship/SKILL.md.tmpl', 'review/greptile-triage.md', 'retro/SKILL.md', 'retro/SKILL.md.tmpl'],
'baseline score pinning': ['SKILL.md', 'SKILL.md.tmpl', 'test/fixtures/eval-baselines.json'],
// Ship & Release
'ship/SKILL.md workflow': ['ship/SKILL.md', 'ship/SKILL.md.tmpl'],
'document-release/SKILL.md workflow': ['document-release/SKILL.md', 'document-release/SKILL.md.tmpl'],
// Plan Reviews
'plan-ceo-review/SKILL.md modes': ['plan-ceo-review/SKILL.md', 'plan-ceo-review/SKILL.md.tmpl'],
'plan-eng-review/SKILL.md sections': ['plan-eng-review/SKILL.md', 'plan-eng-review/SKILL.md.tmpl'],
'plan-design-review/SKILL.md passes': ['plan-design-review/SKILL.md', 'plan-design-review/SKILL.md.tmpl'],
// Design skills
'design-review/SKILL.md fix loop': ['design-review/SKILL.md', 'design-review/SKILL.md.tmpl'],
'design-consultation/SKILL.md research': ['design-consultation/SKILL.md', 'design-consultation/SKILL.md.tmpl'],
// Office Hours
'office-hours/SKILL.md spec review': ['office-hours/SKILL.md', 'office-hours/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'office-hours/SKILL.md design sketch': ['office-hours/SKILL.md', 'office-hours/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
// Deploy skills
'land-and-deploy/SKILL.md workflow': ['land-and-deploy/SKILL.md', 'land-and-deploy/SKILL.md.tmpl'],
'canary/SKILL.md monitoring loop': ['canary/SKILL.md', 'canary/SKILL.md.tmpl'],
'benchmark/SKILL.md perf collection': ['benchmark/SKILL.md', 'benchmark/SKILL.md.tmpl'],
'setup-deploy/SKILL.md platform setup': ['setup-deploy/SKILL.md', 'setup-deploy/SKILL.md.tmpl'],
// Other skills
'retro/SKILL.md instructions': ['retro/SKILL.md', 'retro/SKILL.md.tmpl'],
'qa-only/SKILL.md workflow': ['qa-only/SKILL.md', 'qa-only/SKILL.md.tmpl'],
'gstack-upgrade/SKILL.md upgrade flow': ['gstack-upgrade/SKILL.md', 'gstack-upgrade/SKILL.md.tmpl'],
// Voice directive
'voice directive tone': ['scripts/resolvers/preamble.ts', 'review/SKILL.md', 'review/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
};
/**
* Changes to any of these files trigger ALL tests (both E2E and LLM-judge).
*
* Keep this list minimal — only files that genuinely affect every test.
* Scoped dependencies (gen-skill-docs, llm-judge, test-server, worktree,
* codex/gemini session runners) belong in individual test entries instead.
*/
export const GLOBAL_TOUCHFILES = [
'test/helpers/session-runner.ts', // All E2E tests use this runner
'test/helpers/eval-store.ts', // All E2E tests store results here
'test/helpers/touchfiles.ts', // Self-referential — reclassifying wrong is dangerous
];
// --- Base branch detection ---
/**
* Detect the base branch by trying refs in order.
* Returns the first valid ref, or null if none found.
*/
export function detectBaseBranch(cwd: string): string | null {
for (const ref of ['origin/main', 'origin/master', 'main', 'master']) {
const result = spawnSync('git', ['rev-parse', '--verify', ref], {
cwd, stdio: 'pipe', timeout: 3000,
});
if (result.status === 0) return ref;
}
return null;
}
/**
* Get list of files changed between base branch and HEAD.
*/
export function getChangedFiles(baseBranch: string, cwd: string): string[] {
const result = spawnSync('git', ['diff', '--name-only', `${baseBranch}...HEAD`], {
cwd, stdio: 'pipe', timeout: 5000,
});
if (result.status !== 0) return [];
return result.stdout.toString().trim().split('\n').filter(Boolean);
}
// --- Test selection ---
/**
* Select tests to run based on changed files.
*
* Algorithm:
* 1. If any changed file matches a global touchfile → run ALL tests
* 2. Otherwise, for each test, check if any changed file matches its patterns
* 3. Return selected + skipped lists with reason
*/
export function selectTests(
changedFiles: string[],
touchfiles: Record<string, string[]>,
globalTouchfiles: string[] = GLOBAL_TOUCHFILES,
): { selected: string[]; skipped: string[]; reason: string } {
const allTestNames = Object.keys(touchfiles);
// Global touchfile hit → run all
for (const file of changedFiles) {
if (globalTouchfiles.some(g => matchGlob(file, g))) {
return { selected: allTestNames, skipped: [], reason: `global: ${file}` };
}
}
// Per-test matching
const selected: string[] = [];
const skipped: string[] = [];
for (const [testName, patterns] of Object.entries(touchfiles)) {
const hit = changedFiles.some(f => patterns.some(p => matchGlob(f, p)));
(hit ? selected : skipped).push(testName);
}
return { selected, skipped, reason: 'diff' };
}
Reference in New Issue View Git Blame Copy Permalink