CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-05-01 19:25:10 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
dde55103fcc42bd446d804ddc15567ced8455ac1
gstack/test/helpers/touchfiles.ts
T
Garry Tan dde55103fc v1.15.0.0 feat: slim preamble + real-PTY plan-mode E2E harness (#1215) 
* chore: add gstack skill routing rules to CLAUDE.md

Per routing-injection preamble — once-per-project addition that lets
agents auto-invoke the right gstack skill instead of answering generically.

* refactor: slim preamble resolvers + sidecar-symlink helper

Compress prose across 18 preamble resolvers — Voice, Writing Style,
AskUserQuestion Format, Completeness Principle, Confusion Protocol,
Context Health, Context Recovery, Continuous Checkpoint, Lake Intro,
Proactive Prompt, Routing Injection, Telemetry Prompt, Upgrade Check,
Vendoring Deprecation, Writing Style Migration, Brain Sync Block,
Completion Status, and Question Tuning. Same semantic contract, ~half
the bytes. Restored "Treat the skill file as executable instructions"
phrase in the plan-mode info section after diagnosing it as load-bearing.
Restored "Effort both-scales" rule in AskUserQuestion format.

Bonus: scripts/skill-check.ts gains isRepoRootSymlink() so dev installs
that mount the repo root at host/skills/gstack as a runtime sidecar
(e.g., codex's .agents/skills/gstack) get skipped instead of double-counted.

opus-4-7 model overlay gets a Fan-Out directive — explicit instruction
to launch parallel reads/checks before synthesis.

Net token impact across all generated SKILL.md files: ~140K tokens
removed across 47 outputs. Plan-* skills retain full preamble surface
(Brain Sync, Context Recovery, Routing Injection) — load-bearing
functionality that early slim attempts incorrectly cut.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate SKILL.md outputs after preamble slim

bun run gen:skill-docs --host all output. Mirrors the resolver changes
in the previous commit. 47 generated SKILL.md files plus 3 ship-skill
golden fixtures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): real-PTY harness for plan-mode E2E tests

Adds test/helpers/claude-pty-runner.ts. Spawns the actual claude binary
via Bun.spawn({terminal:}) (Bun 1.3.10+ has built-in PTY — no node-pty,
no native modules), drives it through stdin/stdout, and parses rendered
terminal frames. Pattern adapted from the cc-pty-import branch's
terminal-agent.ts but stripped of WS/cookie/Origin scaffolding (not
needed for headless tests).

Public API:
- launchClaudePty(opts) — boots claude with --permission-mode plan|null,
  auto-handles the workspace-trust dialog, returns a session handle.
- session.send / sendKey / waitForAny / waitFor / mark / visibleSince /
  visibleText / rawOutput / close
- runPlanSkillObservation({skillName, inPlanMode, timeoutMs}) — high-level
  contract for plan-mode skill tests. Returns { outcome, summary, evidence,
  elapsedMs }. outcome ∈ {asked, plan_ready, silent_write, exited, timeout}.

Replaces the SDK-based runPlanModeSkillTest from plan-mode-helpers.ts
which never worked. Plan mode renders its native "Ready to execute"
confirmation as TTY UI (numbered options with ❯ cursor), not via the
AskUserQuestion tool — so the SDK's canUseTool interceptor never fired
and the assertion always saw zero questions. Real PTY observes the
rendered output directly.

Deletes test/helpers/plan-mode-helpers.ts. No production callers remained.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: rewrite 5 plan-mode E2E tests on the real-PTY harness

Replaces SDK-based assertions with runPlanSkillObservation contract. Each
test launches real claude --permission-mode plan, invokes the skill, and
asserts the outcome reaches 'asked' or 'plan_ready' within a 300s budget
(no silent Write/Edit, no crash, no timeout).

Affected:
- test/skill-e2e-plan-ceo-plan-mode.test.ts
- test/skill-e2e-plan-eng-plan-mode.test.ts
- test/skill-e2e-plan-design-plan-mode.test.ts
- test/skill-e2e-plan-devex-plan-mode.test.ts
- test/skill-e2e-plan-mode-no-op.test.ts (inPlanMode: false; tests the
  preamble plan-mode-info no-op path)

test/e2e-harness-audit.test.ts — recognize runPlanSkillObservation as a
valid coverage path alongside the legacy canUseTool / runPlanModeSkillTest.

test/helpers/touchfiles.ts — point the 5 plan-mode test selections and
the e2e-harness-audit selection at test/helpers/claude-pty-runner.ts
instead of the deleted plan-mode-helpers.ts.

Proof: bun test EVALS=1 EVALS_TIER=gate on these 5 files runs sequentially
in 790s and passes 5/5. Same tests were 0/5 on origin/main, on v1.0.0.0,
and on this branch with the SDK harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: align unit tests with slim resolvers + exempt 27MB security fixture

- test/skill-validation.test.ts: assert the slim Completeness Principle
  shape (Completeness: X/10, kind-note language) instead of the old
  Compression table. Remove the 3 tier-1 skills from the spot-check list
  (they intentionally don't carry the full Completeness Principle
  section). Exempt browse/test/fixtures/security-bench-haiku-responses.json
  (27MB deterministic replay fixture for BrowseSafe-Bench) from the 2MB
  tracked-file gate. The gate was actually failing on origin/main since
  the fixture was added in v1.6.4.0 — this is a side-fix to a real
  regression.

- test/brain-sync.test.ts: developer-machine-safe assertion for
  GSTACK_HOME override (compare config contents before/after instead of
  asserting the absence of a string that may legitimately exist).

- test/gen-skill-docs.test.ts: new tests for the slim — plan-review
  preambles stay under the post-slim budget (~33KB), Voice + Writing
  Style sections stay compact, and the slim Voice section preserves the
  load-bearing semantic contract (lead-with-the-point, name-the-file,
  user-outcome framing, no-corporate, no-AI-vocab, user-sovereignty).
  Update path-leakage scan to allow repo-root sidecar symlinks.

- test/writing-style-resolver.test.ts: assert the compact contract
  (gloss-on-first-use, outcome-framing, user-impact, terse-mode override)
  instead of the old 6-numbered-rules shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v1.13.1.0)

Slim preamble work + real-PTY plan-mode E2E harness on top of v1.13.0.0.
SKILL.md corpus -25.5% (3.08 MB → 2.30 MB, ~196K tokens). 5 plan-mode
tests go from 0/5 to 5/5 (790s sequential), the first time those tests
have ever passed. Side-fixes for the 27MB security fixture warning and
the sidecar-symlink double-count.

Reverts the Fan-Out directive accidentally restored to opus-4-7.md —
v1.10.1.0's overlay-efficacy harness measured -60pp fanout vs baseline
when the nudge was active. The intentional removal stays.

TODOS:
- Pre-existing test failures from v1.12.0.0 ship: RESOLVED on main + this branch
- security-bench-haiku-responses.json size gate: RESOLVED via warn-only + exemption

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): harness primitives — parseNumberedOptions + budget regression utils

claude-pty-runner.ts:
- parseNumberedOptions(visible) anchors on the latest "❯ 1." cursor and
  returns {index, label}[]; tests that route on option labels can find
  indices without hard-coding positions
- isPermissionDialogVisible(visible) detects file-grant + workspace-trust
  + bash-permission shapes (multiple regex variants)
- isNumberedOptionListVisible: replaced \b2\. word-boundary regex with
  [^0-9]2\. — stripAnsi removes TTY cursor-positioning escapes that
  collapse "Option 2." to "Option2.", and \b fails on word-to-word

eval-store.ts:
- findBudgetRegressions(comparison, opts?) — pure function returning
  tests where tools or turns grew >cap× vs prior run; floors at 5 prior
  tools / 3 prior turns to avoid noise on tiny numbers
- assertNoBudgetRegression() — wrapper that throws with full violation
  list. Env override GSTACK_BUDGET_RATIO

helpers-unit.test.ts: 23 unit tests covering empty/sparse/wrap-around
buffers for parseNumberedOptions, plus regression-floor + env-override
cases for findBudgetRegressions/assertNoBudgetRegression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: register 6 real-PTY E2E touchfiles + UI-heavy plan fixture

touchfiles.ts:
- 6 new entries in E2E_TOUCHFILES keyed to the new test files
- 6 matching E2E_TIERS classifications: 3 gate (auq-format-pty,
  plan-design-with-ui-scope, budget-regression-pty), 3 periodic
  (plan-ceo-mode-routing, ship-idempotency-pty, autoplan-chain-pty)
- gate ones are cheap/deterministic; periodic ones run weekly

touchfiles.test.ts:
- update the "skill-specific change selects only that skill" count
  from 15 → 18 (plan-ceo-review/SKILL.md change now also selects
  auq-format-pty, plan-ceo-mode-routing, autoplan-chain-pty)

test/fixtures/plans/ui-heavy-feature.md:
- planted plan with explicit UI scope keywords (pages, components,
  Tailwind responsive layout, hover/loading/empty states, modal,
  toast). Used by plan-design-with-ui-scope and autoplan-chain tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): 3 gate-tier real-PTY E2E tests

skill-e2e-auq-format-compliance.test.ts (~$0.50/run, 90-130s):
- Asserts /plan-ceo-review's first AUQ contains all 7 mandated format
  elements (ELI10, Recommendation, Pros/Cons with ✅/❌, Net,
  (recommended) label). Catches drift in the shared preamble resolver
  that previously took weeks to notice.
- Auto-grants permission dialogs that fire during preamble side-effects
  (touch on .feature-prompted markers in fresh user environments).
- Verified PASS in 126s.

skill-e2e-plan-design-with-ui.test.ts (~$0.80/run, 50-90s):
- Counterpart to the existing no-UI early-exit test. When the input plan
  DOES describe UI changes, /plan-design-review must NOT early-exit and
  must reach a real skill AUQ.
- Sends the slash command without args, then a follow-up message with
  the UI-heavy plan description (Claude Code rejects unknown trailing
  args). Asserts evidence does NOT contain "no UI scope".
- Verified PASS in 54s.

skill-budget-regression.test.ts (free, gate):
- Library-only assertion. Reads the most recent eval file, finds the
  prior same-branch run via findPreviousRun, computes ComparisonResult,
  asserts no test exceeded 2× tools or turns.
- Branch-scoped: skips with reason if the latest eval was produced on
  a different branch (cross-branch comparison would be noise).
- First-run grace (vacuous pass) when no prior data exists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(test): 3 periodic-tier real-PTY E2E tests

skill-e2e-plan-ceo-mode-routing.test.ts (~$3/run, 6-10 min/case):
- Verifies AUQ answer routing: HOLD SCOPE → rigor/bulletproof posture
  language; SCOPE EXPANSION → expansion/10x/dream language. Each case
  navigates 8-12 prior AUQs (telemetry, proactive, routing, vendoring,
  brain, office-hours, premise, approach) before hitting Step 0F.
- Periodic, not gate: navigation phase too slow for PR-blocking.
  V2 expansion to 4 modes (SELECTIVE + REDUCTION) when nav is faster.

skill-e2e-ship-idempotency.test.ts (~$3/run, 5-10 min):
- Builds a real git fixture with VERSION 0.0.2 already bumped, matching
  package.json, CHANGELOG entry, pushed to a local bare remote. Runs
  /ship in plan mode and asserts STATE: ALREADY_BUMPED echoes from the
  Step 12 idempotency check, OR plan_ready terminates without mutation.
- Snapshots VERSION + package.json + CHANGELOG entry count + commit
  count + branch HEAD before/after; fails if any changed.

skill-e2e-autoplan-chain.test.ts (~$8/run, 12-18 min):
- Asserts /autoplan phases run sequentially: tees timestamps as each
  "**Phase N complete.**" marker first appears. Phase 1 (CEO) must
  precede Phase 3 (Eng); Phase 2 (Design) is optional but if it
  appears, must sit between 1 and 3.
- Auto-grants permission dialogs that fire during phase transitions.

All three auto-handle permission dialogs (preamble side-effects on
fresh user envs without .feature-prompted-* markers).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: spell out AskUserQuestion everywhere instead of AUQ

Per user feedback: don't shorten AskUserQuestion to AUQ — the
abbreviation reads as cryptic. Apply across all the new code from this
branch:

- Rename test/skill-e2e-auq-format-compliance.test.ts →
  test/skill-e2e-ask-user-question-format-compliance.test.ts
- Touchfile entry auq-format-pty → ask-user-question-format-pty
  (touchfiles.ts + matching assertion in touchfiles.test.ts)
- Function rename navigateToModeAuq → navigateToModeAskUserQuestion
- Variable auqVisible → askUserQuestionVisible
- Outcome literal 'real_auq' → 'real_question'
- All comments + JSDoc + CHANGELOG entry write AskUserQuestion in full
- "AUQs" plural → "AskUserQuestions"

No behavior change. 49/49 free tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: harden v1.15.0.0 CHANGELOG entry against hostile readers

Per Garry: write the entry assuming a critic will screencap one line
and try to use it as ammunition.

Reframed the v1.15.0.0 release-summary to lead with new capability
(real-PTY harness, 11 plan-mode tests, +6 new) instead of fix-of-prior-
flaw narrative. Removed phrases that critics could weaponize:

- "0/5 → 5/5 passing", "finally pass", "∞ (never green)" — drop
- "Skill prompts get a 25% haircut" — implied self-inflicted bloat
- "770K → 574K tokens" — absolute number lets critics quote "still 574K
  of bloat"; replaced with relative "−196K tokens per invocation"
- "5 plan-mode E2E tests turned out to have never actually passed" —
  literal admission of long-term breakage; cut entirely
- Itemized "Fixed: tests finally pass" entry — moved to Changed with
  neutral "rewritten on the new harness" framing
- "Removed: harness with the runPlanModeSkillTest API that never
  worked" — replaced with "superseded by claude-pty-runner.ts"

Added concrete code receipts to pre-empt "it's just markdown":

- Net branch size: −11,609 lines (89 files, +7,240 / −18,849)
- 654 lines of TypeScript in test/helpers/claude-pty-runner.ts
- 8 new test files, ~1,453 lines of new TS code
- 23 helper unit tests + 6 new gate/periodic E2E tests

The deletion-heavy net diff (−11.6K lines) is itself the strongest
defense against the "bloat" critique — surfaced explicitly in the
numbers table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 13:55:13 -07:00

623 lines
34 KiB
TypeScript
Raw Blame History

/**
* Diff-based test selection for E2E and LLM-judge evals.
*
* Each test declares which source files it depends on ("touchfiles").
* The test runner checks `git diff` and only runs tests whose
* dependencies were modified. Override with EVALS_ALL=1 to run everything.
*/
import { spawnSync } from 'child_process';
// --- Glob matching ---
/**
* Match a file path against a glob pattern.
* Supports:
* ** — match any number of path segments
* * — match within a single segment (no /)
*/
export function matchGlob(file: string, pattern: string): boolean {
const regexStr = pattern
.replace(/\./g, '\\.')
.replace(/\*\*/g, '{{GLOBSTAR}}')
.replace(/\*/g, '[^/]*')
.replace(/\{\{GLOBSTAR\}\}/g, '.*');
return new RegExp(`^${regexStr}$`).test(file);
}
// --- Touchfile maps ---
/**
* E2E test touchfiles — keyed by testName (the string passed to runSkillTest).
* Each test lists the file patterns that, if changed, require the test to run.
*/
export const E2E_TOUCHFILES: Record<string, string[]> = {
// Browse core (+ test-server dependency)
'browse-basic': ['browse/src/**', 'browse/test/test-server.ts'],
'browse-snapshot': ['browse/src/**', 'browse/test/test-server.ts'],
// SKILL.md setup + preamble (depend on ROOT SKILL.md + gen-skill-docs)
'skillmd-setup-discovery': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'skillmd-no-local-binary': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'skillmd-outside-git': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'session-awareness': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'operational-learning': ['scripts/resolvers/preamble.ts', 'bin/gstack-learnings-log'],
// QA (+ test-server dependency)
'qa-quick': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts'],
'qa-b6-static': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval.html', 'test/fixtures/qa-eval-ground-truth.json'],
'qa-b7-spa': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval-spa.html', 'test/fixtures/qa-eval-spa-ground-truth.json'],
'qa-b8-checkout': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts', 'test/helpers/llm-judge.ts', 'browse/test/fixtures/qa-eval-checkout.html', 'test/fixtures/qa-eval-checkout-ground-truth.json'],
'qa-only-no-fix': ['qa-only/**', 'qa/templates/**'],
'qa-fix-loop': ['qa/**', 'browse/src/**', 'browse/test/test-server.ts'],
'qa-bootstrap': ['qa/**', 'ship/**'],
// Review
'review-sql-injection': ['review/**', 'test/fixtures/review-eval-vuln.rb'],
'review-enum-completeness': ['review/**', 'test/fixtures/review-eval-enum*.rb'],
'review-base-branch': ['review/**'],
'review-design-lite': ['review/**', 'test/fixtures/review-eval-design-slop.*'],
// Review Army (specialist dispatch)
'review-army-migration-safety': ['review/**', 'scripts/resolvers/review-army.ts', 'bin/gstack-diff-scope'],
'review-army-perf-n-plus-one': ['review/**', 'scripts/resolvers/review-army.ts', 'bin/gstack-diff-scope'],
'review-army-delivery-audit': ['review/**', 'scripts/resolvers/review.ts', 'scripts/resolvers/review-army.ts'],
'review-army-quality-score': ['review/**', 'scripts/resolvers/review-army.ts'],
'review-army-json-findings': ['review/**', 'scripts/resolvers/review-army.ts'],
'review-army-red-team': ['review/**', 'scripts/resolvers/review-army.ts'],
'review-army-consensus': ['review/**', 'scripts/resolvers/review-army.ts'],
// Office Hours
'office-hours-spec-review': ['office-hours/**', 'scripts/gen-skill-docs.ts'],
'office-hours-forcing-energy': ['office-hours/**', 'scripts/resolvers/preamble.ts', 'test/fixtures/mode-posture/**', 'test/helpers/llm-judge.ts'],
'office-hours-builder-wildness': ['office-hours/**', 'scripts/resolvers/preamble.ts', 'test/fixtures/mode-posture/**', 'test/helpers/llm-judge.ts'],
// Plan reviews
'plan-ceo-review': ['plan-ceo-review/**'],
'plan-ceo-review-selective': ['plan-ceo-review/**'],
'plan-ceo-review-benefits': ['plan-ceo-review/**', 'scripts/gen-skill-docs.ts'],
'plan-ceo-review-expansion-energy': ['plan-ceo-review/**', 'scripts/resolvers/preamble.ts', 'test/fixtures/mode-posture/**', 'test/helpers/llm-judge.ts'],
'plan-eng-review': ['plan-eng-review/**'],
'plan-eng-review-artifact': ['plan-eng-review/**'],
'plan-review-report': ['plan-eng-review/**', 'scripts/gen-skill-docs.ts'],
// Plan-mode smoke tests — gate-tier safety regression tests. Each fires when
// any of: the interactive skill's template, the plan-mode resolver
// (completion-status owns generatePlanModeInfo), preamble composition, or
// the real-PTY runner (which the tests now use instead of the SDK harness)
// change.
'plan-ceo-review-plan-mode': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/claude-pty-runner.ts'],
'plan-eng-review-plan-mode': ['plan-eng-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/claude-pty-runner.ts'],
'plan-design-review-plan-mode': ['plan-design-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/claude-pty-runner.ts'],
'plan-devex-review-plan-mode': ['plan-devex-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/claude-pty-runner.ts'],
'plan-mode-no-op': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/claude-pty-runner.ts'],
// Real-PTY E2E batch (#6 new tests on the harness).
// Each one tests behavior the SDK harness can't observe (rendered TTY,
// numbered-option lists, multi-phase ordering, idempotency state echo).
'ask-user-question-format-pty': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/claude-pty-runner.ts'],
'plan-ceo-mode-routing': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/claude-pty-runner.ts'],
'plan-design-with-ui-scope': ['plan-design-review/**', 'test/fixtures/plans/ui-heavy-feature.md', 'test/helpers/claude-pty-runner.ts'],
'budget-regression-pty': ['test/helpers/eval-store.ts', 'test/skill-budget-regression.test.ts'],
'ship-idempotency-pty': ['ship/**', 'bin/gstack-next-version', 'lib/worktree.ts', 'test/helpers/claude-pty-runner.ts'],
'autoplan-chain-pty': ['autoplan/**', 'plan-ceo-review/**', 'plan-design-review/**', 'plan-eng-review/**', 'plan-devex-review/**', 'test/fixtures/plans/ui-heavy-feature.md', 'test/helpers/claude-pty-runner.ts'],
'e2e-harness-audit': ['plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/claude-pty-runner.ts'],
'brain-privacy-gate': ['scripts/resolvers/preamble/generate-brain-sync-block.ts', 'scripts/resolvers/preamble.ts', 'bin/gstack-brain-sync', 'bin/gstack-brain-init', 'bin/gstack-config', 'test/helpers/agent-sdk-runner.ts'],
// AskUserQuestion format regression (RECOMMENDATION + Completeness: N/10)
// Fires when either template OR the two preamble resolvers change.
'plan-ceo-review-format-mode': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-ceo-review-format-approach': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-eng-review-format-coverage': ['plan-eng-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-eng-review-format-kind': ['plan-eng-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
// v1.7.0.0 Pros/Cons format cadence + format + negative-escape evals.
// Dependencies: same as format-mode + the 4 plan-review templates + overlay.
// All periodic-tier (non-deterministic Opus 4.7 behavior).
'plan-ceo-review-prosons-cadence': ['plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-review-prosons-format': ['plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-review-prosons-hardstop-neg': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'plan-review-prosons-neutral-neg': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
// Expanded coverage (CT3) — 6 non-plan-review skills inherit Pros/Cons via preamble
'ship-prosons-format': ['ship/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'office-hours-prosons-format': ['office-hours/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'investigate-prosons-format': ['investigate/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'qa-prosons-format': ['qa/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'review-prosons-format': ['review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'design-review-prosons-format': ['design-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
'document-release-prosons-format': ['document-release/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
// /plan-tune (v1 observational)
'plan-tune-inspect': ['plan-tune/**', 'scripts/question-registry.ts', 'scripts/psychographic-signals.ts', 'scripts/one-way-doors.ts', 'bin/gstack-question-log', 'bin/gstack-question-preference', 'bin/gstack-developer-profile'],
// Codex offering verification
'codex-offered-office-hours': ['office-hours/**', 'scripts/gen-skill-docs.ts'],
'codex-offered-ceo-review': ['plan-ceo-review/**', 'scripts/gen-skill-docs.ts'],
'codex-offered-design-review': ['plan-design-review/**', 'scripts/gen-skill-docs.ts'],
'codex-offered-eng-review': ['plan-eng-review/**', 'scripts/gen-skill-docs.ts'],
// Ship
'ship-base-branch': ['ship/**', 'bin/gstack-repo-mode'],
'ship-local-workflow': ['ship/**', 'scripts/gen-skill-docs.ts'],
'review-dashboard-via': ['ship/**', 'scripts/resolvers/review.ts', 'codex/**', 'autoplan/**', 'land-and-deploy/**'],
'ship-plan-completion': ['ship/**', 'scripts/gen-skill-docs.ts'],
'ship-plan-verification': ['ship/**', 'scripts/gen-skill-docs.ts'],
// Retro
'retro': ['retro/**'],
'retro-base-branch': ['retro/**'],
// Global discover
'global-discover': ['bin/gstack-global-discover.ts', 'test/global-discover.test.ts'],
// CSO
'cso-full-audit': ['cso/**'],
'cso-diff-mode': ['cso/**'],
'cso-infra-scope': ['cso/**'],
// Learnings
'learnings-show': ['learn/**', 'bin/gstack-learnings-search', 'bin/gstack-learnings-log', 'scripts/resolvers/learnings.ts'],
// Session Intelligence (timeline, context recovery, /context-save + /context-restore)
'timeline-event-flow': ['bin/gstack-timeline-log', 'bin/gstack-timeline-read'],
'context-recovery-artifacts': ['scripts/resolvers/preamble.ts', 'bin/gstack-timeline-log', 'bin/gstack-slug', 'learn/**'],
'context-save-writes-file': ['context-save/**', 'bin/gstack-slug'],
'context-restore-loads-latest': ['context-restore/**', 'bin/gstack-slug'],
// Context skills E2E (live-fire, Skill-tool routing path) — see
// test/skill-e2e-context-skills.test.ts. These are periodic-tier because
// each one spawns claude -p and costs ~$0.20-$0.40. Collectively they
// verify the thing the /checkpoint → /context-save rename was for.
'context-save-routing': ['context-save/**', 'scripts/resolvers/preamble.ts'],
'context-save-then-restore-roundtrip': ['context-save/**', 'context-restore/**', 'bin/gstack-slug'],
'context-restore-fragment-match': ['context-restore/**'],
'context-restore-empty-state': ['context-restore/**'],
'context-restore-list-delegates': ['context-restore/**'],
'context-restore-legacy-compat': ['context-restore/**'],
'context-save-list-current-branch': ['context-save/**'],
'context-save-list-all-branches': ['context-save/**'],
// Document-release
'document-release': ['document-release/**'],
// Codex (Claude E2E — tests /codex skill via Claude)
'codex-review': ['codex/**'],
// Codex E2E (tests skills via Codex CLI + worktree)
'codex-discover-skill': ['codex/**', '.agents/skills/**', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts'],
'codex-review-findings': ['review/**', '.agents/skills/gstack-review/**', 'codex/**', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts'],
// Gemini E2E — smoke test only (Gemini gets lost in worktrees on complex tasks)
'gemini-smoke': ['.agents/skills/**', 'test/helpers/gemini-session-runner.ts', 'lib/worktree.ts'],
// Coverage audit (shared fixture) + triage + gates
'ship-coverage-audit': ['ship/**', 'test/fixtures/coverage-audit-fixture.ts', 'bin/gstack-repo-mode'],
'review-coverage-audit': ['review/**', 'test/fixtures/coverage-audit-fixture.ts'],
'plan-eng-coverage-audit': ['plan-eng-review/**', 'test/fixtures/coverage-audit-fixture.ts'],
'ship-triage': ['ship/**', 'bin/gstack-repo-mode'],
// Plan completion audit + verification
'ship-plan-completion': ['ship/**', 'scripts/gen-skill-docs.ts'],
'ship-plan-verification': ['ship/**', 'qa-only/**', 'scripts/gen-skill-docs.ts'],
'ship-idempotency': ['ship/**', 'scripts/resolvers/utility.ts'],
'review-plan-completion': ['review/**', 'scripts/gen-skill-docs.ts'],
// Design
'design-consultation-core': ['design-consultation/**', 'scripts/gen-skill-docs.ts', 'test/helpers/llm-judge.ts'],
'design-consultation-existing': ['design-consultation/**', 'scripts/gen-skill-docs.ts'],
'design-consultation-research': ['design-consultation/**', 'scripts/gen-skill-docs.ts'],
'design-consultation-preview': ['design-consultation/**', 'scripts/gen-skill-docs.ts'],
'plan-design-review-plan-mode': ['plan-design-review/**', 'scripts/gen-skill-docs.ts'],
'plan-design-review-no-ui-scope': ['plan-design-review/**', 'scripts/gen-skill-docs.ts'],
'design-review-fix': ['design-review/**', 'browse/src/**', 'scripts/gen-skill-docs.ts'],
// Design Shotgun
'design-shotgun-path': ['design-shotgun/**', 'design/src/**', 'scripts/resolvers/design.ts'],
'design-shotgun-session': ['design-shotgun/**', 'scripts/resolvers/design.ts'],
'design-shotgun-full': ['design-shotgun/**', 'design/src/**', 'browse/src/**'],
// gstack-upgrade
'gstack-upgrade-happy-path': ['gstack-upgrade/**'],
// Deploy skills
'land-and-deploy-workflow': ['land-and-deploy/**', 'scripts/gen-skill-docs.ts'],
'land-and-deploy-first-run': ['land-and-deploy/**', 'scripts/gen-skill-docs.ts', 'bin/gstack-slug'],
'land-and-deploy-review-gate': ['land-and-deploy/**', 'bin/gstack-review-read'],
'canary-workflow': ['canary/**', 'browse/src/**'],
'benchmark-workflow': ['benchmark/**', 'browse/src/**'],
'setup-deploy-workflow': ['setup-deploy/**', 'scripts/gen-skill-docs.ts'],
// Sidebar agent
'sidebar-navigate': ['browse/src/server.ts', 'browse/src/sidebar-agent.ts', 'browse/src/sidebar-utils.ts', 'extension/**'],
'sidebar-url-accuracy': ['browse/src/server.ts', 'browse/src/sidebar-agent.ts', 'browse/src/sidebar-utils.ts', 'extension/background.js'],
'sidebar-css-interaction': ['browse/src/server.ts', 'browse/src/sidebar-agent.ts', 'browse/src/write-commands.ts', 'browse/src/read-commands.ts', 'browse/src/cdp-inspector.ts', 'extension/**'],
// Autoplan
'autoplan-core': ['autoplan/**', 'plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**'],
'autoplan-dual-voice': ['autoplan/**', 'codex/**', 'bin/gstack-codex-probe', 'scripts/resolvers/review.ts', 'scripts/resolvers/design.ts'],
// Multi-provider benchmark adapters — live API smoke against real claude/codex/gemini CLIs
'benchmark-providers-live': ['bin/gstack-model-benchmark', 'test/helpers/providers/**', 'test/helpers/benchmark-runner.ts', 'test/helpers/pricing.ts'],
// Skill routing — journey-stage tests (depend on ALL skill descriptions)
'journey-ideation': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-plan-eng': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-debug': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-qa': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-code-review': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-ship': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-docs': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-retro': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-design-system': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'journey-visual-qa': ['*/SKILL.md.tmpl', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
// Opus 4.7 behavior evals — keys match testName: values in the test file.
// Routing sub-tests use template literal `routing-${c.name}` testNames,
// which the touchfile completeness scanner skips; they inherit selection
// from the file-level touchfile entry via GLOBAL_TOUCHFILES.
'fanout-arm-overlay-on':
['model-overlays/claude.md', 'model-overlays/opus-4-7.md', 'scripts/models.ts', 'scripts/resolvers/model-overlay.ts'],
'fanout-arm-overlay-off':
['model-overlays/claude.md', 'model-overlays/opus-4-7.md', 'scripts/models.ts', 'scripts/resolvers/model-overlay.ts'],
// Overlay efficacy harness (SDK) — measures whether overlay nudges change
// behavior under @anthropic-ai/claude-agent-sdk (closer to real Claude Code
// than `claude -p`). testNames in the file are template literals so the
// completeness scanner doesn't require them; these entries exist for
// diff-based selection accuracy.
'overlay-harness-opus-4-7-fanout-toy': [
'model-overlays/**',
'test/fixtures/overlay-nudges.ts',
'test/helpers/agent-sdk-runner.ts',
'scripts/resolvers/model-overlay.ts',
],
'overlay-harness-opus-4-7-fanout-realistic': [
'model-overlays/**',
'test/fixtures/overlay-nudges.ts',
'test/helpers/agent-sdk-runner.ts',
'scripts/resolvers/model-overlay.ts',
],
};
/**
* E2E test tiers — 'gate' blocks PRs, 'periodic' runs weekly/on-demand.
* Must have exactly the same keys as E2E_TOUCHFILES.
*/
export const E2E_TIERS: Record<string, 'gate' | 'periodic'> = {
// Browse core — gate (if browse breaks, everything breaks)
'browse-basic': 'gate',
'browse-snapshot': 'gate',
// SKILL.md setup — gate (if setup breaks, no skill works)
'skillmd-setup-discovery': 'gate',
'skillmd-no-local-binary': 'gate',
'skillmd-outside-git': 'gate',
'session-awareness': 'gate',
'operational-learning': 'gate',
// QA — gate for functional, periodic for quality/benchmarks
'qa-quick': 'gate',
'qa-b6-static': 'periodic',
'qa-b7-spa': 'periodic',
'qa-b8-checkout': 'periodic',
'qa-only-no-fix': 'gate', // CRITICAL guardrail: Edit tool forbidden
'qa-fix-loop': 'periodic',
'qa-bootstrap': 'gate',
// Review — gate for functional/guardrails, periodic for quality
'review-sql-injection': 'gate', // Security guardrail
'review-enum-completeness': 'gate',
'review-base-branch': 'gate',
'review-design-lite': 'periodic', // 4/7 threshold is subjective
'review-coverage-audit': 'gate',
'review-plan-completion': 'gate',
'review-dashboard-via': 'gate',
// Review Army — gate for core functionality, periodic for multi-specialist
'review-army-migration-safety': 'gate', // Specialist activation guardrail
'review-army-perf-n-plus-one': 'gate', // Specialist activation guardrail
'review-army-delivery-audit': 'gate', // Delivery integrity guardrail
'review-army-quality-score': 'gate', // Score computation
'review-army-json-findings': 'gate', // JSON schema compliance
'review-army-red-team': 'periodic', // Multi-agent coordination
'review-army-consensus': 'periodic', // Multi-specialist agreement
// Office Hours
'office-hours-spec-review': 'gate',
'office-hours-forcing-energy': 'gate', // V1.1 mode-posture regression gate (Sonnet generator)
'office-hours-builder-wildness': 'gate', // V1.1 mode-posture regression gate (Sonnet generator)
// Plan reviews — gate for cheap functional, periodic for Opus quality
'plan-ceo-review': 'periodic',
'plan-ceo-review-selective': 'periodic',
'plan-ceo-review-benefits': 'gate',
'plan-ceo-review-expansion-energy': 'gate', // V1.1 mode-posture regression gate (Opus generator, Sonnet judge)
'plan-eng-review': 'periodic',
'plan-eng-review-artifact': 'periodic',
'plan-eng-coverage-audit': 'gate',
'plan-review-report': 'gate',
// Plan-mode handshake — deterministic safety regression, gate-tier
'plan-ceo-review-plan-mode': 'gate',
'plan-eng-review-plan-mode': 'gate',
'plan-design-review-plan-mode': 'gate',
'plan-devex-review-plan-mode': 'gate',
'plan-mode-no-op': 'gate',
'e2e-harness-audit': 'gate',
// Real-PTY E2E batch — tier classification:
// gate: cheap, deterministic, run on every PR
// periodic: long-running or expensive (>$3/run), run weekly
'ask-user-question-format-pty': 'gate', // ~$0.50/run, single skill probe
'plan-ceo-mode-routing': 'periodic', // ~$3/run, deep navigation through 8-12 prior AskUserQuestions
'plan-design-with-ui-scope': 'gate', // ~$0.80/run
'budget-regression-pty': 'gate', // free, library-only assertion
'ship-idempotency-pty': 'periodic', // ~$3/run, real /ship in plan mode
'autoplan-chain-pty': 'periodic', // ~$8/run, all 3 phases sequential
// Privacy gate for gstack-brain-sync — periodic (non-deterministic LLM call,
// costs ~$0.30-$0.50 per run, not needed on every commit)
'brain-privacy-gate': 'periodic',
// AskUserQuestion format regression — periodic (Opus 4.7 non-deterministic benchmark)
'plan-ceo-review-format-mode': 'periodic',
'plan-ceo-review-format-approach': 'periodic',
'plan-eng-review-format-coverage': 'periodic',
'plan-eng-review-format-kind': 'periodic',
// v1.7.0.0 Pros/Cons format — cadence + negative-escape evals (all periodic)
'plan-ceo-review-prosons-cadence': 'periodic',
'plan-review-prosons-format': 'periodic',
'plan-review-prosons-hardstop-neg': 'periodic',
'plan-review-prosons-neutral-neg': 'periodic',
// CT3 expanded coverage — non-plan-review skills inheriting Pros/Cons (all periodic)
'ship-prosons-format': 'periodic',
'office-hours-prosons-format': 'periodic',
'investigate-prosons-format': 'periodic',
'qa-prosons-format': 'periodic',
'review-prosons-format': 'periodic',
'design-review-prosons-format': 'periodic',
'document-release-prosons-format': 'periodic',
// /plan-tune — gate (core v1 DX promise: plain-English intent routing)
'plan-tune-inspect': 'gate',
// Codex offering verification
'codex-offered-office-hours': 'gate',
'codex-offered-ceo-review': 'gate',
'codex-offered-design-review': 'gate',
'codex-offered-eng-review': 'gate',
// Session Intelligence — gate for data flow, periodic for agent integration
'timeline-event-flow': 'gate', // Binary data flow (no LLM needed)
'context-recovery-artifacts': 'gate', // Preamble reads seeded artifacts
'context-save-writes-file': 'gate', // /context-save writes a file
'context-restore-loads-latest': 'gate', // Cross-branch newest-by-filename restore
// Context skills live-fire — periodic (each test spawns claude -p, ~$0.20-$0.40)
'context-save-routing': 'periodic', // Proves /context-save routes via Skill tool
'context-save-then-restore-roundtrip': 'periodic', // Full cycle in one session
'context-restore-fragment-match': 'periodic', // /context-restore <fragment>
'context-restore-empty-state': 'periodic', // Graceful zero-saves message
'context-restore-list-delegates': 'periodic', // /context-restore list redirect
'context-restore-legacy-compat': 'periodic', // Pre-rename files still load
'context-save-list-current-branch': 'periodic', // Default branch filter
'context-save-list-all-branches': 'periodic', // --all flag
// Ship — gate (end-to-end ship path)
'ship-base-branch': 'gate',
'ship-local-workflow': 'gate',
'ship-coverage-audit': 'gate',
'ship-triage': 'gate',
'ship-plan-completion': 'gate',
'ship-plan-verification': 'gate',
'ship-idempotency': 'periodic',
// Retro — gate for cheap branch detection, periodic for full Opus retro
'retro': 'periodic',
'retro-base-branch': 'gate',
// Global discover
'global-discover': 'gate',
// CSO — gate for security guardrails, periodic for quality
'cso-full-audit': 'gate', // Hardcoded secrets detection
'cso-diff-mode': 'gate',
'cso-infra-scope': 'periodic',
// Learnings — gate (functional guardrail: seeded learnings must appear)
'learnings-show': 'gate',
// Document-release — gate (CHANGELOG guardrail)
'document-release': 'gate',
// Codex — periodic (Opus, requires codex CLI)
'codex-review': 'periodic',
// Multi-AI — periodic (require external CLIs)
'codex-discover-skill': 'periodic',
'codex-review-findings': 'periodic',
'gemini-smoke': 'periodic',
// Design — gate for cheap functional, periodic for Opus/quality
'design-consultation-core': 'periodic',
'design-consultation-existing': 'periodic',
'design-consultation-research': 'gate',
'design-consultation-preview': 'gate',
'plan-design-review-plan-mode': 'periodic',
'plan-design-review-no-ui-scope': 'gate',
'design-review-fix': 'periodic',
'design-shotgun-path': 'gate',
'design-shotgun-session': 'gate',
'design-shotgun-full': 'periodic',
// gstack-upgrade
'gstack-upgrade-happy-path': 'gate',
// Deploy skills
'land-and-deploy-workflow': 'gate',
'land-and-deploy-first-run': 'gate',
'land-and-deploy-review-gate': 'gate',
'canary-workflow': 'gate',
'benchmark-workflow': 'gate',
'setup-deploy-workflow': 'gate',
// Sidebar agent
'sidebar-navigate': 'periodic',
'sidebar-url-accuracy': 'periodic',
'sidebar-css-interaction': 'periodic',
// Autoplan — periodic (not yet implemented)
'autoplan-core': 'periodic',
'autoplan-dual-voice': 'periodic',
// Multi-provider benchmark — periodic (requires external CLIs + auth, paid)
'benchmark-providers-live': 'periodic',
// Skill routing — periodic (LLM routing is non-deterministic)
'journey-ideation': 'periodic',
'journey-plan-eng': 'periodic',
'journey-debug': 'periodic',
'journey-qa': 'periodic',
'journey-code-review': 'periodic',
'journey-ship': 'periodic',
'journey-docs': 'periodic',
'journey-retro': 'periodic',
'journey-design-system': 'periodic',
'journey-visual-qa': 'periodic',
// Opus 4.7 overlay evals — periodic (non-deterministic LLM behavior + Opus cost)
'fanout-arm-overlay-on': 'periodic',
'fanout-arm-overlay-off': 'periodic',
// Overlay efficacy harness (SDK, paid) — periodic only
'overlay-harness-opus-4-7-fanout-toy': 'periodic',
'overlay-harness-opus-4-7-fanout-realistic': 'periodic',
};
/**
* LLM-judge test touchfiles — keyed by test description string.
*/
export const LLM_JUDGE_TOUCHFILES: Record<string, string[]> = {
'command reference table': ['SKILL.md', 'SKILL.md.tmpl', 'browse/src/commands.ts'],
'snapshot flags reference': ['SKILL.md', 'SKILL.md.tmpl', 'browse/src/snapshot.ts'],
'browse/SKILL.md reference': ['browse/SKILL.md', 'browse/SKILL.md.tmpl', 'browse/src/**'],
'setup block': ['SKILL.md', 'SKILL.md.tmpl'],
'regression vs baseline': ['SKILL.md', 'SKILL.md.tmpl', 'browse/src/commands.ts', 'test/fixtures/eval-baselines.json'],
'qa/SKILL.md workflow': ['qa/SKILL.md', 'qa/SKILL.md.tmpl'],
'qa/SKILL.md health rubric': ['qa/SKILL.md', 'qa/SKILL.md.tmpl'],
'qa/SKILL.md anti-refusal': ['qa/SKILL.md', 'qa/SKILL.md.tmpl', 'qa-only/SKILL.md', 'qa-only/SKILL.md.tmpl'],
'cross-skill greptile consistency': ['review/SKILL.md', 'review/SKILL.md.tmpl', 'ship/SKILL.md', 'ship/SKILL.md.tmpl', 'review/greptile-triage.md', 'retro/SKILL.md', 'retro/SKILL.md.tmpl'],
'baseline score pinning': ['SKILL.md', 'SKILL.md.tmpl', 'test/fixtures/eval-baselines.json'],
// Ship & Release
'ship/SKILL.md workflow': ['ship/SKILL.md', 'ship/SKILL.md.tmpl'],
'document-release/SKILL.md workflow': ['document-release/SKILL.md', 'document-release/SKILL.md.tmpl'],
// Plan Reviews
'plan-ceo-review/SKILL.md modes': ['plan-ceo-review/SKILL.md', 'plan-ceo-review/SKILL.md.tmpl'],
'plan-eng-review/SKILL.md sections': ['plan-eng-review/SKILL.md', 'plan-eng-review/SKILL.md.tmpl'],
'plan-design-review/SKILL.md passes': ['plan-design-review/SKILL.md', 'plan-design-review/SKILL.md.tmpl'],
// Design skills
'design-review/SKILL.md fix loop': ['design-review/SKILL.md', 'design-review/SKILL.md.tmpl'],
'design-consultation/SKILL.md research': ['design-consultation/SKILL.md', 'design-consultation/SKILL.md.tmpl'],
// Office Hours
'office-hours/SKILL.md spec review': ['office-hours/SKILL.md', 'office-hours/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
'office-hours/SKILL.md design sketch': ['office-hours/SKILL.md', 'office-hours/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
// Deploy skills
'land-and-deploy/SKILL.md workflow': ['land-and-deploy/SKILL.md', 'land-and-deploy/SKILL.md.tmpl'],
'canary/SKILL.md monitoring loop': ['canary/SKILL.md', 'canary/SKILL.md.tmpl'],
'benchmark/SKILL.md perf collection': ['benchmark/SKILL.md', 'benchmark/SKILL.md.tmpl'],
'setup-deploy/SKILL.md platform setup': ['setup-deploy/SKILL.md', 'setup-deploy/SKILL.md.tmpl'],
// Other skills
'retro/SKILL.md instructions': ['retro/SKILL.md', 'retro/SKILL.md.tmpl'],
'qa-only/SKILL.md workflow': ['qa-only/SKILL.md', 'qa-only/SKILL.md.tmpl'],
'gstack-upgrade/SKILL.md upgrade flow': ['gstack-upgrade/SKILL.md', 'gstack-upgrade/SKILL.md.tmpl'],
// Voice directive
'voice directive tone': ['scripts/resolvers/preamble.ts', 'review/SKILL.md', 'review/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
};
/**
* Changes to any of these files trigger ALL tests (both E2E and LLM-judge).
*
* Keep this list minimal — only files that genuinely affect every test.
* Scoped dependencies (gen-skill-docs, llm-judge, test-server, worktree,
* codex/gemini session runners) belong in individual test entries instead.
*/
export const GLOBAL_TOUCHFILES = [
'test/helpers/session-runner.ts', // All E2E tests use this runner
'test/helpers/eval-store.ts', // All E2E tests store results here
'test/helpers/touchfiles.ts', // Self-referential — reclassifying wrong is dangerous
];
// --- Base branch detection ---
/**
* Detect the base branch by trying refs in order.
* Returns the first valid ref, or null if none found.
*/
export function detectBaseBranch(cwd: string): string | null {
for (const ref of ['origin/main', 'origin/master', 'main', 'master']) {
const result = spawnSync('git', ['rev-parse', '--verify', ref], {
cwd, stdio: 'pipe', timeout: 3000,
});
if (result.status === 0) return ref;
}
return null;
}
/**
* Get list of files changed between base branch and HEAD.
*/
export function getChangedFiles(baseBranch: string, cwd: string): string[] {
const result = spawnSync('git', ['diff', '--name-only', `${baseBranch}...HEAD`], {
cwd, stdio: 'pipe', timeout: 5000,
});
if (result.status !== 0) return [];
return result.stdout.toString().trim().split('\n').filter(Boolean);
}
// --- Test selection ---
/**
* Select tests to run based on changed files.
*
* Algorithm:
* 1. If any changed file matches a global touchfile → run ALL tests
* 2. Otherwise, for each test, check if any changed file matches its patterns
* 3. Return selected + skipped lists with reason
*/
export function selectTests(
changedFiles: string[],
touchfiles: Record<string, string[]>,
globalTouchfiles: string[] = GLOBAL_TOUCHFILES,
): { selected: string[]; skipped: string[]; reason: string } {
const allTestNames = Object.keys(touchfiles);
// Global touchfile hit → run all
for (const file of changedFiles) {
if (globalTouchfiles.some(g => matchGlob(file, g))) {
return { selected: allTestNames, skipped: [], reason: `global: ${file}` };
}
}
// Per-test matching
const selected: string[] = [];
const skipped: string[] = [];
for (const [testName, patterns] of Object.entries(touchfiles)) {
const hit = changedFiles.some(f => patterns.some(p => matchGlob(f, p)));
(hit ? selected : skipped).push(testName);
}
return { selected, skipped, reason: 'diff' };
}
Reference in New Issue View Git Blame Copy Permalink