Files
gstack/test/catalog-trim.test.ts
T
6237994016 v1.91.157.0 perf(gen-skill-docs): drop the "When to invoke" body section from Claude SKILL.md files (port of #3129) (#3305)
* perf(gen-skill-docs): stop emitting the "When to invoke" body section

The Claude catalog trim moved each skill's routing prose ("Use when...",
"Proactively invoke...", voice triggers) out of the frontmatter description
into a "## When to invoke this skill" section at the top of the body. Claude
Code reads only name + description to choose a skill and loads the body after
the skill is invoked, so that text cannot route anything and costs tokens on
every invocation. Keep the one-line trimmed description; drop the section.

The routing prose stays in each template's description, which
--catalog-mode=full and the other hosts still render. Tests that pinned the
section now assert its absence, or check the template instead.

(cherry picked from commit 707ccd6f38)

* feat(gen-skill-docs): keep voice triggers in the root routing block

The Claude catalog trim drops each skill's "Voice triggers (speech-to-text
aliases)" line from its description, and the body section that used to hold
it is gone. {{VOICE_TRIGGER_ROUTING}} renders them in the root router's
routing block instead: one "Voice trigger ... → invoke /<skill>" line per
skill, read from the templates' voice-triggers lists, so disabled-skill
filtering applies to them like any other routing line. Other hosts keep the
line in each description and render nothing here.

extractVoiceTriggers moves to the new resolver so the generator and the
resolver share one parser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 645e75b1e8)

* chore: regenerate SKILL.md files without the "When to invoke" section

Output of `bun run gen:skill-docs --host all` after the previous commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit ff4c813337)

* test: refresh context-budget ceilings for the smaller skill bodies

`bun test/helpers/capture-context-budget.ts`. Against a recapture of the base
commit, 55 skills that lost the section get a lower ceiling and the root
gstack router rises by 532 tokens for the voice-trigger routing lines (5,369
tokens lower in total). devex-review and eval-plan differ from the committed
fixture only because it already lagged main. alwaysOnTotal is unchanged: the
catalog descriptions did not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit d886fcecc1)

* test: refresh the claude ship golden without the "When to invoke" section

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f1789dca5c)

* test: lower the claude design-review size pin

The size guard (#3199) asks for it: claude/design-review shrank by 437 chars
without its "When to invoke" section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 3a446fcc62)

* test: skip alias symlinks in the voice-routing check, keep unrelated budget ceilings, add #3129 CHANGELOG entry

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: rebuild the historical cookie approval with the dropped section, record the skill-fixture change as non-behavioral

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* v1.91.157.0 perf(gen-skill-docs): drop the "When to invoke" body section from Claude SKILL.md files (port of #3129)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: thanhnguyentung95 <17350411+thanhnguyentung95@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-19 22:55:42 +00:00

304 lines
14 KiB
TypeScript

/**
* Unit tests for catalog-trim helpers (gen-skill-docs.ts T4 functions).
*
* splitCatalogDescription, buildTrimmedDescription, applyCatalogTrim — these handle every skill's frontmatter rewrite at gen
* time. Two bugs already shipped here:
*
* v1.45.0.0 design-consultation: when the first sentence exceeded 200 chars,
* the routing-prose extraction lost the entire tail. design-consultation's
* "Use when asked to..." silently disappeared from the body section.
*
* v1.45.0.0 CI freshness: the root-skill key leaked the checkout directory
* name ("seville-v3" vs "gstack") and aggregate order was filesystem-
* iteration order. Two machines produced two different JSON files.
*
* Both are regression-tested here. Future bugs in these functions surface as
* unit-test failures before they hit CI or production.
*/
import { describe, test, expect } from 'bun:test';
import {
splitCatalogDescription,
buildTrimmedDescription,
applyCatalogTrim,
toYamlInlineScalar,
} from '../scripts/gen-skill-docs';
describe('toYamlInlineScalar', () => {
const parses = (out: string) => Bun.YAML.parse(`d: ${out}`);
test("scalar containing '...' (YAML document-end marker) is quoted and round-trips", () => {
const out = toYamlInlineScalar('Truncated lead ends with... more (gstack)');
expect(out.startsWith('"')).toBe(true);
expect((parses(out) as { d: string }).d).toContain('...');
});
test("interior ': ' is quoted (nested-mapping ambiguity, #1778)", () => {
const out = toYamlInlineScalar('Ship workflow: detect and merge');
expect(out.startsWith('"')).toBe(true);
expect((parses(out) as { d: string }).d).toBe('Ship workflow: detect and merge');
});
test('plain safe scalar passes through unquoted', () => {
expect(toYamlInlineScalar('Simple description here')).toBe('Simple description here');
});
test('leading YAML indicator char is quoted', () => {
expect(toYamlInlineScalar('- leading dash').startsWith('"')).toBe(true);
});
test('trailing whitespace is quoted', () => {
expect(toYamlInlineScalar('has trailing space ').startsWith('"')).toBe(true);
});
});
describe('splitCatalogDescription', () => {
test('extracts lead sentence + routing prose from simple multi-line description', () => {
const desc =
'Pre-landing PR review. Analyzes diff against the base branch for SQL safety, LLM trust\n' +
'boundary violations, conditional side effects, and other structural issues. Use when\n' +
'asked to "review this PR", "code review", "pre-landing review", or "check my diff".\n' +
'Proactively suggest when the user is about to merge or land code changes. (gstack)';
const parts = splitCatalogDescription(desc);
expect(parts.lead).toBe('Pre-landing PR review.');
expect(parts.hasGstackTag).toBe(true);
expect(parts.voiceLine).toBeNull();
expect(parts.routingProse).toContain('Use when');
expect(parts.routingProse).toContain('Proactively suggest');
expect(parts.routingProse).toContain('Analyzes diff');
// (gstack) tag stripped from routingProse
expect(parts.routingProse).not.toContain('(gstack)');
});
test('REGRESSION (design-consultation v1.45.0.0): >200 char first sentence keeps routing', () => {
// This is the exact shape that broke. First sentence (with embedded periods)
// is 207 chars. Original bug: routing extraction ran AFTER lead truncation,
// so collapsed.indexOf(lead) returned -1 (lead ended in "...") and the
// entire "Use when..." + "Proactively..." tail dropped to empty string.
const desc =
'Design consultation: understands your product, researches the landscape, ' +
'proposes a complete design system (aesthetic, typography, color, layout, ' +
'spacing, motion), and generates font+color preview pages. ' +
'Creates DESIGN.md as your project\'s design source of truth. ' +
'For existing sites, use /plan-design-review to infer the system instead. ' +
'Use when asked to "design system", "brand guidelines", or "create DESIGN.md". ' +
'Proactively suggest when starting a new project\'s UI with no existing ' +
'design system or DESIGN.md. (gstack)';
const parts = splitCatalogDescription(desc);
// Lead may be truncated with "..." since it exceeds 200 chars
expect(parts.lead.length).toBeLessThanOrEqual(205);
// Critical: routing MUST contain the "Use when..." and "Proactively..." prose
expect(parts.routingProse).toContain('Use when asked to');
expect(parts.routingProse).toContain('design system');
expect(parts.routingProse).toContain('Proactively suggest');
expect(parts.routingProse).toContain('Creates DESIGN.md');
});
test('extracts voice-triggers line when present', () => {
const desc =
'Quick fix. Use when asked to fix the bug. ' +
'Voice triggers (speech-to-text aliases): "fix it", "patch this", "make it work". ' +
'(gstack)';
const parts = splitCatalogDescription(desc);
expect(parts.lead).toBe('Quick fix.');
expect(parts.voiceLine).toContain('Voice triggers');
expect(parts.voiceLine).toContain('"fix it"');
expect(parts.routingProse).toContain('Use when asked to fix');
// Voice line should NOT leak into routing
expect(parts.routingProse).not.toContain('speech-to-text');
});
test('handles description without (gstack) tag', () => {
const desc = 'Single sentence description. With routing prose afterward.';
const parts = splitCatalogDescription(desc);
expect(parts.lead).toBe('Single sentence description.');
expect(parts.hasGstackTag).toBe(false);
expect(parts.routingProse).toBe('With routing prose afterward.');
});
test('REGRESSION: embedded-period first sentence splits at real boundary', () => {
// The old regex `^([^.!?]*[.!?])(?:\s|$)` could not cross ANY period —
// [^.!?]* stops at the first `.` even mid-token ("DESIGN.md"), the
// boundary check then fails, and the whole match failed. The code
// silently fell back to a 20-word cut mid-phrase (observed with a
// description mentioning "TODOS.md"). The fixed regex
// `^((?:[^.!?]|[.!?](?!\s|$))*[.!?])(?:\s|$)` consumes terminators NOT
// followed by whitespace/end, so embedded periods no longer break it.
const desc =
'Skill that mentions DESIGN.md and v1.45.0.0 in the lead. ' +
'Use when asked to do something.';
const parts = splitCatalogDescription(desc);
expect(parts.lead).toBe('Skill that mentions DESIGN.md and v1.45.0.0 in the lead.');
expect(parts.routingProse).toBe('Use when asked to do something.');
});
test('REGRESSION: "TODOS.md backlog" style sentence keeps full lead + routing', () => {
const desc =
'Drive an approved plan to completion on a TODOS.md backlog. ' +
'Use when asked to "autobuilder" or "run the build loop". ' +
'Proactively suggest after a plan is approved. (gstack)';
const parts = splitCatalogDescription(desc);
expect(parts.lead).toBe('Drive an approved plan to completion on a TODOS.md backlog.');
expect(parts.routingProse).toContain('Use when asked to "autobuilder"');
expect(parts.routingProse).toContain('Proactively suggest');
expect(parts.hasGstackTag).toBe(true);
});
test('URL in first sentence does not end the lead early', () => {
const desc =
'See https://example.com/docs/v2 for the workflow. ' +
'Use when asked to consult the docs.';
const parts = splitCatalogDescription(desc);
expect(parts.lead).toBe('See https://example.com/docs/v2 for the workflow.');
expect(parts.routingProse).toBe('Use when asked to consult the docs.');
});
test('>200 char first sentence WITH embedded periods still truncates with ellipsis and keeps routing', () => {
const firstSentence =
'Regenerate the iOS debug bridge against files like Bridge.swift and TODOS.md, ' +
'walking every target listed in Project.xcodeproj while preserving v1.45.0.0 ' +
'compatibility shims, custom entitlements, and the long tail of per-target ' +
'build settings nobody remembers configuring.';
expect(firstSentence.length).toBeGreaterThan(200);
const desc = firstSentence + ' Use when asked to resync the bridge. (gstack)';
const parts = splitCatalogDescription(desc);
// Lead is the truncated first sentence (ellipsis path), not a 20-word cut.
expect(parts.lead.endsWith('...')).toBe(true);
expect(parts.lead.length).toBeLessThanOrEqual(200);
expect(parts.lead).toContain('TODOS.md');
// Routing prose survives intact.
expect(parts.routingProse).toContain('Use when asked to resync the bridge.');
expect(parts.hasGstackTag).toBe(true);
});
test('description without a period uses first ~20 words as lead', () => {
const desc = 'A long fragment with no sentence terminator drifting on and on across many words for an unusual frontmatter shape';
const parts = splitCatalogDescription(desc);
expect(parts.lead.length).toBeGreaterThan(0);
expect(parts.lead.split(/\s+/).length).toBeLessThanOrEqual(21);
});
test('idempotent: calling on already-trimmed output returns the same parts', () => {
const desc = 'Already trimmed. (gstack)';
const parts1 = splitCatalogDescription(desc);
const parts2 = splitCatalogDescription(buildTrimmedDescription(parts1));
// Re-split of a one-line trimmed result keeps lead identical, routing empty.
expect(parts2.lead).toBe(parts1.lead);
expect(parts2.hasGstackTag).toBe(true);
expect(parts2.routingProse).toBe('');
});
});
describe('buildTrimmedDescription', () => {
test('appends (gstack) when hasGstackTag is true', () => {
const out = buildTrimmedDescription({
lead: 'Some lead.',
routingProse: 'routing',
voiceLine: null,
hasGstackTag: true,
});
expect(out).toBe('Some lead. (gstack)');
});
test('omits (gstack) when hasGstackTag is false', () => {
const out = buildTrimmedDescription({
lead: 'No tag.',
routingProse: '',
voiceLine: null,
hasGstackTag: false,
});
expect(out).toBe('No tag.');
});
test('trims whitespace from lead', () => {
const out = buildTrimmedDescription({
lead: ' Lead with whitespace. ',
routingProse: '',
voiceLine: null,
hasGstackTag: true,
});
expect(out).toBe('Lead with whitespace. (gstack)');
});
});
describe('applyCatalogTrim', () => {
const minimalSkill = `---
name: example
description: |
Example skill: this is the first sentence of the description, intended to be
the lead displayed in the catalog. Use when asked to do an example task.
Proactively suggest when the user mentions examples. (gstack)
preamble-tier: 2
---
<!-- AUTO-GENERATED from SKILL.md.tmpl — do not edit directly -->
<!-- Regenerate: bun run gen:skill-docs -->
# Example body
Original body content here.
`;
test('rewrites multi-line description into one line and adds no body section', () => {
const result = applyCatalogTrim(minimalSkill, 'example');
expect(result).not.toBeNull();
const { content, parts } = result!;
// Frontmatter description is now ONE line ending with (gstack). #1778: a
// description with an interior colon ("Example skill:") is YAML-quoted, so
// the value is wrapped in double quotes — tolerate the optional quotes.
expect(content).toMatch(/^description: "?Example skill:[^\n]*\(gstack\)"?\n/m);
// The body gains no routing section: Claude Code reads a skill body only
// after invocation, so routing prose there routes nothing.
expect(content).not.toContain('## When to invoke this skill');
expect(content).not.toContain('Use when asked to do an example task.');
// Original body still present
expect(content).toContain('# Example body');
expect(content).toContain('Original body content here.');
// parts is populated for the aggregator
expect(parts.lead).toContain('Example skill');
expect(parts.hasGstackTag).toBe(true);
});
test('returns null for already-short descriptions (no-op)', () => {
const shortSkill = minimalSkill.replace(
/description: \|[\s\S]*?(?=preamble-tier:)/,
'description: Already short. (gstack)\n',
);
const result = applyCatalogTrim(shortSkill, 'example');
expect(result).toBeNull();
});
test('keeps the newline between description and next YAML field (no field collision)', () => {
// Bug shape from v1.45.0.0 first attempt: produced
// `description: ... (gstack)preamble-tier:` with no newline.
const result = applyCatalogTrim(minimalSkill, 'example');
expect(result).not.toBeNull();
expect(result!.content).not.toMatch(/\(gstack\)preamble-tier/);
expect(result!.content).not.toMatch(/\(gstack\)allowed-tools/);
// #1778: optional closing quote when the description was YAML-quoted.
expect(result!.content).toMatch(/\(gstack\)"?\n[a-z-]+:/);
});
test('returns null on content without proper frontmatter', () => {
expect(applyCatalogTrim('no frontmatter here', 'whatever')).toBeNull();
expect(applyCatalogTrim('---\nincomplete frontmatter', 'whatever')).toBeNull();
});
});
describe('proactive-suggestions.json stays retired', () => {
test('the generator no longer emits scripts/proactive-suggestions.json', () => {
// The aggregated routing registry was removed (no consumer ever read it).
// If someone re-adds the emitter, this pins the decision to delete it —
// reintroduce only with an actual consumer, and restore the determinism
// tests (sorted keys, root keyed as "gstack", no timestamp fields) that
// lived here before.
const fs = require('fs');
const path = require('path');
expect(fs.existsSync(path.join(__dirname, '..', 'scripts', 'proactive-suggestions.json'))).toBe(false);
});
});