mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-03 15:27:56 +01:00
* fix(preamble): completeness means complete coverage with minimal code (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(preamble): tier 2+ skills load the shared preamble once per session (GSTA-96) Every tier 2+ SKILL.md inlined the same ~23 KB preamble. It now renders once per host into sections/preamble.md beside each tier 2+ SKILL.md; the SKILL.md keeps a pointer that asks for the Read only when the file's id is not already in context. Only sections byte-identical to the shared render move, so every skill keeps exactly its pre-hoist paragraphs (verified across all 41 skills). Parity caps for ten skills raised to their measured unions (owner-approved, Garry 2026-10-25: "do it"); 63 size pins lowered and 26 deleted. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(paid-select): a generated sections/preamble.md takes its skill template's identity in the PR planner (GSTA-96) screen-summary failed on #3434 with 'PR profile requires full validation' for nine skills whose only paid coverage is periodic: their new sections/preamble.md had no dependency identity, while the SKILL.md beside it inherits its template's. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(section-manifest): the shared sections/preamble.md renders from the hoist resolver, not a section template (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (CHANGELOG says 3 characters shorter; model-overlays guards the coverage half) The completeness section shrank by 3 characters (5 bytes), matching the 89 pins lowered by 3. The default-copy test pinned only the minimal-code half; an expectMentions check now fails if the full-coverage or never-excuses-a-gap clause is dropped. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (model-overlays binds smaller diff to equal coverage) The default-copy check now also fails if 'on equal coverage' is dropped from the smaller-diff rule (which would tell models to prefer the smaller diff unconditionally) or if the no-impossible-state-checks clause is removed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG lists the model-overlays coverage checks The [Unreleased] Tests bullet now says the default-copy test checks both halves of the rule (full coverage of the requirement; smaller diff only at equal coverage; no impossible-state checks; never excuses a gap), matching88e3cc0and44eacaf. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: plan-ceo-review and ETHOS say completeness means coverage, not more code Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (CEO bullet bounds code by accepted scope; tests bind each completeness rule) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: cover the shared-preamble hoist (ship Step 7) Adds test/preamble-hoist.test.ts (tier 1 unchanged; tier 2+ SKILL.md plus sections/preamble.md reproduces the pre-hoist preamble; pointer position and id) and two gen-skill-docs-checks cases (one shared preamble per host, beside every pointer and named by its id; preamble artifact kind). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (ship Step 9.4) - setup: follow a symlinked sections/ dir in the carved-pointer status check - touchfiles: select preamble-hoist.ts alongside every preamble.ts entry - section-manifest: exempt sections/preamble.md only for skills whose SKILL.md carries the shared-preamble pointer - skill-size-budget: growth ceiling measures SKILL.md plus its shared preamble - host-conformance: tolerate a missing sections/ dir, as setup does - skill-validation: read bin/gstack-skill-start directly - skill-fixture: error messages and docblocks name the actual boundary (harness-version non-behavioral note updated) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: describe the size-budget growth check's startup measure (ship Step 9.4, pass 2) The growth ceiling now measures SKILL.md plus the shared preamble, the same measure as the shrink floor; the docblock said SKILL.md only. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG says simplification.md quotes the old rule and CLAUDE.md restates the new one Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG lists the hoist tests and the growth-ceiling measure (ship Step 9.4) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: cover the symlinked sections/ status check and the ETHOS house-rule quote (ship Step 7) host-conformance: --status still finds a broken bare sections/ pointer when a carved Claude skill's installed sections/ is a directory symlink (fails at the pre-repair0e5d11fd9). gen-skill-docs: the house rule review/specialists/ simplification.md quotes from ETHOS.md still appears in ETHOS.md. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: describe the shared preamble file (ship Step 14.5 docs audit) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: the shared-preamble skip needs the skill's own Read; digest and first-run intro say complete coverage, minimal code (GSTA-96) The pointer skipped the shared file when any text carrying its id was in context, so a page or file with that heading could stand in for the anti-spoofing rules (security, red team and adversarial review, ship run 2). It now requires the skill's own earlier Read. The digest's Boil the Ocean bullet and the first-run intro still said 'AI makes completeness cheap, so do the complete thing' (docs audit blocker); both now carry Garry's 'complete coverage, minimal code' (2026-10-25: '1/ yes fix this'). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: pin the shared-preamble provenance skip and the new Boil the Ocean wording (ship Step 7) preamble-hoist: on every host, tier and render option the pointer allows the skip only when the skill's own Read is in context. agents-digest and gstack-skill-start: the digest bullet and the lake intro say complete coverage, minimal code, never the retired phrasing. Each fails at9af36d010. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: describe the provenance skip where the docs still described the id-only one (ship Step 9.4) The preamble-hoist.ts docblock and ARCHITECTURE.md still said the shared preamble is re-read only when its id is absent; since2acf1a44athe pointer requires the skill's own earlier Read of the file. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG lists the provenance-pointer and Boil the Ocean wording tests (ship Step 13) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * revert(digest): keep the agents digest at main's bytes; an unstamped head cannot change a release output (GSTA-96) CI 'Check VERSION is not stale vs queue' classified #3434 as stamped because it touched agents-digest/gstack-AGENTS.md (a release_outputs path), while regenerating only the generator fails the digest freshness test. The first-run intro rewording stays. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: voice judge reads the hoisted preamble; register ETHOS.md on its reader cases Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: refresh the claude ship golden the main merge took from main (ship Step 5 preflight) The1a4dcaf07merge resolved test/fixtures/golden/claude-ship-SKILL.md to main's pre-hoist bytes. The golden is a copy of ship/SKILL.md; regenerated with bun scripts/gen-ship-goldens.ts (pregate regen: REGEN_STALE). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: pr-dependencies' unplaceable-file example uses SECURITY.md now that ETHOS.md is mapped6183ca8aeregistered ETHOS.md on its reader cases in touchfiles-data.ts, so a PR touching only ETHOS.md plans in pr mode and no longer restores the full gate. The fail-closed test and the real-unknown label check now use SECURITY.md, which is tracked and unmapped; the assertions are unchanged. CHANGELOG names the third test file. (CI free shard 7 at 7c21720ac; remote pregate shard 6.) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: host path-leak checks read the moved shared preamble too; name SHARED_PREAMBLE_FILE (ship review pass 2) The external-host path-leak assertions in gen-skill-docs.test.ts (Codex, Factory) and host-config.test.ts (Copilot, Antigravity) read SKILL.md alone, so a Claude path that moved into sections/preamble.md with the hoist went unchecked. They now read the skill's startup text through readSkillStartup. workflow-judge-input and host-conformance use the SHARED_PREAMBLE_FILE constant instead of a bare 'preamble.md' literal, and preamble.ts's header describes the hoisted composition. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG token savings name the per-skill skip and the autoplan follow-up (ship review pass 2) The pointer's skip is bound to the skill's own copy of sections/preamble.md, so "every later one saves about 5,800" holds per skill, not per session, and /autoplan's skip list and the inline /office-hours loader do not yet skip the pointer section. The entry now says so instead of claiming the full /autoplan saving. Adding the skip entry grows size-pinned skills, so it is a follow-up. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: corpus budget measures startup bytes; carve-safety counts shared preamble as uncarved (ship review pass 2) skill-size-budget's growth ceiling and shrink floor already measured SKILL.md plus sections/preamble.md, but the total-corpus check still summed hoisted SKILL.md bytes against the pre-hoist baseline, hiding about 0.9 MB of headroom. It now sums the same startup bytes (3,114,885 -> 2,590,484, x0.832); the unreachable SKILL.md-missing fallback is gone. auq-format-always-loaded's carve-safety guard treated any sections/ dir as carved, so every skill with only the shared preamble ran it and its sanity check passed on preamble.md. It now uses setup --status's rule: a section other than the shared preamble. Those skills keep the per-skill checks. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: remaining SKILL.md-only negatives read the shared preamble too (ship review pass 3) The parameterized external-host path-leak smoke test, the two Codex .claude path checks and both vestigial-handshake negatives read SKILL.md alone, so text that moved into sections/preamble.md went unchecked. They now read readSkillStartup. preamble.ts's header no longer says tier 1 renders through generatePreamble(). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: ratchet skeleton caps and eager context ceilings down after the hoist (ship review pass 3) The hoist moved about 22 KB out of every tier 2+ SKILL.md, which left each hoisted maxSkeletonBytes cap and each eagerPerInvocation ceiling roughly 22 KB / 5-6K tokens slack. The 20 hoisted skeleton caps now sit at the post-hoist skeleton plus their pre-hoist headroom, and 42 context-budget ceilings take the recaptured (lower) values. Nothing was raised: the four ceilings the recapture would have raised keep their old values. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: preamble-hoist's section splitter closes a fence only on a bare delimiter (ship Step 11) Inside an open fence, a line such as "```bash" is content in CommonMark, but splitSections treated it as the closing fence, so a later "## " line in the same block could split a section mid-code-block and hoist half of it. A closing delimiter now needs nothing but whitespace after it. Every host's render is unchanged (gen-skill-docs --dry-run FRESH on all twelve hosts). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: skill-startup.ts is a global, pinned harness file like skill-fixture.ts (ship review pass 3) skill-fixture.ts is a global touchfile pinned in scripts/harness-version.json because it reshapes the skill content most E2E suites read, and it now builds that content through test/helpers/skill-startup.ts. That helper was in neither list, so a later edit to it would change fixture inputs without the harness-version guard and select only 27 keys. It is now global and pinned, recorded non-behavioral (its fixtures are byte-identical to the pre-hoist output). ARCHITECTURE.md and the CHANGELOG stop calling the pointer id a content id: it hashes the shared text before host rewrites. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG matches the merged skip lists and the baseline diff (ship review, run 4) The headline still called the autoplan and inline /office-hours skip entries a follow-up, but the main merge added them; it now says the loaded skills skip their shared Read. The #3451 sentence is dropped, because that fix shipped in v1.91.237.0 and is recorded there. The typecheck baseline line says the TS2769 entry was re-keyed for the 'preamble' artifact kind with its count unchanged, not fixed and lowered. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: pin the pointer entry in the loader skip lists; drop a carve-safety tautology (ship review, run 4) A new INVOKE_SKILL test checks that plan-ceo-review's inline /office-hours skip list and autoplan's section skip list both name the shared-preamble pointer heading, and that office-hours and the four plan reviews carry it. The carve-safety "sanity" assertion repeated the predicate discoverSkills() now uses to set sectionsDir, so it could never fail; it is removed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: gstack-doctor finds the Codex model patch in the hoisted shared preamble (ship Step 11, run 4) The doctor's patch row grepped only the Codex review SKILL.md for the "Model-Specific Behavioral Patch (" heading. The hoist moves that section into sections/preamble.md, so the required row (for ship, review, the plan reviews, office-hours, investigate and more) would fail on every correct post-hoist Codex install and block install.sh's "gstack: ok". It now checks the SKILL.md and then its shared preamble and names the file it found. A new doctor test covers a hoisted render. preamble-hoist's variantSubsections also returns null when no subsection differs, instead of emitting a replacing heading with no subsections. Every host's render is unchanged. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(docsync): fixtureDocs installs the generated sections/preamble.md (hunk taken from #3472) A generated skill that loads the shared preamble names sections/preamble.md as a required asset before skill start, and a spawned document-release child blocks on its absence, so docsync-spawned stayed red on every #3434 screen. fixtureDocs now copies that file when the generated skill has it. This single hunk is taken unchanged from gstack#3472 (head2e3f7ba1, GSTA-107), which trains ahead of #3434, so the train merge sees identical lines. Nothing else from #3472 is included. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG notes the docsync fixture hunk taken from #3472 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG names the post-merge context-budget ratchet (captain hoisted) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(auq): planted skill fixtures resolve the shared preamble like an install (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(plan-format): mode menu delivers its question without output-harvesting wording (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): record the v1.91.270.0 merge values (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(auq): parallel-capture contract expects the planted skill with its sections path mapped (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): record the v1.91.273.0 merge re-measurements (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): restore the released entries the previous commit truncated (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): end two sentences the prose gate flags (GSTA-96) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
512 lines
26 KiB
TypeScript
512 lines
26 KiB
TypeScript
/**
|
|
* Unit + pin tests for test/helpers/skill-fixture.ts (free tier, no EVALS).
|
|
*
|
|
* Two layers:
|
|
* 1. Semantics against a synthetic SKILL.md: frontmatter always included,
|
|
* sections concatenated in caller order, missing section throws with the
|
|
* section name, fenced `## ` template headings do not split sections,
|
|
* body extraction drops exactly the shared preamble block, head
|
|
* extraction truncates the body.
|
|
* 2. Pins against the REAL generated SKILL.md files: every exported section
|
|
* list extracts cleanly from the skill it targets, and the body/head
|
|
* helpers work for every skill the E2E fixtures feed through them. A
|
|
* heading rename in gen-skill-docs fails HERE (free, <1s) instead of
|
|
* mid-flight in a paid E2E run.
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import * as fs from 'fs';
|
|
import * as path from 'path';
|
|
import * as os from 'os';
|
|
import {
|
|
extractSkillSections,
|
|
extractSkillBody,
|
|
extractSkillHead,
|
|
REVIEW_E2E_SECTIONS,
|
|
REVIEW_ARMY_E2E_SECTIONS,
|
|
RETRO_E2E_SECTIONS,
|
|
CODEX_REVIEW_E2E_SECTIONS,
|
|
} from './helpers/skill-fixture';
|
|
import { E2E_TOUCHFILES, GLOBAL_TOUCHFILES, selectTests } from './helpers/touchfiles';
|
|
import { readShippedSkillRouting } from './helpers/shipped-skill-routing';
|
|
import { hasNarrationLeak } from './helpers/skill-body-narration';
|
|
import { expectMentions } from './helpers/prompt-structure';
|
|
import { SHARED_POINTER_HEADING, readSkillStartup } from './helpers/skill-startup';
|
|
|
|
const ROOT = path.resolve(import.meta.dir, '..');
|
|
|
|
// ─── Synthetic fixture ──────────────────────────────────────────────────────
|
|
|
|
const SYNTHETIC_SKILL = `---
|
|
name: fixture-test
|
|
description: synthetic skill for skill-fixture unit tests
|
|
---
|
|
Intro line before any section.
|
|
|
|
## When to invoke this skill
|
|
Invoke text.
|
|
|
|
## Preamble (run first)
|
|
preamble junk that fixtures must drop
|
|
|
|
## AskUserQuestion Format
|
|
more shared-preamble junk
|
|
|
|
## Plan Status Footer
|
|
footer junk, last shared-preamble section
|
|
|
|
## Step 1 — Do the thing
|
|
step one body
|
|
\`\`\`markdown
|
|
## Embedded Template Heading
|
|
template content inside a fence
|
|
\`\`\`
|
|
step one continues after the fence
|
|
|
|
## Step 2 — Other
|
|
step two body
|
|
`;
|
|
|
|
let tmpDir: string;
|
|
let skillDir: string;
|
|
|
|
beforeAll(() => {
|
|
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-fixture-test-'));
|
|
skillDir = path.join(tmpDir, 'fixture-test');
|
|
fs.mkdirSync(skillDir, { recursive: true });
|
|
fs.writeFileSync(path.join(skillDir, 'SKILL.md'), SYNTHETIC_SKILL);
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
describe('extractSkillSections (synthetic)', () => {
|
|
test('always includes frontmatter and concatenates sections in caller order', () => {
|
|
const out = extractSkillSections(skillDir, ['Step 2 — Other', 'Step 1 — Do the thing']);
|
|
expect(out.startsWith('---\nname: fixture-test')).toBe(true);
|
|
expect(out).toContain('## Step 1 — Do the thing');
|
|
expect(out).toContain('## Step 2 — Other');
|
|
// Caller order preserved: Step 2 requested first, so it appears first.
|
|
expect(out.indexOf('## Step 2 — Other')).toBeLessThan(out.indexOf('## Step 1 — Do the thing'));
|
|
// Unrequested sections are dropped.
|
|
expect(out).not.toContain('preamble junk');
|
|
expect(out).not.toContain('Intro line before any section');
|
|
});
|
|
|
|
test('fenced ## headings do not terminate a section', () => {
|
|
const out = extractSkillSections(skillDir, ['Step 1 — Do the thing']);
|
|
expect(out).toContain('template content inside a fence');
|
|
expect(out).toContain('step one continues after the fence');
|
|
expect(out).not.toContain('step two body');
|
|
});
|
|
|
|
test('a carved step ends the preceding H2 without changing its content', () => {
|
|
const pointer = '> **STOP.** Before the next step, Read `~/review/sections/next.md` and execute it\n'
|
|
+ '> in full. Do not work from memory — that section is the source of truth for this step.';
|
|
const file = path.join(tmpDir, 'carved.md');
|
|
const original = path.join(tmpDir, 'before-carve.md');
|
|
const source = SYNTHETIC_SKILL.replace('\n## Step 2 — Other', '\n---\n\n## Step 2 — Other');
|
|
fs.writeFileSync(original, source);
|
|
fs.writeFileSync(file, source.replace(
|
|
'\n## Step 2 — Other', `\n${pointer}\n\n---\n\n## Step 2 — Other`,
|
|
));
|
|
expect(extractSkillSections(file, ['Step 1 — Do the thing']))
|
|
.toBe(extractSkillSections(original, ['Step 1 — Do the thing']));
|
|
expect(extractSkillSections(file, ['Step 2 — Other']))
|
|
.toBe(extractSkillSections(original, ['Step 2 — Other']));
|
|
expect(extractSkillBody(file)).toContain(pointer);
|
|
});
|
|
|
|
test('nested pointers and ordinary STOP quotes remain part of the requested H2', () => {
|
|
const pointer = '> **STOP.** Before the next step, Read `~/review/sections/next.md` and execute it\n'
|
|
+ '> in full. Do not work from memory — that section is the source of truth for this step.';
|
|
for (const prose of [
|
|
`\`\`\`md\n---\n\n${pointer}\n\n---\n\`\`\``,
|
|
`### Nested step\n\nInstructions for this step.\n\n${pointer}\nContinue this step.`,
|
|
`---\n\n${pointer.replace('> in full.', '> First,')}\n\n---`,
|
|
]) {
|
|
const file = path.join(tmpDir, 'quoted.md');
|
|
fs.writeFileSync(file, SYNTHETIC_SKILL.replace('step one body', `step one body\n${prose}`));
|
|
const out = extractSkillSections(file, ['Step 1 — Do the thing']);
|
|
expect(out).toContain(prose);
|
|
expect(out).toContain('step one continues after the fence');
|
|
}
|
|
});
|
|
|
|
test('missing section throws with the section name and the file path', () => {
|
|
expect(() => extractSkillSections(skillDir, ['Step 99 — Renamed'])).toThrow(/Step 99 — Renamed/);
|
|
expect(() => extractSkillSections(skillDir, ['Step 99 — Renamed'])).toThrow(/SKILL\.md/);
|
|
});
|
|
|
|
test('a fenced heading is not findable as a section', () => {
|
|
expect(() => extractSkillSections(skillDir, ['Embedded Template Heading'])).toThrow(/not found/);
|
|
});
|
|
});
|
|
|
|
describe('extractSkillBody (synthetic)', () => {
|
|
test('keeps frontmatter + intro + full body, drops the shared preamble block', () => {
|
|
const out = extractSkillBody(skillDir);
|
|
expect(out.startsWith('---\nname: fixture-test')).toBe(true);
|
|
expect(out).toContain('Intro line before any section');
|
|
expect(out).toContain('## When to invoke this skill');
|
|
expect(out).toContain('## Step 1 — Do the thing');
|
|
expect(out).toContain('template content inside a fence');
|
|
expect(out).toContain('## Step 2 — Other');
|
|
expect(out).not.toContain('preamble junk');
|
|
expect(out).not.toContain('shared-preamble junk');
|
|
expect(out).not.toContain('footer junk');
|
|
});
|
|
|
|
test.each([
|
|
['plain', '', '# Actual'],
|
|
['one-space indent', '', ' # Actual'],
|
|
['three-space indent', '', ' # Actual'],
|
|
['tab separator', '', '#\tActual'],
|
|
['empty title', '', '#'],
|
|
['backtick fence', '```md\n# FALSE TITLE\n## FALSE SECTION\n```\n', '# Actual'],
|
|
['tilde fence', '~~~md\n# FALSE TITLE\n~~~\n', '# Actual'],
|
|
['short nested fence', '````md\n```\n# FALSE TITLE\n```\n````\n', '# Actual'],
|
|
['mismatched fence', '~~~md\n```\n# FALSE TITLE\n~~~\n', '# Actual'],
|
|
['nonclosing suffix', '```md\n```not-a-close\n# FALSE TITLE\n```\n', '# Actual'],
|
|
['indented code', ' # FALSE TITLE\n', '# Actual'],
|
|
['missing separator', '#FALSE TITLE\n', '# Actual'],
|
|
['quoted title', '> # FALSE TITLE\n', '# Actual'],
|
|
])('preserves a post-preamble H1 introduction: %s', (_name, decoy, title) => {
|
|
const file = path.join(tmpDir, 'title-boundary.md');
|
|
fs.writeFileSync(file, SYNTHETIC_SKILL.replace(
|
|
'footer junk, last shared-preamble section\n\n',
|
|
`footer junk, last shared-preamble section\n${decoy}${title}\nREAL INTRODUCTION\n\n`,
|
|
));
|
|
const out = extractSkillBody(file);
|
|
expect(out).toBe(extractSkillBody(skillDir).replace(
|
|
'## Step 1 — Do the thing', `${title}\nREAL INTRODUCTION\n\n## Step 1 — Do the thing`,
|
|
));
|
|
expect(out).not.toContain('FALSE TITLE');
|
|
expect(out).not.toContain('footer junk');
|
|
expect(extractSkillSections(file, ['Plan Status Footer']))
|
|
.toContain(`${decoy}${title}\nREAL INTRODUCTION`);
|
|
});
|
|
|
|
test('a post-preamble H1 and introduction are a complete body without another H2', () => {
|
|
const file = path.join(tmpDir, 'title-only-body.md');
|
|
fs.writeFileSync(file, SYNTHETIC_SKILL.slice(0, SYNTHETIC_SKILL.indexOf('## Step 1 — Do the thing'))
|
|
+ '# Actual\nREAL INTRODUCTION\n');
|
|
const out = extractSkillBody(file);
|
|
expect(out).toContain('# Actual\nREAL INTRODUCTION');
|
|
expect(out).not.toContain('footer junk');
|
|
});
|
|
|
|
test('throws when the preamble markers are missing', () => {
|
|
const bare = path.join(tmpDir, 'bare');
|
|
fs.mkdirSync(bare, { recursive: true });
|
|
fs.writeFileSync(path.join(bare, 'SKILL.md'), '---\nname: bare\n---\n## Only Section\nbody\n');
|
|
expect(() => extractSkillBody(bare)).toThrow(/Preamble \(run first\)/);
|
|
});
|
|
|
|
test('both exact preamble headings preserve the complete intro/scope gate and tail', () => {
|
|
for (const heading of ['Preamble (run first)', 'Preamble (after scope gate)']) {
|
|
const file = path.join(tmpDir, 'scoped-body.md');
|
|
const scope = '## Scope gate\nAnnounce the actual selected plan before any tools.\n\n';
|
|
const input = SYNTHETIC_SKILL.replace('## Preamble (run first)', scope + `## ${heading}`);
|
|
fs.writeFileSync(file, input);
|
|
const out = extractSkillBody(file);
|
|
const expected = extractSkillBody(skillDir).replace('Invoke text.\n\n', 'Invoke text.\n\n' + scope);
|
|
expect(out).toBe(expected);
|
|
expect(out).toContain(scope.trimEnd());
|
|
expect(out).not.toContain(`## ${heading}`);
|
|
}
|
|
});
|
|
|
|
test('unknown, missing, duplicated and reversed boundary sections fail loudly', () => {
|
|
const cases: Array<[string, string, RegExp]> = [
|
|
['unknown start', SYNTHETIC_SKILL.replace('Preamble (run first)', 'Preamble (after setup)'), /Preamble/],
|
|
['suffixed scoped start', SYNTHETIC_SKILL.replace('Preamble (run first)', 'Preamble (after scope gate) extra'), /Preamble/],
|
|
['suffixed old start', SYNTHETIC_SKILL.replace('Preamble (run first)', 'Preamble (run first) extra'), /Preamble/],
|
|
['missing footer', SYNTHETIC_SKILL.replace('## Plan Status Footer', '## Renamed Footer'), /Plan Status Footer/],
|
|
['suffixed footer', SYNTHETIC_SKILL.replace('## Plan Status Footer', '## Plan Status Footer extra'), /Plan Status Footer/],
|
|
['both starts', SYNTHETIC_SKILL.replace('## AskUserQuestion Format', '## Preamble (after scope gate)'), /ambiguous/],
|
|
['duplicate start', SYNTHETIC_SKILL.replace('## AskUserQuestion Format', '## Preamble (run first)'), /ambiguous/],
|
|
['duplicate footer', SYNTHETIC_SKILL.replace('## Step 2 — Other', '## Plan Status Footer'), /ambiguous/],
|
|
['footer before start', SYNTHETIC_SKILL.replace('## Preamble (run first)', '## Plan Status Footer').replace('## Plan Status Footer\nfooter junk', '## Preamble (after scope gate)\nfooter junk'), /precedes/],
|
|
['missing tail', SYNTHETIC_SKILL.slice(0, SYNTHETIC_SKILL.indexOf('## Step 1 — Do the thing')), /no content after/],
|
|
];
|
|
for (const [name, input, error] of cases) {
|
|
const file = path.join(tmpDir, 'invalid-boundaries.md'); fs.writeFileSync(file, input);
|
|
expect(() => extractSkillBody(file), name).toThrow(error);
|
|
expect(() => extractSkillBody(file), name).toThrow(/invalid-boundaries\.md/);
|
|
}
|
|
});
|
|
|
|
test('quoted and fenced boundary examples do not become live section markers', () => {
|
|
for (const [open, close] of [['```markdown\n', '```\n'], ['~~~markdown\n', '~~~\n']]) {
|
|
const file = path.join(tmpDir, 'fenced-boundaries.md');
|
|
const examples = `${open}## Preamble (after scope gate)\n## Plan Status Footer\n${close}> ## Preamble (after scope gate)\n\n`;
|
|
fs.writeFileSync(file, SYNTHETIC_SKILL.replace('Intro line before any section.', examples + 'Intro line before any section.'));
|
|
expect(extractSkillBody(file)).toContain(examples.trimEnd());
|
|
fs.writeFileSync(file, SYNTHETIC_SKILL.replace('## Preamble (run first)\npreamble junk', `${open}## Preamble (after scope gate)\n${close}preamble junk`));
|
|
expect(() => extractSkillBody(file)).toThrow(/Preamble/);
|
|
}
|
|
});
|
|
|
|
test('other section extraction keeps its existing prefix matching', () => {
|
|
const file = path.join(tmpDir, 'section-prefix.md');
|
|
fs.writeFileSync(file, SYNTHETIC_SKILL.replace('Preamble (run first)', 'Preamble (after scope gate)'));
|
|
expect(extractSkillSections(file, ['Preamble (after'])).toContain('preamble junk');
|
|
expect(extractSkillSections(file, ['Step 1'])).toContain('step one continues after the fence');
|
|
});
|
|
|
|
test('the existing global source dependency selects every affected workflow fixture', () => {
|
|
expect(GLOBAL_TOUCHFILES).toContain('test/helpers/skill-fixture.ts');
|
|
const selection = selectTests(['test/helpers/skill-fixture.ts'], E2E_TOUCHFILES, GLOBAL_TOUCHFILES);
|
|
expect(selection.selected.sort()).toEqual(Object.keys(E2E_TOUCHFILES).sort());
|
|
});
|
|
});
|
|
|
|
describe('extractSkillHead (synthetic)', () => {
|
|
test('keeps frontmatter + first N body lines only', () => {
|
|
const out = extractSkillHead(skillDir, 3);
|
|
expect(out.startsWith('---\nname: fixture-test')).toBe(true);
|
|
expect(out).toContain('Intro line before any section');
|
|
expect(out).toContain('## When to invoke this skill');
|
|
expect(out).not.toContain('## Step 1 — Do the thing');
|
|
expect(out).toContain('body truncated by test/helpers/skill-fixture.ts');
|
|
});
|
|
});
|
|
|
|
describe('error polarity', () => {
|
|
test('missing SKILL.md throws (never writes an empty fixture)', () => {
|
|
expect(() => extractSkillSections(path.join(tmpDir, 'nope'), ['x'])).toThrow(/no SKILL\.md/);
|
|
});
|
|
|
|
test('file without frontmatter throws', () => {
|
|
const nofm = path.join(tmpDir, 'nofm');
|
|
fs.mkdirSync(nofm, { recursive: true });
|
|
fs.writeFileSync(path.join(nofm, 'SKILL.md'), '# no frontmatter\n## Section\n');
|
|
expect(() => extractSkillHead(nofm)).toThrow(/frontmatter/);
|
|
});
|
|
});
|
|
|
|
// ─── Pins against the real generated SKILL.md files ─────────────────────────
|
|
// These turn "someone renamed a section in gen-skill-docs" into a FREE test
|
|
// failure instead of a paid E2E setup throw.
|
|
|
|
describe('real-skill pins: section lists used by E2E fixtures', () => {
|
|
test('REVIEW_E2E_SECTIONS extracts from review/SKILL.md', () => {
|
|
const out = extractSkillSections(path.join(ROOT, 'review'), REVIEW_E2E_SECTIONS);
|
|
expect(out).toContain('## Step 4: Critical pass (core review)');
|
|
expect(out).toContain('## Important Rules');
|
|
// Drops the shared preamble and the untested workflow tail.
|
|
expect(out).not.toContain('## Telemetry (run last)');
|
|
expect(out).not.toContain('## Step 5: Fix-First Review');
|
|
expect(out).not.toContain('review/sections/review-army.md');
|
|
// Step 4's sideways-sweep paragraph (plan D2 replaced the enum-only one) survives the extraction.
|
|
expect(out).toContain('Sideways sweep (every finding; reads code OUTSIDE the diff).');
|
|
// Meaningfully smaller than the source.
|
|
const full = fs.readFileSync(path.join(ROOT, 'review', 'SKILL.md'), 'utf-8');
|
|
expect(out.length).toBeLessThan(full.length * 0.5);
|
|
});
|
|
|
|
test('REVIEW_ARMY_E2E_SECTIONS extracts from review/SKILL.md + carved sections', () => {
|
|
// /review is carved (T9): the Step 4.5 dispatch body lives in
|
|
// sections/review-army.md and the Plan Completion Audit in
|
|
// sections/plan-completion.md — the skeleton keeps only STOP-Read pointers.
|
|
// The E2E fixture (test/skill-e2e-review-army.test.ts) extracts the
|
|
// skeleton H2s minus Step 4.5, then appends both section files; this pin
|
|
// mirrors that exact construction so a section rename or an empty carve
|
|
// still fails FREE before a paid E2E setup throw.
|
|
const skeletonSections = REVIEW_ARMY_E2E_SECTIONS.filter(
|
|
(s) => !s.startsWith('Step 4.5'),
|
|
);
|
|
const out = extractSkillSections(path.join(ROOT, 'review'), skeletonSections);
|
|
expect(out).toContain('## Step 1.5: Scope Drift Detection');
|
|
expect(out).not.toContain('## Telemetry (run last)');
|
|
|
|
const planCompletion = fs.readFileSync(
|
|
path.join(ROOT, 'review', 'sections', 'plan-completion.md'), 'utf-8');
|
|
expect(planCompletion).toContain('PLAN COMPLETION AUDIT');
|
|
|
|
const army = fs.readFileSync(
|
|
path.join(ROOT, 'review', 'sections', 'review-army.md'), 'utf-8');
|
|
expect(army).toContain('## Step 4.5: Review Army — Specialist Dispatch');
|
|
expect(army).toContain('quality_score');
|
|
expect(army).toContain('MULTI-SPECIALIST CONFIRMED');
|
|
});
|
|
|
|
test('RETRO_E2E_SECTIONS skeleton extracts from retro/SKILL.md + carved section', () => {
|
|
// Carved (retro wave): the '## Engineering Retro: [date range]' report
|
|
// format lives in retro/sections/report-format.md; the E2E fixture builds
|
|
// skeleton sections + the section file (see skill-e2e-retro.test.ts).
|
|
const skeletonSections = RETRO_E2E_SECTIONS.filter(
|
|
(s) => s !== 'Engineering Retro: [date range]',
|
|
);
|
|
const out = extractSkillSections(path.join(ROOT, 'retro'), skeletonSections);
|
|
// Steps 0.5-14 live under Prior Learnings / Capture Learnings.
|
|
expect(out).toContain('### Step 1: Gather');
|
|
expect(out).toContain('### Step 14: Write the Narrative');
|
|
expect(out).not.toContain('## Global Retrospective Mode');
|
|
expect(out).toContain('Read `~/.claude/skills/gstack/retro/sections/report-format.md` and execute it');
|
|
expectMentions(out, [['stop', 'repo-scoped', 'result-save']], 'out');
|
|
expect(out).not.toContain('## Telemetry (run last)');
|
|
|
|
const reportFormat = fs.readFileSync(
|
|
path.join(ROOT, 'retro', 'sections', 'report-format.md'), 'utf-8');
|
|
expect(reportFormat).toContain('## Engineering Retro: [date range]');
|
|
expect(reportFormat).toContain('### Team Breakdown');
|
|
});
|
|
|
|
test('CODEX_REVIEW_E2E_SECTIONS extracts from the Codex host variant when present', () => {
|
|
const codexReview = path.join(ROOT, '.agents', 'skills', 'gstack-review');
|
|
if (!fs.existsSync(path.join(codexReview, 'SKILL.md'))) return; // gitignored artifact, absent in fresh checkouts
|
|
const out = extractSkillSections(codexReview, CODEX_REVIEW_E2E_SECTIONS);
|
|
expect(out).toContain('## Step 4: Critical pass (core review)');
|
|
expect(out).not.toContain('## Telemetry (run last)');
|
|
});
|
|
});
|
|
|
|
describe('real-skill pins: body/head extraction used by E2E fixtures', () => {
|
|
test.each(['qa-only', 'skillify', 'context-save', 'context-restore'])(
|
|
'extractSkillBody(%s) preserves the complete real H1 introduction', (skill) => {
|
|
const file = path.join(ROOT, skill, 'SKILL.md');
|
|
const full = fs.readFileSync(file, 'utf8');
|
|
const title = full.indexOf(`\n# /${skill}`, full.indexOf('## Plan Status Footer'));
|
|
const nextSection = full.indexOf('\n## ', title);
|
|
expect(title).toBeGreaterThan(0);
|
|
expect(nextSection).toBeGreaterThan(title);
|
|
expect(extractSkillBody(file)).toContain(full.slice(title + 1, nextSection).trimEnd());
|
|
},
|
|
);
|
|
|
|
// scrape/skillify/context-*: skill-e2e-skillify + skill-e2e-context-skills.
|
|
// review/plan-eng-review/ship: skill-e2e-coverage-audit + skill-e2e-triage.
|
|
const BODY_EXTRACTED_SKILLS = [
|
|
'scrape', 'skillify', 'context-save', 'context-restore',
|
|
'review', 'plan-eng-review', 'ship',
|
|
];
|
|
|
|
for (const skill of BODY_EXTRACTED_SKILLS) {
|
|
test(`extractSkillBody(${skill}) drops the shared preamble, keeps the flow`, () => {
|
|
const out = extractSkillBody(path.join(ROOT, skill));
|
|
expect(out).not.toContain('## Preamble (run first)');
|
|
expect(out).not.toContain('## Preamble (after scope gate)');
|
|
expect(out).not.toContain('## Telemetry (run last)');
|
|
expect(out).not.toContain(SHARED_POINTER_HEADING);
|
|
const full = readSkillStartup(path.join(ROOT, skill, 'SKILL.md'));
|
|
expect(out.length).toBeLessThan(full.length * 0.75);
|
|
expect(out.length).toBeGreaterThan(500);
|
|
});
|
|
}
|
|
|
|
test('body extraction keeps the sections the skillify/context E2E tests assert on', () => {
|
|
expect(extractSkillBody(path.join(ROOT, 'skillify'))).toContain('## Step 1 — Provenance guard (D1)');
|
|
expect(extractSkillBody(path.join(ROOT, 'scrape'))).toContain('## Step 2 — Refuse mutating intents');
|
|
expect(extractSkillBody(path.join(ROOT, 'context-save'))).toContain('## List flow');
|
|
expect(extractSkillBody(path.join(ROOT, 'context-restore'))).toContain('## If no saved contexts exist');
|
|
});
|
|
|
|
test('the scoped Eng render retains its original scope gate before the workflow', () => {
|
|
const file = path.join(ROOT, 'plan-eng-review', 'SKILL.md');
|
|
const full = fs.readFileSync(file, 'utf-8');
|
|
const scopeStart = full.indexOf('## Scope gate');
|
|
const preamble = full.indexOf(SHARED_POINTER_HEADING);
|
|
expect(scopeStart).toBeGreaterThan(0); expect(preamble).toBeGreaterThan(scopeStart);
|
|
expect(extractSkillBody(file)).toContain(full.slice(scopeStart, preamble).trimEnd());
|
|
});
|
|
|
|
// The union of skills installed by the routing + opus-47 discovery fixtures.
|
|
const HEAD_EXTRACTED_SKILLS = [
|
|
'', 'qa', 'qa-only', 'ship', 'review', 'plan-ceo-review', 'plan-eng-review',
|
|
'plan-design-review', 'design-review', 'design-consultation', 'retro',
|
|
'document-release', 'investigate', 'office-hours', 'browse',
|
|
'setup-browser-cookies', 'gstack-upgrade', 'humanizer',
|
|
];
|
|
|
|
test('extractSkillHead works for every discovery-fixture skill', () => {
|
|
for (const skill of HEAD_EXTRACTED_SKILLS) {
|
|
const src = path.join(ROOT, skill, 'SKILL.md');
|
|
if (!fs.existsSync(src)) continue; // mirrors the fixtures' existsSync guard
|
|
const out = extractSkillHead(src);
|
|
expect(out.startsWith('---\n')).toBe(true);
|
|
expect(out).toContain('description:');
|
|
// Frontmatter length varies (allowed-tools + triggers); the invariant
|
|
// is "frontmatter + 30 body lines + marker", never the full body.
|
|
const fullLines = fs.readFileSync(src, 'utf-8').split('\n').length;
|
|
expect(out.split('\n').length).toBeLessThan(Math.min(150, fullLines));
|
|
}
|
|
});
|
|
});
|
|
|
|
// Execute only the actual installation function, never the paid module setup.
|
|
test('routing catalog contains installed project names without a request-to-skill answer key', () => {
|
|
const source = fs.readFileSync(path.join(ROOT, 'test/skill-routing-e2e.test.ts'), 'utf8');
|
|
const start = source.indexOf('function installSkills(tmpDir: string)');
|
|
const end = source.indexOf('/** Init a git repo', start);
|
|
expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start);
|
|
const installed = new Map<string, string>([
|
|
[path.join(ROOT, 'SKILL.md'), '---\nname: gstack\ndescription: Route project workflows.\n---\n'],
|
|
[path.join(ROOT, 'review/SKILL.md'), '---\nname: review\ndescription: Deliberately unrelated photography text.\n---\n'],
|
|
[path.join(ROOT, 'qa/SKILL.md'), '---\nname: qa\ndescription: Test browser flows.\n---\n'],
|
|
]);
|
|
const writes = new Map<string, string>();
|
|
const fixtureRoot = path.join(os.tmpdir(), 'routing-catalog-no-files-written');
|
|
const mockFs = {
|
|
existsSync: (file: string) => installed.has(file),
|
|
mkdirSync: () => {},
|
|
writeFileSync: (file: string, content: string) => writes.set(file, content),
|
|
};
|
|
const compiled = new Bun.Transpiler({ loader: 'ts' }).transformSync(source.slice(start, end) + '\ninstallSkills(fixtureRoot);');
|
|
new Function('ROOT', 'fs', 'path', 'extractSkillHead', 'readShippedSkillRouting', 'fixtureRoot', compiled)(
|
|
ROOT, mockFs, path, (file: string) => installed.get(file), readShippedSkillRouting, fixtureRoot,
|
|
);
|
|
const instructions = writes.get(path.join(fixtureRoot, 'CLAUDE.md'))!;
|
|
expect(instructions).toContain(readShippedSkillRouting().instruction);
|
|
expect(instructions).not.toContain('→ invoke');
|
|
expect(instructions).toContain('installed gstack skills: gstack, qa, review.');
|
|
expect(instructions).not.toContain('photography');
|
|
expect(instructions).not.toContain('code-review');
|
|
expect(instructions).not.toContain('ship');
|
|
expect(instructions).not.toContain("I'm about to merge");
|
|
expect(writes.get(path.join(fixtureRoot, '.claude/skills/review/SKILL.md')))
|
|
.toBe(installed.get(path.join(ROOT, 'review/SKILL.md')));
|
|
expect(writes.size).toBe(4);
|
|
});
|
|
|
|
test('routing fixtures read the ## Skill routing section gstack ships', () => {
|
|
const routing = readShippedSkillRouting();
|
|
expect(routing.instruction).toStartWith('## Skill routing\n\n');
|
|
expect(routing.instruction).toContain('Skill tool');
|
|
expect(routing.rules.some(rule => rule.includes('/context-save'))).toBe(true);
|
|
expect(routing.rules.some(rule => rule.includes('/context-restore'))).toBe(true);
|
|
expect(routing.section).not.toContain('If B:');
|
|
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'shipped-routing-drift-'));
|
|
try {
|
|
fs.mkdirSync(path.join(root, 'bin'));
|
|
fs.writeFileSync(path.join(root, 'bin', 'gstack-skill-start'), '#!/bin/bash\necho no routing here\n');
|
|
expect(() => readShippedSkillRouting(root)).toThrow('routing-injection heredoc not found');
|
|
} finally { fs.rmSync(root, { recursive: true, force: true }); }
|
|
});
|
|
|
|
// Stored synthesized SKILL.md bodies for the skillify narration grader.
|
|
{
|
|
const DIR = path.join(import.meta.dir, 'fixtures', 'skillify-narration');
|
|
const read = (name: string) => fs.readFileSync(path.join(DIR, name), 'utf8');
|
|
const KNOWN_BAD = fs.readdirSync(DIR).filter(name => name.startsWith('known-bad-')).sort();
|
|
|
|
describe('skillify narration grader', () => {
|
|
test('accepts a clean body that says "let me know" in a trigger', () => {
|
|
expect(hasNarrationLeak(read('known-good.md'))).toBe(false);
|
|
});
|
|
test('stored known-bad fixtures exist', () => {
|
|
expect(KNOWN_BAD.length).toBeGreaterThanOrEqual(4);
|
|
});
|
|
test.each(KNOWN_BAD)('still rejects %s', (name) => {
|
|
expect(hasNarrationLeak(read(name))).toBe(true);
|
|
});
|
|
test('the previous grader rejected every known-bad fixture and also the known-good one', () => {
|
|
const previous = (body: string) => /^I /m.test(body) || /Let me /i.test(body) || /^I'll /m.test(body);
|
|
for (const name of KNOWN_BAD) expect(previous(read(name))).toBe(true);
|
|
expect(previous(read('known-good.md'))).toBe(true);
|
|
});
|
|
});
|
|
}
|