mirror of
https://github.com/garrytan/gstack.git
synced 2026-07-09 17:38:40 +02:00
test(coverage): close 5 remaining v1.46.0.0 test gaps (A-E)
Five behaviors that v1.46 ships but had no test coverage. All now pinned. A) --host all idempotency (test/gen-skill-docs-idempotency.test.ts) The default test ran Claude host only. Non-Claude hosts (Codex, Factory, Cursor, OpenClaw, GBrain, Slate, OpenCode, Hermes, Kiro) each have their own output paths and could carry their own non-deterministic fields. We hit a "--host all needed for freshness check" mid-/ship. Now: two consecutive `bun run gen:skill-docs --host all` runs must produce byte-identical outputs across a per-host sample (.agents/, .cursor/, .factory/, .gbrain/). Catches per-host adapter regressions before CI. B) --catalog-mode=full opt-out (test/catalog-mode-full.test.ts) The legacy escape hatch had zero tests. 6 new tests across two layers: static (CATALOG_MODE_ARG parsed; conditional gate present; default is "trim"; invalid value throws) + smoke (actual --catalog-mode=full run produces a multi-line `description: |` block + omits "## When to invoke" body section; mutates the working tree then restores in a finally block). C) parity-baseline-v1.44.1.json integrity (test/parity-baseline-integrity.test.ts) The baseline is the source of every v1→v2 number cited in the CHANGELOG v1.46.0.0 entry. Anyone could edit it without test failure until now. 8 new tests pin: existence, tag, capturedFromCommit allowlist, expected v1.44 numbers (51 skills, ~2,915 KB, ~9,319 catalog tokens), CHANGELOG references this file by path, per-skill shape, and a SHA256 byte-stability hash. Any edit fails with a clear "if intentional, update EXPECTED_HASH AND the CHANGELOG numbers" signal. D) Live appliesTo gate end-to-end (test/resolver-entry.test.ts extended) The unwrapResolver unit tests covered the function; the gen-skill-docs.ts substitution loop that USES the gate had no integration coverage. 6 new tests simulate the exact 4-line shape from gen-skill-docs.ts:457-467 against synthetic registries: plain-function fires unconditionally, gated fires when true / empty-string when false, mixed registries compose, parameterized resolvers respect gates, unknown resolvers throw. E) Per-skill min-size floor (test/skill-size-budget.test.ts extended) The existing 200-byte body coverage-floor is a noise floor — a skill that lost 99.75% of content still passes. 1 new test asserts every skill stays ≥80% of its v1.44.1 baseline size (the parity-suite content invariants only covered 10 of 51 skills; the remaining 41 were uncovered). SECTIONS_EXTRACTED hook in place for v2.0.0.0 when the sections/ pattern legitimately shrinks ship/plan-ceo/etc. past the floor. Test plan: - bun test focused 17-file suite: 1202 pass, 0 fail (+23 new tests vs the pre-fill 1179 baseline) - catalog-mode=full mutates working tree then restores cleanly - --host all idempotency runs two full gen passes in <1s on this machine Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -126,6 +126,75 @@ describe('SKILL.md size budget regression (gate, free)', () => {
|
||||
);
|
||||
});
|
||||
|
||||
/**
|
||||
* Gap E (v1.46.0.0): per-skill min-size floor.
|
||||
*
|
||||
* The existing skill-coverage-floor enforces body ≥ 200 bytes, which is
|
||||
* a tiny noise floor. A skill that was 100 KB at v1.44.1 and shrinks to
|
||||
* 250 bytes passes that check despite losing 99.75% of content. The
|
||||
* parity-suite content invariants cover this for 10 hand-picked skills
|
||||
* (cso, ship, plan-ceo, etc.); the remaining 41 skills had no per-skill
|
||||
* shrinkage floor.
|
||||
*
|
||||
* Floor: 80% of the v1.44.1 baseline. v1.46 actual shrinkage is <1% per
|
||||
* skill, so this is a comfortable ceiling that still catches accidental
|
||||
* mass deletion (e.g., a refactor that strips the body of a skill).
|
||||
*
|
||||
* v2.0.0.0 will introduce the sections/ pattern for 5 heavyweights
|
||||
* (ship, plan-ceo-review, office-hours, plan-eng-review,
|
||||
* plan-design-review). Those skills will legitimately shrink to ~15 KB
|
||||
* skeletons. When that lands, add them to SECTIONS_EXTRACTED so the floor
|
||||
* relaxes for them.
|
||||
*/
|
||||
test('no skill shrinks past 80% of v1.44.1 baseline (catches accidental body strip)', () => {
|
||||
const baseline: ParityBaseline = JSON.parse(fs.readFileSync(BASELINE_PATH, 'utf-8'));
|
||||
const current = captureBaseline({ repoRoot: REPO_ROOT });
|
||||
const MIN_RATIO = 0.80; // a skill at <80% of its v1.44 size signals mass-deletion
|
||||
const SECTIONS_EXTRACTED = new Set<string>(); // populate in v2.0.0.0 when sections/ lands
|
||||
|
||||
const undershoots: Array<{
|
||||
skill: string; beforeBytes: number; afterBytes: number; ratio: number;
|
||||
}> = [];
|
||||
for (const [skill, before] of Object.entries(baseline.skills)) {
|
||||
if (SECTIONS_EXTRACTED.has(skill)) continue;
|
||||
const after = current.skills[skill];
|
||||
if (!after) continue; // skill removed since baseline — separate concern
|
||||
const ratio = after.skillMdBytes / before.skillMdBytes;
|
||||
if (ratio < MIN_RATIO) {
|
||||
undershoots.push({
|
||||
skill, beforeBytes: before.skillMdBytes, afterBytes: after.skillMdBytes, ratio,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (undershoots.length === 0) return;
|
||||
|
||||
const overrideReason = process.env.GSTACK_SIZE_BUDGET_OVERRIDE_REASON?.trim();
|
||||
if (overrideReason) {
|
||||
logBudgetOverride({
|
||||
scope: 'skill-size-budget-floor',
|
||||
reason: overrideReason,
|
||||
details: { min_ratio: MIN_RATIO, undershoots },
|
||||
});
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
`[skill-size-budget-floor] OVERRIDE APPLIED (${overrideReason}) — ${undershoots.length} undershoot(s) allowed`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const msg = undershoots.map(u =>
|
||||
` ${u.skill}: ${u.beforeBytes} → ${u.afterBytes} bytes (×${u.ratio.toFixed(2)} — below ${MIN_RATIO} floor)`,
|
||||
).join('\n');
|
||||
throw new Error(
|
||||
`${undershoots.length} skill(s) shrunk past v1.44.1 × ${MIN_RATIO} floor:\n${msg}\n` +
|
||||
`This usually signals accidental body strip (e.g., a resolver returning empty, a ` +
|
||||
`template losing a section). If the shrinkage is intentional (e.g., the skill moved ` +
|
||||
`to the sections/ pattern), add it to SECTIONS_EXTRACTED in this test. Override: ` +
|
||||
`GSTACK_SIZE_BUDGET_OVERRIDE_REASON="why" allows + audit-logs.`,
|
||||
);
|
||||
});
|
||||
|
||||
test('catalog token estimate stays compressed (v1.45 target ≤ 7000)', () => {
|
||||
const current = captureBaseline({ repoRoot: REPO_ROOT });
|
||||
const v145Target = 7000;
|
||||
|
||||
Reference in New Issue
Block a user