test(parity): T0a — capture v1.44.1 baseline + capture helper + diff utility

Cathedral parity-eval suite primitive. captureBaseline() walks every
top-level SKILL.md and records bytes, lines, estimated tokens, frontmatter
description length, and eval coverage. diffBaselines() reports per-skill
delta + total corpus delta + catalog tokens delta.

Locks the v1.44.1 reference snapshot at test/fixtures/parity-baseline-v1.44.1.json.
After Phase A+B+C land, scripts/capture-baseline.ts --tag v1.45.0.0 produces
a comparable snapshot; diff supplies the real numbers the v2 CHANGELOG quotes.
Never invent baseline numbers; ship them only if they came from a real run.

v1.44.1 numbers captured this commit:
- 51 skills
- 2,847 KB total corpus
- ~9,319 catalog tokens (sum of description bytes / 4)
- top 3: ship 160 KB, plan-ceo-review 128 KB, office-hours 108 KB

Test plan:
- bun test test/helpers/capture-parity-baseline.test.ts passes 4/4
- The baseline JSON file is committed so reviewers can audit v1→v2 numbers

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-25 20:29:47 -07:00
parent 74bc80545f
commit e274e5ec82
4 changed files with 998 additions and 0 deletions
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env bun
/**
* CLI for capturing a parity baseline snapshot.
*
* Usage:
* bun run scripts/capture-baseline.ts # default path
* bun run scripts/capture-baseline.ts --tag v1.44.1 # tag the snapshot
* bun run scripts/capture-baseline.ts --out path/to/baseline.json
*
* The default output path is test/fixtures/parity-baseline-<tag>.json,
* or test/fixtures/parity-baseline-current.json when no tag is given.
*/
import * as fs from 'fs';
import * as path from 'path';
import { captureBaseline } from '../test/helpers/capture-parity-baseline';
const ROOT = path.resolve(import.meta.dir, '..');
function arg(name: string): string | undefined {
const i = process.argv.indexOf(name);
if (i === -1) return undefined;
return process.argv[i + 1];
}
const tag = arg('--tag');
const outOverride = arg('--out');
const defaultOut = path.join(
ROOT,
'test',
'fixtures',
`parity-baseline-${tag ?? 'current'}.json`,
);
const outPath = outOverride ? path.resolve(outOverride) : defaultOut;
const baseline = captureBaseline({ repoRoot: ROOT, tag });
fs.mkdirSync(path.dirname(outPath), { recursive: true });
fs.writeFileSync(outPath, JSON.stringify(baseline, null, 2) + '\n');
const totalKB = Math.round(baseline.totalCorpusBytes / 1024);
const top3 = baseline.topHeaviest.slice(0, 3);
console.log(`Parity baseline captured: ${outPath}`);
console.log(` tag: ${baseline.tag}`);
console.log(` commit: ${baseline.capturedFromCommit}`);
console.log(` branch: ${baseline.capturedFromBranch}`);
console.log(` skills: ${baseline.totalSkills}`);
console.log(` total corpus: ${totalKB} KB`);
console.log(` catalog tokens: ~${baseline.estTotalCatalogTokens}`);
console.log(` top 3 heaviest:`);
for (const s of top3) {
const kb = Math.round(s.skillMdBytes / 1024);
console.log(` ${s.skill.padEnd(28)} ${kb} KB (${s.skillMdLines} lines, ~${s.estTokens} tokens)`);
}