mirror of
https://github.com/garrytan/gstack.git
synced 2026-06-10 12:03:59 +02:00
v1.57.0.0 feat: carve-guard system + carve cso/document-release/design-consultation (#1907)
* test: canonical CARVE_GUARDS registry; derive parity + size-budget from it Single source of truth for the carved-skill set + per-skill invariants (EQ1). parity-harness.ts sectioned entries and skill-size-budget.ts SECTIONS_EXTRACTED now derive from it instead of hand-maintained lists. Closes a pre-existing drift: plan-devex-review was in SECTIONS_EXTRACTED but had no sectioned parity invariant; now generated. carve-guards.ts is a pure leaf data module (import type only) to avoid an import cycle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: shared carve-guard check fns with injectable root discoverCarvedSkills/checkOrdering/checkCompleteness take a root param so the negative tests can point the real guards at a fixture dir. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: E2 data-driven carve static ordering guard (gate) Per-PR backstop for every carved skill, one test() per skill, driven by CARVE_GUARDS staticInvariants. Generalizes + retires the ceo-specific ordering test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: E1 carve-guard completeness meta-guard (gate) Asserts filesystem carved set == CARVE_GUARDS set both directions, so a future carve without a registry entry fails CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: ET1 guard-of-guards negative tests (gate) Temp fixture broken 3 ways proves E1/E2 actually throw, via the injectable root. Kills the silent-pass-guard failure class. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: T2 data-driven behavioral section-loading guard (periodic) One file iterating CARVE_GUARDS, one test() per skill with GSTACK_CARVE_SKILL cost-scoping (D-CODEX A). external carves (ship, plan-ceo) keep bespoke tests; testNames aligned to their touchfile keys. Registered in touchfiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: defer E3 real-session carve canary to TODOS Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: carve document-release into skeleton + on-demand section Steps 2-9 (per-file audit, auto-updates, risky-change asks, CHANGELOG voice polish, cross-doc consistency, TODOS cleanup, VERSION bump, commit + PR body) move to sections/release-body.md, read on demand after the Step 1.5 coverage map. Skeleton 59,256 -> 45,797 B (-23%); union preserved. Adds the CARVE_GUARDS entry (auto-extends parity + size-budget via EQ1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: carve design-consultation into skeleton + on-demand section Phases 3-6 (complete proposal, drill-downs, design preview, writing DESIGN.md) move to sections/proposal-and-preview.md, read on demand after product context + research. Skeleton 80,719 -> 59,229 B (-27%); union preserved. Adds the CARVE_GUARDS entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: carve cso into skeleton + on-demand section (security-safe) Scope-dependent audit Phases 2-11 move to sections/audit-phases.md. Mode dispatch (## Arguments, ## Mode Resolution), always-run Phases 0/1, and the Phase 12 false-positive-filtering exceptions stay ALWAYS-LOADED in the skeleton. Skeleton 79,383 -> 65,117 B (-18%); union preserved. Adds a cso CARVE_GUARDS entry with an earliest-use invariant (mustPrecedeStop): mode dispatch must appear before any STOP-Read, so a directive that decides which sections to read can't be stranded behind the STOP that reads them (codex outside-voice #6). carve-guard-checks gains the mustPrecedeStop check. parity moves cso monolith -> generated carved entry. cso-preserved.test.ts strengthened: phrases checked against the union, plus an always-loaded contract on the skeleton (dispatch + FP-filtering, codex #5). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: make redaction/taxonomy tests union-aware for cso + document-release carves The cso carve moved Secrets Archaeology (prefixes, lib/redact-patterns.ts pointer, git-history scan) into sections/audit-phases.md, and the document-release carve moved the Step 9 PR-body redaction scan into sections/release-body.md. Three content-presence tests asserted that content in the skeleton SKILL.md/.md.tmpl; they now read the skeleton+sections union (same fix as cso-preserved + parity). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v1.57.0.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address pre-landing review (codex) on the carve - cso section: add a scope-gate header so '--owasp' (and other scoped modes) run only their selected phases, not every phase bundled in the section ('execute in full' no longer overrides Mode Resolution). - carve-guard-checks: gateAfterStop now compares against the LAST STOP, not the first, so a gate stranded between two STOPs in a multi-STOP skeleton fails. - TODOS: behavioral section-loading hermeticity (verifier matches global-install path, not the fixture) — pre-existing in auq-sdk-capture.ts, deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
+74
-53
@@ -1,16 +1,22 @@
|
||||
/**
|
||||
* cso security-guidance preservation test (v1.45.0.0 T6).
|
||||
* cso security-guidance preservation test.
|
||||
*
|
||||
* The cso skill carries load-bearing security prose: OWASP Top 10 mappings,
|
||||
* STRIDE threat-model phrasing, "do not auto-fix without user approval"
|
||||
* gates. Codex 2nd-pass critique #9: "cso exemption too broad ... should
|
||||
* still get resolver dedup, catalog trim, sectioning if safe, and targeted
|
||||
* evals around must-not-miss checks."
|
||||
* cso carries load-bearing security prose: OWASP Top 10 mappings, STRIDE
|
||||
* threat-model phrasing, mode dispatch, and false-positive-filtering exceptions
|
||||
* that must NOT be auto-discarded.
|
||||
*
|
||||
* This test pins the must-not-miss checks. cso gets the same resolver gate
|
||||
* (T2), jargon dedup (T3), and catalog trim (T4) as every other skill — but
|
||||
* its security-guidance body content stays intact. Future compression work
|
||||
* that would strip this content fails CI here.
|
||||
* cso is now carved (skeleton SKILL.md + sections/audit-phases.md). The
|
||||
* scope-dependent audit phases (2-11) moved to the section; the mode dispatch
|
||||
* (## Arguments, ## Mode Resolution), the always-run phases (0, 1), and the
|
||||
* FP-filtering exceptions (Phase 12) stay always-loaded in the skeleton.
|
||||
*
|
||||
* Two distinct guarantees (codex outside-voice #5 — earliest-use, not loose
|
||||
* substrings):
|
||||
* 1. PRESERVATION — the security phrases survive somewhere in the union
|
||||
* (skeleton + sections); a carve relocates, it never drops.
|
||||
* 2. ALWAYS-LOADED CONTRACT — dispatch + FP-filtering directives stay in the
|
||||
* skeleton, and mode dispatch precedes any STOP-Read (a directive that
|
||||
* decides which sections to read can't sit behind the STOP that reads them).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
@@ -18,69 +24,84 @@ import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
const REPO_ROOT = path.resolve(import.meta.dir, '..');
|
||||
const CSO_SKILL = path.join(REPO_ROOT, 'cso', 'SKILL.md');
|
||||
const CSO_DIR = path.join(REPO_ROOT, 'cso');
|
||||
const CSO_SKELETON = path.join(CSO_DIR, 'SKILL.md');
|
||||
|
||||
const MUST_PRESERVE_PHRASES = [
|
||||
// OWASP / STRIDE positioning
|
||||
'OWASP',
|
||||
'STRIDE',
|
||||
// Mode discipline
|
||||
'daily',
|
||||
'comprehensive',
|
||||
// Severity language
|
||||
'confidence',
|
||||
// Active verification requirement (codex critique: "active verification")
|
||||
'verif', // covers "verify", "verification", "verified"
|
||||
];
|
||||
function readSkeleton(): string {
|
||||
return fs.readFileSync(CSO_SKELETON, 'utf-8');
|
||||
}
|
||||
function readUnion(): string {
|
||||
let text = readSkeleton();
|
||||
const dir = path.join(CSO_DIR, 'sections');
|
||||
if (fs.existsSync(dir)) {
|
||||
for (const f of fs.readdirSync(dir).sort()) {
|
||||
if (f.endsWith('.md') && !f.endsWith('.md.tmpl')) {
|
||||
text += '\n' + fs.readFileSync(path.join(dir, f), 'utf-8');
|
||||
}
|
||||
}
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
const MUST_PRESERVE_HEADINGS = [
|
||||
'## Preamble', // from PREAMBLE resolver
|
||||
];
|
||||
// Security content that must survive the carve (checked against the UNION).
|
||||
const MUST_PRESERVE_PHRASES = ['OWASP', 'STRIDE', 'daily', 'comprehensive', 'confidence', 'verif'];
|
||||
|
||||
describe('cso skill preserves load-bearing security guidance', () => {
|
||||
test('cso/SKILL.md exists and is non-trivial', () => {
|
||||
expect(fs.existsSync(CSO_SKILL)).toBe(true);
|
||||
const content = fs.readFileSync(CSO_SKILL, 'utf-8');
|
||||
// cso is a content-heavy security skill; under 30 KB suggests stripping went too far.
|
||||
expect(content.length).toBeGreaterThan(30_000);
|
||||
test('cso skeleton exists and is non-trivial', () => {
|
||||
expect(fs.existsSync(CSO_SKELETON)).toBe(true);
|
||||
// Skeleton stays substantial: dispatch + always-run phases + FP filtering +
|
||||
// report phases are all always-loaded. Under 30 KB means too much moved out.
|
||||
expect(readSkeleton().length).toBeGreaterThan(30_000);
|
||||
});
|
||||
|
||||
test('cso preserves required security phrases (case-insensitive)', () => {
|
||||
const content = fs.readFileSync(CSO_SKILL, 'utf-8').toLowerCase();
|
||||
const missing: string[] = [];
|
||||
for (const phrase of MUST_PRESERVE_PHRASES) {
|
||||
if (!content.includes(phrase.toLowerCase())) missing.push(phrase);
|
||||
}
|
||||
test('security phrases survive in the union (skeleton + sections)', () => {
|
||||
const union = readUnion().toLowerCase();
|
||||
const missing = MUST_PRESERVE_PHRASES.filter((p) => !union.includes(p.toLowerCase()));
|
||||
if (missing.length > 0) {
|
||||
throw new Error(
|
||||
`cso/SKILL.md is missing required security phrases: ${missing.join(', ')}. ` +
|
||||
`These are load-bearing for the skill's audit posture. If you intentionally ` +
|
||||
`removed them, update this test with the new phrasing.`,
|
||||
`cso union is missing required security phrases: ${missing.join(', ')}. ` +
|
||||
`These are load-bearing. A carve relocates them; it must not drop them.`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('cso preserves required headings', () => {
|
||||
const content = fs.readFileSync(CSO_SKILL, 'utf-8');
|
||||
for (const heading of MUST_PRESERVE_HEADINGS) {
|
||||
expect(content).toContain(heading);
|
||||
test('ALWAYS-LOADED: mode dispatch + FP-filtering stay in the skeleton', () => {
|
||||
const skeleton = readSkeleton();
|
||||
// Dispatch must be always-loaded — the agent resolves scope before reading sections.
|
||||
expect(skeleton).toContain('## Arguments');
|
||||
expect(skeleton).toContain('## Mode Resolution');
|
||||
// FP-filtering with its critical exceptions is mandatory and must not be on-demand.
|
||||
expect(skeleton).toContain('Phase 12');
|
||||
// The "SKILL.md files are NOT documentation" exception is a must-not-miss
|
||||
// security directive (skill supply-chain findings); it stays always-loaded.
|
||||
expect(skeleton).toContain('NOT documentation');
|
||||
});
|
||||
|
||||
test('EARLIEST-USE: mode dispatch precedes any STOP-Read directive (codex #6)', () => {
|
||||
const skeleton = readSkeleton();
|
||||
const stop = skeleton.indexOf('> **STOP.**');
|
||||
const modeRes = skeleton.indexOf('## Mode Resolution');
|
||||
const args = skeleton.indexOf('## Arguments');
|
||||
expect(modeRes).toBeGreaterThan(-1);
|
||||
expect(args).toBeGreaterThan(-1);
|
||||
if (stop >= 0) {
|
||||
// A dispatch directive stranded after the STOP can't govern which sections to read.
|
||||
expect(args).toBeLessThan(stop);
|
||||
expect(modeRes).toBeLessThan(stop);
|
||||
}
|
||||
});
|
||||
|
||||
test('cso catalog trim landed (frontmatter description ≤ 200 chars)', () => {
|
||||
const content = fs.readFileSync(CSO_SKILL, 'utf-8');
|
||||
const content = readSkeleton();
|
||||
const fmMatch = content.match(/^---\n([\s\S]*?)\n---/);
|
||||
expect(fmMatch).not.toBeNull();
|
||||
const fm = fmMatch![1];
|
||||
const descMatch = fm.match(/^description:\s+(.+)$/m);
|
||||
expect(descMatch).not.toBeNull();
|
||||
const desc = descMatch![1].trim();
|
||||
expect(desc.length).toBeLessThanOrEqual(200);
|
||||
expect(desc).toContain('(gstack)');
|
||||
const desc = fmMatch![1].match(/^description:\s+(.+)$/m);
|
||||
expect(desc).not.toBeNull();
|
||||
expect(desc![1].trim().length).toBeLessThanOrEqual(200);
|
||||
expect(desc![1]).toContain('(gstack)');
|
||||
});
|
||||
|
||||
test('cso routing prose moved to "## When to invoke" body section', () => {
|
||||
const content = fs.readFileSync(CSO_SKILL, 'utf-8');
|
||||
expect(content).toContain('## When to invoke this skill');
|
||||
expect(readSkeleton()).toContain('## When to invoke this skill');
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user