v1.57.0.0 feat: carve-guard system + carve cso/document-release/design-consultation (#1907)

* test: canonical CARVE_GUARDS registry; derive parity + size-budget from it Single source of truth for the carved-skill set + per-skill invariants (EQ1). parity-harness.ts sectioned entries and skill-size-budget.ts SECTIONS_EXTRACTED now derive from it instead of hand-maintained lists. Closes a pre-existing drift: plan-devex-review was in SECTIONS_EXTRACTED but had no sectioned parity invariant; now generated. carve-guards.ts is a pure leaf data module (import type only) to avoid an import cycle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: shared carve-guard check fns with injectable root discoverCarvedSkills/checkOrdering/checkCompleteness take a root param so the negative tests can point the real guards at a fixture dir. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: E2 data-driven carve static ordering guard (gate) Per-PR backstop for every carved skill, one test() per skill, driven by CARVE_GUARDS staticInvariants. Generalizes + retires the ceo-specific ordering test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: E1 carve-guard completeness meta-guard (gate) Asserts filesystem carved set == CARVE_GUARDS set both directions, so a future carve without a registry entry fails CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: ET1 guard-of-guards negative tests (gate) Temp fixture broken 3 ways proves E1/E2 actually throw, via the injectable root. Kills the silent-pass-guard failure class. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: T2 data-driven behavioral section-loading guard (periodic) One file iterating CARVE_GUARDS, one test() per skill with GSTACK_CARVE_SKILL cost-scoping (D-CODEX A). external carves (ship, plan-ceo) keep bespoke tests; testNames aligned to their touchfile keys. Registered in touchfiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: defer E3 real-session carve canary to TODOS Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: carve document-release into skeleton + on-demand section Steps 2-9 (per-file audit, auto-updates, risky-change asks, CHANGELOG voice polish, cross-doc consistency, TODOS cleanup, VERSION bump, commit + PR body) move to sections/release-body.md, read on demand after the Step 1.5 coverage map. Skeleton 59,256 -> 45,797 B (-23%); union preserved. Adds the CARVE_GUARDS entry (auto-extends parity + size-budget via EQ1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: carve design-consultation into skeleton + on-demand section Phases 3-6 (complete proposal, drill-downs, design preview, writing DESIGN.md) move to sections/proposal-and-preview.md, read on demand after product context + research. Skeleton 80,719 -> 59,229 B (-27%); union preserved. Adds the CARVE_GUARDS entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: carve cso into skeleton + on-demand section (security-safe) Scope-dependent audit Phases 2-11 move to sections/audit-phases.md. Mode dispatch (## Arguments, ## Mode Resolution), always-run Phases 0/1, and the Phase 12 false-positive-filtering exceptions stay ALWAYS-LOADED in the skeleton. Skeleton 79,383 -> 65,117 B (-18%); union preserved. Adds a cso CARVE_GUARDS entry with an earliest-use invariant (mustPrecedeStop): mode dispatch must appear before any STOP-Read, so a directive that decides which sections to read can't be stranded behind the STOP that reads them (codex outside-voice #6). carve-guard-checks gains the mustPrecedeStop check. parity moves cso monolith -> generated carved entry. cso-preserved.test.ts strengthened: phrases checked against the union, plus an always-loaded contract on the skeleton (dispatch + FP-filtering, codex #5). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: make redaction/taxonomy tests union-aware for cso + document-release carves The cso carve moved Secrets Archaeology (prefixes, lib/redact-patterns.ts pointer, git-history scan) into sections/audit-phases.md, and the document-release carve moved the Step 9 PR-body redaction scan into sections/release-body.md. Three content-presence tests asserted that content in the skeleton SKILL.md/.md.tmpl; they now read the skeleton+sections union (same fix as cso-preserved + parity). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v1.57.0.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address pre-landing review (codex) on the carve - cso section: add a scope-gate header so '--owasp' (and other scoped modes) run only their selected phases, not every phase bundled in the section ('execute in full' no longer overrides Mode Resolution). - carve-guard-checks: gateAfterStop now compares against the LAST STOP, not the first, so a gate stranded between two STOPs in a multi-STOP skeleton fails. - TODOS: behavioral section-loading hermeticity (verifier matches global-install path, not the fixture) — pre-existing in auq-sdk-capture.ts, deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 12:03:59 +02:00 · 2026-06-07 19:13:24 -07:00
parent 476b0ec597
commit e722c5bf89
34 changed files with 2981 additions and 2071 deletions
@@ -0,0 +1,97 @@
+/**
+ * T2 — data-driven behavioral section-loading guard (PERIODIC tier, paid, SDK capture).
+ *
+ * The behavioral proof that a REAL agent actually Reads each carved skill's
+ * required sections at runtime — not just that the skeleton structure looks right
+ * (that's E2, free, per-PR). One file iterating the canonical CARVE_GUARDS
+ * registry (EQ2): registry membership IS the test, so "registered ⇒ asserted" is
+ * structural — a carve can't be registered yet behaviorally unguarded.
+ *
+ * Per codex refined-plan pass:
+ *   #2 — ONE test() per skill, each with its own timeout + named failure output;
+ *        a hung claude -p fails only its skill, not the whole file.
+ *   #3 / D-CODEX(A) — GSTACK_CARVE_SKILL=<name> runs only that skill's case, so
+ *        the touchfile selector can scope cost to the changed skill; unset runs all.
+ *   #7 — each case drives the run with the registry's `scenario` (built to force
+ *        the STOP-Read path) and asserts the required sections were Read.
+ *
+ * 'external' skills (ship, plan-ceo-review) have bespoke fixtures (git state,
+ * Step-0 mode loop) and keep their dedicated tests; E1 asserts those exist.
+ */
+
+import { describe, test, expect } from 'bun:test';
+import { setupSkillDir, skillFromWorktree, captureSectionReads } from './helpers/auq-sdk-capture';
+import { CARVE_GUARDS } from './helpers/carve-guards';
+
+const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'periodic';
+const describeE2E = shouldRun ? describe : describe.skip;
+const runId = `carve-section-loading-${process.env.EVALS_RUN_ID ?? 'local'}`;
+const only = process.env.GSTACK_CARVE_SKILL?.trim();
+
+// A generic plan fixture for 'plan' behavioral skills (the review family).
+const PLAN_MD = [
+  '# Plan: add an in-memory cache layer',
+  '',
+  '## Context',
+  'Reads hit the DB on every request. Add a process-local LRU cache in front of the',
+  'read path to cut DB load.',
+  '',
+  '## Approach',
+  '- Wrap the read repository in a cache that stores the last 1000 keys.',
+  '- Invalidate on write.',
+  '',
+  '## Out of scope',
+  'Distributed cache, cross-process coherence.',
+  '',
+].join('\n');
+
+describeE2E('carve behavioral section-loading (periodic, SDK capture)', () => {
+  for (const guard of Object.values(CARVE_GUARDS)) {
+    // 'external' carves keep their dedicated bespoke tests (E1 verifies those exist).
+    if (guard.behavioral === 'external') continue;
+    // Cost-scoped selection: when GSTACK_CARVE_SKILL is set, run only that skill.
+    if (only && only !== guard.skill) continue;
+
+    test(
+      `${guard.skill}: a real run Reads ${guard.requiredReads.join(', ')}`,
+      async () => {
+        const { skillMd, sectionsFrom } = skillFromWorktree(guard.skill);
+        const fixtures = guard.behavioral === 'plan' ? { 'PLAN.md': PLAN_MD } : {};
+        const planDir = setupSkillDir({
+          skillName: guard.skill,
+          skillMd,
+          sectionsFrom,
+          fixtures,
+          tmpPrefix: `gstack-${guard.skill}-secload-`,
+        });
+
+        const { readSections, reportProduced, output } = await captureSectionReads({
+          planDir,
+          skillName: guard.skill,
+          scenario: guard.scenario,
+          reportMarker: /report|review|summary|design doc|handoff/i,
+          testName: `${guard.skill} section-loading`,
+          runId,
+        });
+
+        const missing = guard.requiredReads.filter((s) => !readSections.has(s));
+        // Named failure output (codex #2): skill + expected + observed.
+        expect({
+          skill: guard.skill,
+          reportProduced,
+          expected: guard.requiredReads,
+          observed: [...readSections],
+          missing,
+        }).toEqual({
+          skill: guard.skill,
+          reportProduced: true,
+          expected: guard.requiredReads,
+          observed: expect.any(Array),
+          missing: [],
+        });
+        expect(output.trim().length).toBeGreaterThan(200);
+      },
+      360_000,
+    );
+  }
+});