CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-08-29 17:30:40 +02:00
Code Issues Packages Projects Releases Wiki Activity

v1.71.0.0 feat: token-load reduction — preamble runtime scripts, gated onboarding, 20 skill carves, CLAUDE.md trim (#2691)

Browse Source 
* feat(gen): strip gen-time-only frontmatter keys from Claude renders

interactive + benefits-from are read from the .tmpl by buildContext at
generation time; no runtime, host, or test reader consumes them from the
generated SKILL.md (e2e-harness-audit reads .tmpl; benefits-from tests
assert rendered prose). gbrain: stays (bin/gstack-brain-context-load reads
it from the installed render); hooks: stays (Claude Code host wires
PreToolUse from it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(gen): regenerate SKILL.md — dead frontmatter keys removed

Mechanical regen after hosts/claude.ts stripFields change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(test): context-budget ratchet — CI ceilings on always-on + eager token ledgers

New free test grades the two ledgers nothing else guards: the full-frontmatter
always-on catalog (aggregate) and per-skill eager tokens (SKILL.md +
forced-read refs), via checkBudget from lib/context-bill.ts. Ceilings live in
test/fixtures/context-budget.json with x1.05/x1.10 headroom; regenerate with
bun test/helpers/capture-context-budget.ts. New skills fail until consciously
budgeted; removed skills fail until the fixture is refreshed; reductions
ratchet the ceilings down so wins lock in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(todos): file output-template carve wave + plan-ceo doctrine revisit; mark preamble-carve P3 in flight

Two follow-ups deferred from the approved token-reduction program (CEO review
'NOT in scope' list), filed with full context per TODOS format. The existing
P3 preamble-carve entry gets a status update pointing at the program that
supersedes it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): review findings — Windows path normalization, full totals rebuild, ratchet coverage

Pre-landing review (5 specialists) found one critical: the ratchet test runs
in the curated Windows lane, where path.relative yields backslash skill names
that miss the test/ filter and mismatch every POSIX fixture key. Names are now
normalized once in buildRatchetBill (toPosixName) and the fixture filter is
tightened to test/fixtures/. All eight Bill.totals fields are rebuilt from the
filtered list (no fixture-polluted perInvocation/totalMd numbers for future
consumers). New coverage: Windows-separator normalization pins, a
captureContextBudget round-trip against tree-a (headroom math exact), a
stripFields regression pin (interactive/benefits-from absent from renders,
hooks/gbrain preserved), and the ceilings test no longer double-reports
stale-fixture entries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): adversarial findings — stable root key, symlink-alias dedupe, fixture-shape guard

Adversarial review (Claude subagent) verified the fixture's root-skill key was
the capture machine's checkout dirname: any non-gstack-named clone (every
Conductor worktree) failed the free suite, and the documented re-run-the-capture
recovery baked the local dirname into the committed fixture — silent corruption
through the tool's own protocol. The root skill is now pinned to ROOT_SKILL_KEY
('gstack', its frontmatter name). Symlink aliases are realpath-deduped (census
precedent): connect-chrome no longer gets its own ceiling, so Windows checkouts
that materialize the symlink as a plain file can't fail the stale-ceiling
set-equality test. New guards: fixture-shape validation (a string alwaysOnTotal
can no longer silently disable the ceiling), a mutation pin that the filter
shrinks the always-on ledger vs the raw bill, an alwaysOnTotal violation test
(the branch was load-bearing with only under-budget coverage), and an atomic
temp+rename fixture write. Fixture regenerated: 59 ceilings, alwaysOnTotal 6344.
Deferred with a TODO: anchoring transformFrontmatter's denylist strip to the
frontmatter block (latent, zero live collisions, pre-existing path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v1.69.1.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v1.69.1.0

CLAUDE.md: Token ceiling section documents the context-budget ratchet as
the third guard (test file, fixture, new-skill budgeting, capture command).
CONTRIBUTING.md: Tier 1 guard list gains a Context-budget ratchet bullet;
the Adding-a-new-skill checklist gains the budget-capture step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: pin exact guard semantics for the context-budget ratchet in CLAUDE.md

Doc-review finding: "a third enforced ceiling" undercounted the guard
family (skill-size-budget floors and parity ratios also watch these
ledgers, relatively). Rephrased to match the ratchet test's own header:
absolute ceilings vs relative floors/ratios.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): heaviest-skill claim matches the fixture (land-and-deploy edges review by 0.2%)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bin): gstack-skill-start + gstack-skill-end — the preamble runtime, consolidated

Absorbs the ~13KB of bash every tier-2+ SKILL.md inlined twice over (bootstrap
fence + artifacts-sync fence) and the skill-end telemetry/sync fences. Same
KEY: value STATUS-line contract the prose interprets, plus SKILL_START_PROTO
handshake (OV5), SESSION_ID/TEL_START echoes, GSTACK_HOME-normalized state
paths (EOV7), --parent-pid session identity (EOV5: $PPID inside the script is
the ephemeral tool-call shell), OV4 sanitization of passthrough output, and a
receipted daily artifacts pull (_receipted_git, brain-sync class, fail-closed).
Per-line || true error style throughout (F3) — a mid-script failure never drops
later STATUS lines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(gen): preamble resolvers emit a script invocation fence instead of inline bash

generate-preamble-bash: ~6.3KB fence -> 4-line gstack-skill-start invocation
(quoted-tilde pitfall handled: leading ~ interpolates through $HOME; env-var
hosts keep $GSTACK_BIN) + degraded-mode prose (F1/EOV8: safe defaults, consent
gates deferred-never-lost; OV5: proto rule). generate-brain-sync-block: ~6.8KB
bash -> interpretation prose + the privacy stop-gate (stays inline until
Phase 2's gated emission). generate-completion-status: telemetry fence -> one
gstack-skill-end call with SESSION_ID/TEL_START handoff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(gen): regenerate all skills + golden fixtures — inline preamble bash removed

Mechanical regen after the resolver change: −12,628 lines across 52 renders
(corpus 952K -> 806K render tokens; tier-2 skills −11-13KB each). Golden
per-host ship fixtures refreshed from the fresh claude/codex/factory renders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: skill-start contract suite + preamble A/B eval + touchfiles registration

test/gstack-skill-start.test.ts (11 free tests): STATUS-key contract vs the
prose (F2), per-host fence resolution shapes (E1), proto-first, OV4 marker
sanitization, --parent-pid identity, headless suppression, skill-end duration
math + pending cleanup. test/skill-e2e-preamble-script-ab.test.ts (gate tier,
OV7): inline-bash render (pinned from 29785978) vs script render with the
fence redirected at the worktree bin (EOV2 — hermetic evals otherwise resolve
the operator install and silently exercise degraded mode). 21 touchfiles dep
lists gain the two bin scripts (EOV9) so future script edits select the
preamble evals; selection-count pin updated 23->24.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: repin ~70 assertions to the script contract — every literal gets a successor

Assertions that pinned inline-bash internals (update-check guard, _SESSIONS
reaping, telemetry start/end blocks, routing probe, repo-strip producer,
first-task gating, EXPLAIN_LEVEL/QUESTION_TUNING echoes, #2499 jq scope
resolution, Issue-8 CONDUCTOR gate) now pin the same invariants in their new
home: bin/gstack-skill-start / bin/gstack-skill-end file content for script
internals, the invocation fence + interpretation prose for render-side
behavior. No assertion deleted without a successor; live-execution tests
(routing probe, brain-sync jq) run against script bytes unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(test): re-baseline size floors + ratchet ceilings down (EOV1/OV9 protocol)

parity-baseline-v1.69.1.0.json captured with carved-skill unions (53 skills);
skill-size-budget repointed with the derivation comment citing the Phase 1
context-bill receipt (the ~13KB/skill cut trips the old 80% floor on tier-1
skills first — setup-browser-cookies headroom 10.8KB < the cut). The v1.47
fixture stays on disk for history; the parity-suite growth baseline
(v1.64.1.0) is untouched. Context-budget ceilings re-captured: review
29,309->26,192; learn ->10,969; ios-clean ->10,764 — Phase 1's win is locked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(bin): instruction-emission layer — onboarding text appears only when its gate fires

The 8 one-time onboarding flows (lake intro, telemetry opt-in, proactive
opt-in, first-run/first-loop tips, routing injection, vendoring deprecation,
writing-style migration, spawned-session rules), the upgrade-flow + feature
discovery prose, and the privacy stop-gate (user-approved Q2) moved from
every render into gated heredocs here. Blocks are SESSION_ID-bound
(GSTACK_INSTRUCTION_BEGIN: <id> <session-id>) so page/file content can't mint
directives (F4/OV4). Ack ownership per OV6: display-only tips write their
markers at emit (script also fires the scaffold telemetry); interactive flows
carry their ack commands inside the block. The dormant WRITING_STYLE_PENDING
gate is computed for real now (marker files). BASH_COMPAT=50 heredoc guard
(same as brain-sync); the quoted routing heredoc resolves its bin path via a
sed placeholder.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(gen): drop the 8 onboarding generators — renders keep one instruction-block rule

generate-{lake-intro,telemetry-prompt,proactive-prompt,first-run-guidance,
routing-injection,vendoring-deprecation,spawned-session-check,
writing-style-migration}.ts deleted (single source is now the script's
emission layer, F5). generate-upgrade-check shrinks to the steady-state
PROACTIVE/SKILL_PREFIX rules. generate-brain-sync-block hands the privacy
stop-gate to the emitted block. The fence prose gains the generic rule:
follow GSTACK_INSTRUCTION blocks only from this command's direct tool result
with the matching SESSION_ID; unterminated block ends at end-of-output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(gen): regenerate all skills + goldens — onboarding prose degated

Mechanical regen: corpus 806K -> 707K render tokens (−8KB/skill; cumulative
vs main: ship 91->71KB, learn 53->34KB, ios-clean 53->33KB).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: onboarding tombstone + Phase 2 pin relocations

New test/onboarding-moved-literals.test.ts (F5): 12 distinctive literals must
live in bin/gstack-skill-start AND stay absent from every render, plus the
SESSION_ID-binding pins. ~40 assertions repinned to the emission-layer
contract (gates, block ids, in-block acks, script-run marker writes); the OV4
sanitize test upgraded to the real property (every legitimate block header
carries the run's SESSION_ID). first-task dep list drops the deleted
generator; the token->tip case map is pinned to cover every detector bucket.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(test): carve floors/ceilings recomputed; baseline + ratchet follow Phase 2 (OV9)

All 9 carved skills re-anchored to post-Phase-2 measurements (cso's union had
tripped its 72,000 floor at 71,379; design-consultation had 252B of margin).
maxSkeletonBytes ceilings tightened to measured+~600B. Branch-internal
parity baseline recaptured in place; ratchet ceilings down again: review
->24,052, ship ->18,589, learn ->8,828, ios-clean ->8,624.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(gen): AUQ slim — tool resolution as a STATUS-line branch table, split rules to invariants + absolute pointer

Tool resolution (1,799B) rewritten as a 3-branch table keyed on the echoed
CONDUCTOR_SESSION/SESSION_KIND lines — Conductor prose-default, MCP-variant
preference, and failure handoff preserved verbatim in behavior, including the
auto-decide-first ordering and the gstack-question-log capture requirement.
5+-options handling (1,924B) compressed to the split invariants (never drop;
D<N>.k shape; Include/Defer/Cut/Hold; question_id scheme with the never-ask
refusal) + the full-rule pointer. Both doc pointers now interpolate the
absolute install root (Codex outside-voice #7 convention) instead of the bare
'in the gstack repo'. Failure-fallback, Format, and self-check sections are
byte-identical — all 14 MANDATORY always-loaded pins pass with zero test
edits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(gen): regenerate all skills + goldens — AUQ slim

Mechanical regen: −1.3KB per tier-2+ skill (ship 69.9KB, learn 32.5KB).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(test): baseline + ratchet follow Phase 3 (OV9); OV8 evaluated — shrink floor stays

Branch-internal baseline recaptured; ratchet ceilings down again. OV8's
floor-retirement question, evaluated as planned after Phase 3: the 80% shrink
floor stays — it uniquely catches accidental body deletion in non-carved
skills BETWEEN ratchet recaptures, and the capture command has amortized the
fixture-refresh cost that motivated retiring it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(review): carve adversarial, plan-completion, and review-army into sections

The three resolver macros ship already carves as siblings now load on demand
for /review too: skeleton 100.2KB -> 55.0KB (-45%), union 93.4KB. Resolvers
stay the single source of truth (sections wrap the macros). Step 0/1, scope
drift, critical pass, confidence calibration, and fix-first stay always-loaded.
Fixtures and pins follow the moved content (codex-hardening wrapped-sites,
review-army E2E fixture builds skeleton+sections with an empty-fixture guard).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(codex): carve the three mutually exclusive modes into sections

Review/Challenge/Consult mode bodies (34.7KB where at most one ever runs)
load on demand: skeleton 81.0KB -> 55.2KB, union 1.04x the monolith. The mode
dispatch, filesystem boundary, and a new always-loaded 'Synthesis
recommendation (REQUIRED) — all modes' block stay skeleton-side (the AUQ
per-skill pins pass unchanged); the plan-file report + exit gate render after
the last section pointer per the gateAfterStop pattern.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(land-and-deploy): carve first-run validation, readiness gate, and merge/deploy into sections

The once-per-repo dry-run validation, the pre-merge readiness gate, and the
merge + deploy-strategy steps (37.8KB) load on demand: skeleton 91.1KB ->
55.7KB. Step 1.5 keeps its detection bash as the dispatch; the first-run
section's fingerprint-save block gained {{SLUG_EVAL}} so it is self-contained.
Zero content lost (line-coverage checked against HEAD).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ios): demote the four ios skills to preamble-tier 2 (Phase 5)

They never consume the tier-3 sections (repo-mode ownership, search-before-
building) but do fire AskUserQuestion, which tier >=2 provides — verified by
grep before the plan review. -2.2KB per skill. Render assertions pin the
demotion (tier-3 sections absent, AUQ format present).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(guards): register wave-1 carves; monolith invariants retire; baselines + ratchet follow

CARVE_GUARDS gains review/codex/land-and-deploy (12 carved skills total);
their MONOLITH_INVARIANTS entries retire (invariants now generate from the
registry, cso precedent). Touchfiles: carve-section-loading covers the three
new carves; the codex + land-and-deploy LLM-judge dep lists widen to their
sections. Regen + goldens + branch-internal baseline + ratchet ceilings
recaptured (review 24,052 -> skeleton-based ceiling; union floors hold).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(gen-skill-docs): review render pins read the carved union

The review carve's readSkillUnion conversions (same pattern its neighbor
carved-skill pins already use).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(autoplan): carve the four review phases + tasks aggregator into sections

Phase bodies (CEO/Design/Eng/DX consensus flows) and the Implementation Tasks
aggregator load on demand; Design and DX stay separate sections because each
is independently conditional on scope. Skeleton 83.7KB -> 58.7KB (-30%
always-loaded); the 6 decision principles, classification, sequencing, and
explicit skip-condition dispatch stay always-loaded. The chain E2E's
phase-complete markers now live only in sections, so its assertions double as
section-read proof (behavioral: external).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(spec): carve the post-confirmation gate-and-file tail into one section

Phases 1-4 are the turn-1 conversational spine — carving them would force the
Read on the first user message for zero real savings. The mechanical tail
(4.5/4.5a/4.5b redaction gates + Phase 5 filing + TTHW telemetry) fires only
after draft confirmation: a genuine lazy boundary, kept as ONE section so the
gh-issue-create bash can never load without the fail-closed redaction gate
that precedes it. Skeleton 65.4KB -> 50.7KB; all ~85 phase-structure
invariants migrated location-aware plus a new carve-shape suite (56 tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(setup-gbrain): carve the branch-exclusive install paths into sections

Brain-init (Paths 1/2/3/4 bodies), engine remediation, transcript gate, and
CLAUDE.md persist load on demand — at most one install route ever runs.
Skeleton 75.3KB -> 57.0KB; the Step 1 detect and Step 2 path dispatch stay
always-loaded. New buildSetupGbrainFixture helper gives the periodic E2Es
extract-don't-copy fixtures with a non-empty guard; the voyage-code-3 gate
counts scan the tmpl union (the third init site lives in engine-remediation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(guards): register wave-2 carves (15 carved skills); autoplan monolith retires; baselines follow

CARVE_GUARDS gains autoplan (behavioral: external via the chain eval), spec,
and setup-gbrain; autoplan's MONOLITH_INVARIANTS entry retires. Touchfiles:
setup-gbrain periodic dep lists gain the section tmpls + fixture helper; the
stale-brain-refs scan covers setup-gbrain/sections. Regen + goldens + branch
baseline + ratchet recaptured.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(qa): carve QA patterns + health rubric into on-demand sections (68→48KB skeleton)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(browse): carve full command list + snapshot flags into sections/command-list.md (39→27KB skeleton)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(retro): absorb inline git/awk metrics into bin/gstack-retro-metrics + carve report format

RETRO_METRICS_PROTO: 1 contract, local git reads only (fetch stays in the
skill prose), degraded path documented in the skeleton.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: register wave-3 carves (qa, browse, retro) — guards, touchfiles, pins, baselines

CARVE_GUARDS gains the three entries; qa's monolith invariant retires.
auq-format carve-safety now keys on the skeleton+sections union shipping
the AUQ block (first tier-1 carve: browse never renders it by design).
Baselines: parity v1.69.1.0 at 18 sectioned skills; ratchet recaptured.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): drop stale generate-lake-intro import (generator deleted in the emission-layer move)

Sol scope discipline stays pinned via the model overlay + completeness
section; the lake intro is now a single script-emitted blurb.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(office-hours): carve Phase 2A/2B into mode-exclusive sections (81→67KB skeleton)

A session runs exactly one mode, so a builder session never loads the
13KB startup diagnostic. Mode mapping and the vibe-shift upgrade rule
stay in the skeleton.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(design): carve UX doctrine + Pretext patterns into read-on-demand sections

design-html 57→49KB, design-shotgun 53→50KB. Sections wrap
{{UX_PRINCIPLES}} so scripts/resolvers/design.ts stays the source of
truth; the pretext-patterns STOP sits at the top of Step 3 so the read
provably precedes the Write.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: register wave-4 carves (office-hours ext, design-html, design-shotgun) — 20 carved skills

Both design entries carry requiredReads + loading-eval scenarios (D3A
condition). office-hours phase sections are mode-exclusive, so only the
always-reached design/handoff section is a deterministic requiredRead.
Baselines and ratchet recaptured.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: trim CLAUDE.md 66.4→44.9KB — verbatim moves to docs/, pointers stay inline

Moved: browser/sidebar/server internals, CHANGELOG release-summary format
spec, project tree, hermetic-E2E detail, slop-scan reference, OpenClaw
publishing. Kept inline: every hard behavioral rule (dist/ ban, redaction
scan-at-sink, egress receipts, bisect commits, eval detach, CHANGELOG
entry rules), the machine-managed GBrain block (byte-identical), and the
'## Deploying to the active skill' header with gbrain-refresh in range
(pinned by test/gbrain-refresh-install-render.test.ts). No voice rewrites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): seed onboarding markers into the hermetic child GSTACK_HOME

EOV7 made bin/gstack-skill-start honor GSTACK_HOME, so the operator-HOME
seeding in e2e-helpers.ts no longer reaches hermetic children — the
emission layer fired lake-intro/telemetry prompts that burned turns and
stalled PTY tests waiting on an answer (observed: plan-mode-no-op derailed
by the telemetry question). Onboarding-specific tests pin their own
GSTACK_HOME per-test, which merges over this seed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: raise carve-section-loading wall clock to 480s SDK / 540s bun

The heavy full-workflow scenarios satisfy their required section reads
inside 60s but need 300-450s to finish the report on slower sandboxes;
the 300s default read as a loading failure when the carve invariant held
(traces: plan-eng-review read its section at 8s, office-hours all three
at 24s, design-html both at 50s — all timed out mid-report).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): harden the skill-start trust boundary — review-army findings

Session ID gains a urandom suffix (block binding unforgeable by reflected
content); _sanitize also neutralizes spoofed SESSION_ID: lines; branch
names are charset-clamped before JSON embedding (skill-start + skill-end);
.brain-last-push reads first line only with a charset clamp; the artifacts
URL echo routes through _sanitize; the privacy consent gate fires in
interactive sessions only (spawned auto-choose could accept consent no
human gave — emission order is not a safety property); the daily pull gets
non-interactive + slow-network git guards and stamps only when the
receipted path ran; ~/.claude.json gets a grep pre-filter before the jq
parse.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(resolvers): question-log session_id becomes a substitution placeholder + stale-comment sweep

The question-log block bound $_SESSION_ID, a shell variable the
consolidated fence never sets — hook-less hosts logged empty session_id,
breaking /plan-tune per-session grouping. It now uses the same
substitute-from-the-skill-start-echoes contract as the telemetry block.
Also: retired the pre-Phase-2 stop-gate docstring, repointed the
gbrain-local-status cross-reference at the script's inline jq, dropped an
orphaned section comment, documented retro-metrics' suffix-only census.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: regenerate renders for the question-log placeholder; goldens + baselines follow

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: hermetic update-check, onboarding gate sequencing, seeding parity

The contract test's child did a live git ls-remote + curl to github.com on
every bun run test (update_check config now gates it off); the headless
test gets a fresh GSTACK_HOME so the suppression is actually exercised; a
new OV6 test drives the script three times to pin ack-at-emit and gate
sequencing; hermetic seeding covers the config-keyed privacy gate; the
EVALS_HERMETIC=0 debug seeding reaches marker parity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(ci): demote the preamble A/B to periodic (OV7) and add it to the periodic matrix

Post-Phase-3 demotion per the plan; the eval needs fetch-depth 0 (it git
shows a pre-Phase-1 sha), which only the periodic workflow provides — and
a static matrix entry so it can't silently never run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v1.70.0.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v1.70.0.0

ARCHITECTURE.md: the preamble section now describes the v1.70 runtime —
the rendered {{PREAMBLE}} block invokes bin/gstack-skill-start and reads
STATUS lines, gstack-skill-end logs telemetry, and one-time onboarding
text arrives as gated GSTACK_INSTRUCTION blocks instead of riding in
every render.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: doc-review fixes — repair moved-file links, drop unbacked session-count claim

docs/BROWSER_INTERNALS.md: the two ARCHITECTURE.md anchor links broke when
the section moved from repo-root CLAUDE.md into docs/ — now ../ARCHITECTURE.md.
ARCHITECTURE.md: the preamble's session-tracking item claimed an active-session
count and an "ELI16 mode" that no shipped code implements (the count
computation was deleted with the inline preamble); describe the real
touch-and-prune behavior instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): correct numeric claims against measured counts

50 of 62 installed skills dropped (fixture/alias entries have no preamble);
11 new carves + a deeper office-hours carve = 9→20; test counts match the
files (13 / 11 / 3 / 7).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: repoint the preamble-runtime version reference after the queue rebump (v1.71.0.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(e2e-design): widen the Aesthetic synonym set — vocabulary variance, not a regression

Both attempts in run 33090283032 produced judge-praised DESIGN.md files
phrased as 'design principles'/'design language' without any of the four
original literals; inputs were identical to the prior passing run
32899975845 (design-consultation untouched by the intervening merge).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): stage design-consultation's sections/ into the E2E fixture

The skill has been carved since v1.57.0.0 — the DESIGN.md structure
prescription (the AESTHETIC proposal template) lives in
sections/proposal-and-preview.md behind a STOP-read. The fixture only
copied SKILL.md, so the agent improvised structure from the skeleton and
the section-synonym check has been a coin flip since the carve (CI run
33090283032 trace shows 'no sections dir'; the local eval store has the
same failure on 2026-08-25 while that day's CI run passed on lucky
vocabulary).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-27 09:50:31 -07:00
committed by GitHub
co-authored by Claude Fable 5
parent a3749bfa4b
commit 394db326f2
227 changed files with 16814 additions and 34123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/skill-llm-eval.test.ts
+83 -34
View File
@@ -34,6 +34,38 @@ import {
// Eval result collector
const evalCollector = createEvalCollector('llm-judge');
/**
* Browse carve (token-reduction Phase 4): the '## Snapshot Flags' and
* '## Full Command List' reference blocks moved from browse/SKILL.md into the
* generated on-demand section browse/sections/command-list.md ('## Snapshot
* Flags' first, then '## Full Command List'). '## SETUP', '## Core QA
* Patterns', and '## CSS Inspector' stay in the skeleton. Non-empty guard:
* judging an empty slice would silently pass garbage to the judge.
*/
function readBrowseCommandSection(): string {
const p = path.join(ROOT, 'browse', 'sections', 'command-list.md');
const content = fs.readFileSync(p, 'utf-8');
if (!content.includes('## Snapshot Flags') || !content.includes('## Full Command List')) {
throw new Error(
`${p} is missing the expected headers — regenerate with: bun run gen:skill-docs`,
);
}
return content;
}
/** Slice a section out of the command-list section file, guarded non-empty. */
function sliceBrowseSection(startHeader: string, endHeader?: string): string {
const content = readBrowseCommandSection();
const start = content.indexOf(startHeader);
if (start < 0) throw new Error(`browse/sections/command-list.md: "${startHeader}" not found`);
const end = endHeader ? content.indexOf(endHeader) : -1;
const section = end > start ? content.slice(start, end) : content.slice(start);
if (section.trim().length < 200) {
throw new Error(`browse/sections/command-list.md slice at "${startHeader}" is empty/stub — regenerate with: bun run gen:skill-docs`);
}
return section;
}
// --- Diff-based test selection (LLM_JUDGE_TOUCHFILES, not the E2E table) ---
const selectedTests = computeDiffSelection(LLM_JUDGE_TOUCHFILES, 'LLM-judge');
@@ -53,10 +85,9 @@ describeIfSelected('LLM-as-judge quality evals', [
], () => {
testIfSelected('command reference table', async () => {
const t0 = Date.now();
// P2 (v1.2.0): the command reference moved from the root router to browse/SKILL.md.
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
const start = content.indexOf('## Full Command List');
const section = content.slice(start);
// Browse carve: the command reference lives in the generated on-demand
// section browse/sections/command-list.md now (read via non-empty guard).
const section = sliceBrowseSection('## Full Command List');
const scores = await judge('command reference table', section);
console.log('Command reference scores:', JSON.stringify(scores, null, 2));
@@ -82,11 +113,10 @@ describeIfSelected('LLM-as-judge quality evals', [
testIfSelected('snapshot flags reference', async () => {
const t0 = Date.now();
// P2 (v1.2.0): snapshot flags moved from the root router to browse/SKILL.md.
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
const start = content.indexOf('## Snapshot Flags');
const end = content.indexOf('## CSS Inspector');
const section = content.slice(start, end);
// Browse carve: snapshot flags live in browse/sections/command-list.md now,
// ordered before '## Full Command List' (the '## CSS Inspector' end boundary
// stayed in the skeleton).
const section = sliceBrowseSection('## Snapshot Flags', '## Full Command List');
const scores = await judge('snapshot flags reference', section);
console.log('Snapshot flags scores:', JSON.stringify(scores, null, 2));
@@ -109,9 +139,8 @@ describeIfSelected('LLM-as-judge quality evals', [
testIfSelected('browse/SKILL.md reference', async () => {
const t0 = Date.now();
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
const start = content.indexOf('## Snapshot Flags');
const section = content.slice(start);
// Browse carve: flags + commands are the whole generated section file.
const section = sliceBrowseSection('## Snapshot Flags');
const scores = await judge('browse skill reference (flags + commands)', section);
console.log('Browse SKILL.md scores:', JSON.stringify(scores, null, 2));
@@ -162,10 +191,8 @@ describeIfSelected('LLM-as-judge quality evals', [
testIfSelected('regression vs baseline', async () => {
const t0 = Date.now();
// P2 (v1.2.0): the command reference moved from the root router to browse/SKILL.md.
const generated = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
const genStart = generated.indexOf('## Full Command List');
const genSection = generated.slice(genStart);
// Browse carve: the command reference lives in browse/sections/command-list.md.
const genSection = sliceBrowseSection('## Full Command List');
const baseline = `## Command Reference
@@ -249,14 +276,38 @@ Scores are 1-5 overall quality.`,
// --- Part 7: QA skill quality evals (C6) ---
describeIfSelected('QA skill quality evals', ['qa/SKILL.md workflow', 'qa/SKILL.md health rubric', 'qa/SKILL.md anti-refusal'], () => {
const qaContent = fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
/**
* QA carve (token-reduction Phase 4): the '## Modes', '## Workflow',
* '## Health Score Rubric', '## Framework-Specific Guidance', and
* '## Important Rules' blocks moved from qa/SKILL.md into the generated
* on-demand section qa/sections/qa-patterns.md. Monolith-tolerant: falls back
* to the skeleton when the section file doesn't exist (pre-carve checkout).
*/
function readQaPatterns(): string {
const sectionPath = path.join(ROOT, 'qa', 'sections', 'qa-patterns.md');
return fs.existsSync(sectionPath)
? fs.readFileSync(sectionPath, 'utf-8')
: fs.readFileSync(path.join(ROOT, 'qa', 'SKILL.md'), 'utf-8');
}
/** Slice out of the qa-patterns section, guarded non-empty: judging an empty
* slice would silently pass garbage to the judge. */
function sliceQaPatterns(startHeader: string, endHeader?: string): string {
const content = readQaPatterns();
const start = content.indexOf(startHeader);
if (start < 0) throw new Error(`qa/sections/qa-patterns.md: "${startHeader}" not found — regenerate with: bun run gen:skill-docs`);
const end = endHeader ? content.indexOf(endHeader, start) : -1;
const section = end > start ? content.slice(start, end) : content.slice(start);
if (section.trim().length < 200) {
throw new Error(`qa/sections/qa-patterns.md slice at "${startHeader}" is empty/stub — regenerate with: bun run gen:skill-docs`);
}
return section;
}
describeIfSelected('QA skill quality evals', ['qa/SKILL.md workflow', 'qa/SKILL.md health rubric', 'qa/SKILL.md anti-refusal'], () => {
testIfSelected('qa/SKILL.md workflow', async () => {
const t0 = Date.now();
const start = qaContent.indexOf('## Workflow');
const end = qaContent.indexOf('## Health Score Rubric');
const section = qaContent.slice(start, end);
const section = sliceQaPatterns('## Workflow', '## Health Score Rubric');
const scores = await callJudge<JudgeScore>(`You are evaluating the quality of a QA testing workflow document for an AI coding agent.
@@ -297,8 +348,7 @@ ${section}`);
testIfSelected('qa/SKILL.md health rubric', async () => {
const t0 = Date.now();
const start = qaContent.indexOf('## Health Score Rubric');
const section = qaContent.slice(start);
const section = sliceQaPatterns('## Health Score Rubric');
const scores = await callJudge<JudgeScore>(`You are evaluating a health score rubric that an AI agent must follow to compute a numeric QA score.
@@ -338,13 +388,14 @@ ${section}`);
testIfSelected('qa/SKILL.md anti-refusal', async () => {
const t0 = Date.now();
// Extract both the diff-aware mode section and Important Rules section
const diffAwareStart = qaContent.indexOf('### Diff-aware');
const diffAwareEnd = qaContent.indexOf('### Full');
const rulesStart = qaContent.indexOf('## Important Rules');
const rulesEnd = qaContent.indexOf('## Framework-Specific');
const diffAwareSection = qaContent.slice(diffAwareStart, diffAwareEnd);
const rulesSection = qaContent.slice(rulesStart, rulesEnd);
// Extract both the diff-aware mode section and Important Rules section.
// (Pre-carve this sliced '## Important Rules' → '## Framework-Specific',
// which was EMPTY — Framework-Specific precedes Important Rules — so the
// judge only ever saw excerpt 1. The section-file slice fixes that: rules
// run to the end of qa-patterns.md, so rule 12 "Never refuse to use the
// browser" now actually reaches the judge.)
const diffAwareSection = sliceQaPatterns('### Diff-aware', '### Full');
const rulesSection = sliceQaPatterns('## Important Rules');
const result = await callJudge<{ would_browse: boolean; fallback_behavior: string; confidence: number; reasoning: string }>(`You are evaluating whether a QA testing skill document would cause an AI agent to USE THE BROWSER or REFUSE to use the browser in a specific scenario.
@@ -470,10 +521,8 @@ describeIfSelected('Baseline score pinning', ['baseline score pinning'], () => {
const baselines = JSON.parse(fs.readFileSync(baselinesPath, 'utf-8'));
const regressions: string[] = [];
// P2 (v1.2.0): the command reference moved from the root router to browse/SKILL.md.
const skillContent = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
const cmdStart = skillContent.indexOf('## Full Command List');
const cmdSection = skillContent.slice(cmdStart);
// Browse carve: the command reference lives in browse/sections/command-list.md.
const cmdSection = sliceBrowseSection('## Full Command List');
const cmdScores = await judge('command reference table', cmdSection);
for (const dim of ['clarity', 'completeness', 'actionability'] as const) {