Files
Garry Tan 22a4451e0e feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040)
* chore: regenerate stale ship golden fixtures

Golden fixtures were missing the VENDORED_GSTACK preamble section that
landed on main. Regression tests failed on all three hosts (claude, codex,
factory). Regenerated from current preamble output.

No code changes, unblocks test suite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: anti-slop design constraints + delete duplicate constants

Tightens design-consultation and design-shotgun to push back on the
convergence traps every AI design tool falls into.

Changes:
- scripts/resolvers/constants.ts: add "system-ui as primary font" to
  AI_SLOP_BLACKLIST. Document Space Grotesk as the new "safe alternative
  to Inter" convergence trap alongside the existing overused fonts.
- scripts/gen-skill-docs.ts: delete duplicate AI slop constants block
  (dead code — scripts/resolvers/constants.ts is the live source).
  Prevents drift between the two definitions.
- design-consultation/SKILL.md.tmpl: add Space Grotesk + system-ui to
  overused/slop lists. Add "anti-convergence directive" — vary across
  generations in the same project. Add Phase 1 "memorable-thing forcing
  question" (what's the one thing someone will remember?). Add Phase 5
  "would a human designer be embarrassed by this?" self-gate before
  presenting variants.
- design-shotgun/SKILL.md.tmpl: anti-convergence directive — each
  variant must use a different font, palette, and layout. If two
  variants look like siblings, one of them failed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: context health soft directive in preamble (T2+)

Adds a "periodically self-summarize" nudge to long-running skills.
Soft directive only — no thresholds, no enforcement, no auto-commit.

Goal: self-awareness during /qa, /investigate, /cso etc. If you notice
yourself going in circles, STOP and reassess instead of thrashing.

Codex review caught that fake precision thresholds (15/30/45 tool calls)
were unimplementable — SKILL.md is a static prompt, not runtime code.
This ships the soft version only.

Changes:
- scripts/resolvers/preamble.ts: add generateContextHealth(), wire into
  T2+ tier. Format: [PROGRESS] ... summary line. Explicit rule that
  progress reporting must never mutate git state.
- All T2+ skill SKILL.md files regenerated to include the new section.
- Golden ship fixtures updated (T4 skill, picks up the change).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: model overlays with explicit --model flag (no auto-detect)

Adds a per-model behavioral patch layer orthogonal to the host axis.
Different LLMs have different tendencies (GPT won't stop, Gemini
over-explains, o-series wants structured output). Overlays nudge each
model toward better defaults for gstack workflows.

Codex review caught three landmines the prior reviews missed:
1. Host != model — Claude Code can run any Claude model, Codex runs
   GPT/o-series, Cursor fronts multiple providers. Auto-detecting from
   host would lie. Dropped auto-detect. --model is explicit (default
   claude). Missing overlay file → empty string (graceful).
2. Import cycle — putting Model in resolvers/types.ts would cycle
   through hosts/index. Created neutral scripts/models.ts instead.
3. "Final say" is dangerous — overlay at the end of preamble could
   override STOP points, AskUserQuestion gates, /ship review gates.
   Placed overlay after spawned-session-check but before voice + tier
   sections. Wrapper heading adds explicit subordination language on
   every overlay: "subordinate to skill workflow, STOP points,
   AskUserQuestion gates, plan-mode safety, and /ship review gates."

Changes:
- scripts/models.ts: new neutral module. ALL_MODEL_NAMES, Model type,
  resolveModel() for family heuristics (gpt-5.4-mini → gpt-5.4, o3 →
  o-series, claude-opus-4-7 → claude), validateModel() helper.
- scripts/resolvers/types.ts: import Model, add ctx.model field.
- scripts/resolvers/model-overlay.ts: new resolver. Reads
  model-overlays/{model}.md. Supports {{INHERIT:base}} directive at
  top of file for concat (gpt-5.4 inherits gpt). Cycle guard.
- scripts/resolvers/index.ts: register MODEL_OVERLAY resolver.
- scripts/resolvers/preamble.ts: wire generateModelOverlay into
  composition before voice. Print MODEL_OVERLAY: {model} in preamble
  bash so users can see which overlay is active. Filter empty sections.
- scripts/gen-skill-docs.ts: parse --model CLI flag. Default claude.
  Unknown model → throw with list of valid options.
- model-overlays/{claude,gpt,gpt-5.4,gemini,o-series}.md: behavioral
  patches per model family. gpt-5.4.md uses {{INHERIT:gpt}} to extend
  gpt.md without duplication.
- test/gen-skill-docs.test.ts: fix qa-only guardrail regex scope.
  Was matching Edit/Glob/Grep anywhere after `allowed-tools:` in the
  whole file. Now scoped to frontmatter only. Body prose (Claude
  overlay references Edit as a tool) correctly no longer breaks it.

Verification:
- bun run gen:skill-docs --host all --dry-run → all fresh
- bun run gen:skill-docs --model gpt-5.4 → concat works, gpt.md +
  gpt-5.4.md content appears in order
- bun run gen:skill-docs --model unknown → errors with valid list
- All generated skills contain MODEL_OVERLAY: claude in preamble
- Golden ship fixtures regenerated

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: continuous checkpoint mode with non-destructive WIP squash

Adds opt-in auto-commit during long sessions so work survives Claude
Code crashes, Conductor workspace handoffs, and context switches.
Local-only by default — pushing requires explicit opt-in.

Codex review caught multiple landmines that would have shipped:
1. checkpoint_push=true default would push WIP commits to shared
   branches, trigger CI/deploys, expose secrets. Now default false.
2. Plan's original /ship squash (git reset --soft to merge base) was
   destructive — uncommitted ALL branch commits, not just WIP, and
   caused non-fast-forward pushes. Redesigned: rebase --autosquash
   scoped to WIP commits only, with explicit fallback for WIP-only
   branches and STOP-and-ask for conflicts.
3. gstack-config get returned empty for missing keys with exit 0,
   ignoring the annotated defaults in the header comments. Fixed:
   get now falls back to a lookup_default() table that is the
   canonical source for defaults.
4. Telemetry default mismatched: header said 'anonymous' but runtime
   treated empty as 'off'. Aligned: default is 'off' everywhere.
5. /checkpoint resume only read markdown checkpoint files, not the
   WIP commit [gstack-context] bodies the plan referenced. Wired up
   parsing of [gstack-context] blocks from WIP commits as a second
   recovery trail alongside the markdown checkpoints.

Changes:
- bin/gstack-config: add checkpoint_mode (default explicit) and
  checkpoint_push (default false) to CONFIG_HEADER. Add lookup_default()
  as canonical default source. get() falls back to defaults when key
  absent. list now shows value + source (set/default). New 'defaults'
  subcommand to inspect the table.
- scripts/resolvers/preamble.ts: preamble bash reads _CHECKPOINT_MODE
  and _CHECKPOINT_PUSH, prints CHECKPOINT_MODE: and CHECKPOINT_PUSH: so
  the mode is visible. New generateContinuousCheckpoint() section in
  T2+ tier describes WIP commit format with [gstack-context] body and
  the rules (never git add -A, never commit broken tests, push only
  if opted in). Example deliberately shows a clean-state context so
  it doesn't contradict the rules.
- ship/SKILL.md.tmpl: new Step 5.75 WIP Commit Squash. Detects WIP
  count, exports [gstack-context] blocks before squash (as backup),
  uses rebase --autosquash for mixed branches and soft-reset only when
  VERIFIED WIP-only. Explicit anti-footgun rules against blind soft-
  reset. Aborts with BLOCKED status on conflict instead of destroying
  non-WIP commits.
- checkpoint/SKILL.md.tmpl: new Step 1.5 to parse [gstack-context]
  blocks from WIP commits via git log --grep="^WIP:". Merges with
  markdown checkpoint for fuller session recovery.
- Golden ship fixtures regenerated (ship is T4, preamble change shows up).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: feature discovery flow gated by per-feature markers

Extends generateUpgradeCheck() to surface new features once per user
after a just-upgraded session. No more silent features.

Codex review caught: spawned sessions (OpenClaw, etc.) must skip the
discovery prompt entirely — they can't interactively answer. Feature
discovery now checks SPAWNED_SESSION first and is silent in those.

Discovery is per-feature, not per-upgrade. Each feature has its own
marker file at ~/.claude/skills/gstack/.feature-prompted-{name}. Once
the user has been shown a feature (accepted, shown docs, or skipped),
the marker is touched and the prompt never fires again for that
feature. Future features get their own markers.

V1 features surfaced:
- continuous-checkpoint: offer to enable checkpoint_mode=continuous
- model-overlay: inform-only note about --model flag and MODEL_OVERLAY
  line in preamble output

Max one prompt per session to avoid nagging. Fires only on JUST_UPGRADED
(not every session), plus spawned-session skip.

Changes:
- scripts/resolvers/preamble.ts: extend generateUpgradeCheck() with
  feature discovery rules, per-marker-file semantics, spawned-session
  exclusion, and max-one-per-session cap.
- All skill SKILL.md files regenerated to include the new section.
- Golden ship fixtures regenerated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: design taste engine with persistent schema

Adds a cross-session taste profile that learns from design-shotgun
approval/rejection decisions. Biases future design-consultation and
design-shotgun proposals toward the user's demonstrated preferences.

Codex review caught that the plan had "taste engine" as a vague goal
without schema, decay, migration, or placeholder insertion points. This
commit ships the full spec.

Schema v1 at ~/.gstack/projects/$SLUG/taste-profile.json:
- version, updated_at
- dimensions: fonts, colors, layouts, aesthetics — each with approved[]
  and rejected[] preference lists
- sessions: last 50 (FIFO truncation), each with ts/action/variant/reason
- Preference: { value, confidence, approved_count, rejected_count, last_seen }
- Confidence: Laplace-smoothed approved/(total+1)
- Decay: 5% per week of inactivity, computed at read time (not write)

Changes:
- bin/gstack-taste-update: new CLI. Subcommands approved/rejected/show/
  migrate. Parses reason string for dimension signals (e.g.,
  "fonts: Geist; colors: slate; aesthetics: minimal"). Emits taste-drift
  NOTE when a new signal contradicts a strong opposing signal. Legacy
  approved.json aggregates migrate to v1 on next write.
- scripts/resolvers/design.ts: new generateTasteProfile() resolver.
  Produces the prose that skills see: how to read the profile, how to
  factor into proposals, conflict handling, schema migration.
- scripts/resolvers/index.ts: register TASTE_PROFILE and a BIN_DIR
  resolver (returns ctx.paths.binDir, used by templates that shell out
  to gstack-* binaries).
- design-consultation/SKILL.md.tmpl: insert {{TASTE_PROFILE}} placeholder
  in Phase 1 right after the memorable-thing forcing question so the
  Phase 3 proposal can factor in learned preferences.
- design-shotgun/SKILL.md.tmpl: taste memory section now reads
  taste-profile.json via {{TASTE_PROFILE}}, falls back to per-session
  approved.json (legacy). Approval flow documented to call
  gstack-taste-update after user picks/rejects a variant.

Known gap: v1 extracts dimension signals from a reason string passed
by the caller ("fonts: X; colors: Y"). Future v2 can read EXIF or an
accompanying manifest written by design-shotgun alongside each variant
for automatic dimension extraction without needing the reason argument.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: multi-provider model benchmark (boil the ocean)

Adds the full spec Codex asked for: real provider adapters with auth
detection, normalized RunResult, pricing tables, tool compatibility
maps, parallel execution with error isolation, and table/JSON/markdown
output. Judge stays on Anthropic SDK as the single stable source of
quality scoring, gated behind --judge.

Codex flagged the original plan as massively under-scoped — the
existing runner is Claude-only and the judge is Anthropic-only. You
can't benchmark GPT or Gemini without real provider infrastructure.
This commit ships it.

New architecture:

  test/helpers/providers/types.ts       ProviderAdapter interface
  test/helpers/providers/claude.ts      wraps `claude -p --output-format json`
  test/helpers/providers/gpt.ts         wraps `codex exec --json`
  test/helpers/providers/gemini.ts      wraps `gemini -p --output-format stream-json --yolo`
  test/helpers/pricing.ts               per-model USD cost tables (quarterly)
  test/helpers/tool-map.ts              which tools each CLI exposes
  test/helpers/benchmark-runner.ts      orchestrator (Promise.allSettled)
  test/helpers/benchmark-judge.ts       Anthropic SDK quality scorer
  bin/gstack-model-benchmark            CLI entry
  test/benchmark-runner.test.ts         9 unit tests (cost math, formatters, tool-map)

Per-provider error isolation:
  - auth → record reason, don't abort batch
  - timeout → record reason, don't abort batch
  - rate_limit → record reason, don't abort batch
  - binary_missing → record in available() check, skip if --skip-unavailable

Pricing correction: cached input tokens are disjoint from uncached
input tokens (Anthropic/OpenAI report them separately). Original
math subtracted them, producing negative costs. Now adds cached at
the 10% discount alongside the full uncached input cost.

CLI:
  gstack-model-benchmark --prompt "..." --models claude,gpt,gemini
  gstack-model-benchmark ./prompt.txt --output json --judge
  gstack-model-benchmark ./prompt.txt --models claude --timeout-ms 60000

Output formats: table (default), json, markdown. Each shows model,
latency, in→out tokens, cost, quality (when --judge used), tool calls,
and any errors.

Known limitations for v1:
- Claude adapter approximates toolCalls as num_turns (stream-json
  would give exact counts; v2 can upgrade).
- Live E2E tests (test/providers.e2e.test.ts) not included — they
  require CI secrets for all three providers. Unit tests cover the
  shape and math.
- Provider CLIs sometimes return non-JSON error text to stdout; the
  parsers fall back to treating raw output as plain text in that case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: standalone methodology skill publishing via gstack-publish

Ships the marketplace-distribution half of Item 5 (reframed): publish
the existing standalone OpenClaw methodology skills to multiple
marketplaces with one command.

Codex review caught that the original plan assumed raw generated
multi-host skills could be published directly. They can't — those
depend on gstack binaries, generated host paths, tool names, and
telemetry. The correct artifact class is hand-crafted standalone
skills in openclaw/skills/gstack-openclaw-* (already exist and work
without gstack runtime). This commit adds the wrapper that publishes
them to ClawHub + SkillsMP + Vercel Skills.sh with per-marketplace
error isolation and dry-run validation.

Changes:
- skills.json: root manifest with 4 skills (office-hours, ceo-review,
  investigate, retro) each pointing at its openclaw/skills source.
  Each skill declares per-marketplace targets with a slug, a publish
  flag, and a compatible-hosts list. Marketplace configs include CLI
  name, login command, publish command template (with placeholder
  substitution), docs URL, and auth_check command.
- bin/gstack-publish: new CLI. Subcommands:
    gstack-publish              Publish all skills
    gstack-publish <slug>       Publish one skill
    gstack-publish --dry-run    Validate + auth-check without publishing
    gstack-publish --list       List skills + marketplace targets
  Features:
    * Manifest validation (missing source files, missing slugs, empty
      marketplace list all reported).
    * Per-marketplace auth check before any publish attempt.
    * Per-skill / per-marketplace error isolation: one failure doesn't
      abort the batch.
    * Idempotent — re-running with the same version is safe; markets
      that reject duplicate versions report it as a failure for that
      single target without affecting others.
    * --dry-run walks the full pipeline but skips execSync; useful in
      CI to validate manifest before bumping version.

Tested locally: clawhub auth detected, skillsmp/vercel CLIs not
installed (marked NOT READY and skipped cleanly in dry-run).

Follow-up work (tracked in TODOS.md later):
- Version-bump helper that reads openclaw/skills/*/SKILL.md frontmatter
  and updates skills.json in lockstep.
- CI workflow that runs gstack-publish --dry-run on every PR and
  gstack-publish on tags.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: split preamble.ts into submodules (byte-identical output)

Splits scripts/resolvers/preamble.ts (841 lines, 18 generator functions +
composition root) into one file per generator under
scripts/resolvers/preamble/. Root preamble.ts becomes a thin composition
layer (~80 lines of imports + generatePreamble).

Before:
  scripts/resolvers/preamble.ts  841 lines

After:
  scripts/resolvers/preamble.ts                                   83 lines
  scripts/resolvers/preamble/generate-preamble-bash.ts            97 lines
  scripts/resolvers/preamble/generate-upgrade-check.ts            48 lines
  scripts/resolvers/preamble/generate-lake-intro.ts               16 lines
  scripts/resolvers/preamble/generate-telemetry-prompt.ts         37 lines
  scripts/resolvers/preamble/generate-proactive-prompt.ts         25 lines
  scripts/resolvers/preamble/generate-routing-injection.ts        49 lines
  scripts/resolvers/preamble/generate-vendoring-deprecation.ts    36 lines
  scripts/resolvers/preamble/generate-spawned-session-check.ts    11 lines
  scripts/resolvers/preamble/generate-ask-user-format.ts          16 lines
  scripts/resolvers/preamble/generate-completeness-section.ts     19 lines
  scripts/resolvers/preamble/generate-repo-mode-section.ts        12 lines
  scripts/resolvers/preamble/generate-test-failure-triage.ts     108 lines
  scripts/resolvers/preamble/generate-search-before-building.ts   14 lines
  scripts/resolvers/preamble/generate-completion-status.ts       161 lines
  scripts/resolvers/preamble/generate-voice-directive.ts          60 lines
  scripts/resolvers/preamble/generate-context-recovery.ts         51 lines
  scripts/resolvers/preamble/generate-continuous-checkpoint.ts    48 lines
  scripts/resolvers/preamble/generate-context-health.ts           31 lines

Byte-identity verification (the real gate per Codex correction):
- Before refactor: snapshotted 135 generated SKILL.md files via
  `find -name SKILL.md -type f | grep -v /gstack/` across all hosts.
- After refactor: regenerated with `bun run gen:skill-docs --host all`
  and re-snapshotted.
- `diff -r baseline after` returned zero differences and exit 0.

The `--host all --dry-run` gate passes too. No template or host behavior
changes — purely a code-organization refactor.

Test fix: audit-compliance.test.ts's telemetry check previously grepped
preamble.ts directly for `_TEL != "off"`. After the refactor that logic
lives in preamble/generate-preamble-bash.ts. Test now concatenates all
preamble submodule sources before asserting — tracks the semantic contract,
not the file layout. Doing the minimum rewrite preserves the test's intent
(conditional telemetry) without coupling it to file boundaries.

Why now: we were in-session with full context. Codex had downgraded this
from mandatory to optional, but the preamble had grown to 841 lines and
was getting harder to navigate. User asked "why not?" given the context
was hot. Shipping it as a clean bisectable commit while all the prior
preamble.ts changes are fresh reduces rebase pain later.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.19.0.0)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: trim verbose preamble + coverage audit prose

Compress without removing behavior or voice. Three targeted cuts:

1. scripts/resolvers/testing.ts coverage diagram example: 40 lines → 14
   lines. Two-column ASCII layout instead of stacked sections.
   Preserves all required regression-guard phrases (processPayment,
   refundPayment, billing.test.ts, checkout.e2e.ts, COVERAGE, QUALITY,
   GAPS, Code paths, User flows, ASCII coverage diagram).

2. scripts/resolvers/preamble/generate-completion-status.ts Plan Status
   Footer: was 35 lines with embedded markdown table example, now 7
   lines that describe the table inline. The footer fires only at
   ExitPlanMode time — Claude can construct the placeholder table from
   the inline description without copying a literal example.

3. Same file's Plan Mode Safe Operations + Skill Invocation During Plan
   Mode sections compressed from ~25 lines combined to ~12. Preserves
   all required test phrases (precedence over generic plan mode behavior,
   Do not continue the workflow, cancel the skill or leave plan mode,
   PLAN MODE EXCEPTION).

NOT touched:
- Voice directive (Garry's voice — protected per CLAUDE.md)
- Office-hours Phase 6 Handoff (Garry's voice + YC pitch)
- Test bootstrap, review army, plan completion (carefully tuned behavior)

Token savings (per skill, system-wide):
  ship/SKILL.md           35474 → 34992 tokens (-482)
  plan-ceo-review         29436 → 28940 (-496)
  office-hours            26700 → 26204 (-496)

Still over the 25K ceiling. Bigger reduction requires restructure
(move large resolvers to externally-referenced docs, split /ship into
ship-quick + ship-full, or refactor the coverage audit + review army
into shorter prose). That's a follow-up — added to TODOS.

Tests: 420/420 pass on gen-skill-docs.test.ts + host-config.test.ts.
Goldens regenerated for claude/codex/factory ship.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): install Node.js from official tarball instead of NodeSource apt setup

The CI Dockerfile's Node install was failing on ubicloud runners. NodeSource's
setup_22.x script runs two internal apt operations that both depend on
archive.ubuntu.com + security.ubuntu.com being reachable:
1. apt-get update (to refresh package lists)
2. apt-get install gnupg (as a prerequisite for its gpg keyring)

Ubicloud's CI runners frequently can't reach those mirrors — last build hit
~2min of connection timeouts to every security.ubuntu.com IP (185.125.190.82,
91.189.91.83, 91.189.92.24, etc.) plus archive.ubuntu.com mirrors. Compounding
this: on Ubuntu 24.04 (noble) "gnupg" was renamed to "gpg" and "gpgconf".
NodeSource's setup script still looks for "gnupg", so even when apt works,
it fails with "Package 'gnupg' has no installation candidate." The subsequent
apt-get install nodejs then fails because the NodeSource repo was never added.

Fix: drop NodeSource entirely. Download Node.js v22.20.0 from nodejs.org as a
tarball, extract to /usr/local. One host, no apt, no script, no keyring.

Before:
  RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
      && apt-get install -y --no-install-recommends nodejs ...

After:
  ENV NODE_VERSION=22.20.0
  RUN curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz" -o /tmp/node.tar.xz \
      && tar -xJ -C /usr/local --strip-components=1 --no-same-owner -f /tmp/node.tar.xz \
      && rm -f /tmp/node.tar.xz \
      && node --version && npm --version

Same installed path (/usr/local/bin/node and npm). Pinned version for
reproducibility. Version is bump-visible in the Dockerfile now.

Does not address the separate apt flakiness that affects the GitHub CLI
install (line 17) or `npx playwright install-deps chromium` (line 33) —
those use apt too. If those fail on a future build we can address then.

Failing job: build-image (71777913820)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: raise skill token ceiling warning from 25K to 40K

The 25K ceiling predated flagship models with 200K-1M windows and assumed
every skill prompt dominates context cost. Modern reality: prompt caching
amortizes the skill load across invocations, and three carefully-tuned
skills (ship, plan-ceo-review, office-hours) legitimately pack 25-35K
tokens of behavior that can't be cut without degrading quality or removing
protected content (Garry's voice, YC pitch, specialist review instructions).

We made the safe prose cuts earlier (coverage diagram, plan status footer,
plan mode operations). The remaining gap is structural — real compression
would require splitting /ship into ship-quick vs ship-full, externalizing
large resolvers to reference docs, or removing detailed skill behavior.
Each is 1-2 days of work. The cost of the warning firing is zero (it's
a warning, not an error). The cost of hitting it is ~15¢ per invocation
at worst, amortized further by prompt caching.

Raising to 40K catches what it's supposed to catch — a runaway 10K+ token
growth in a single release — without crying wolf on legitimately big
skills. Reference doc in CLAUDE.md updated to reflect the new philosophy:
when you hit 40K, ask WHAT grew, don't blindly compress tuned prose.

scripts/gen-skill-docs.ts: TOKEN_CEILING_BYTES 100_000 → 160_000.
CLAUDE.md: document the "watch for feature bloat, not force compression"
intent of the ceiling.

Verification: `bun run gen:skill-docs --host all` shows zero TOKEN
CEILING warnings under the new 40K threshold.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): install xz-utils so Node tarball extraction works

The direct-tarball Node install (switched from NodeSource apt in the last
CI fix) failed with "xz: Cannot exec: No such file or directory" because
Ubuntu 24.04 base doesn't include xz-utils. Node ships .tar.xz by default,
and `tar -xJ` shells out to xz, which was missing.

Add xz-utils to the base apt install alongside git/curl/unzip/etc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(benchmark): pass --skip-git-repo-check to codex adapter

The gpt provider adapter spawns `codex exec -C <workdir>` with arbitrary
working directories (benchmark temp dirs, non-git paths). Without
`--skip-git-repo-check`, codex refuses to run and returns "Not inside a
trusted directory" — surfaced as a generic error.code='unknown' that
looks like an API failure.

Benchmarks don't care about codex's git-repo trust model; we just want
the prompt executed. Surfaced by the new provider live E2E test on a
temp workdir.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(benchmark): add --dry-run flag to gstack-model-benchmark

Matches gstack-publish --dry-run semantics. Validates the provider list,
resolves per-adapter auth, echoes the resolved flag values, and exits
without invoking any provider CLI. Zero-cost pre-flight for CI pipelines
and for catching auth drift before starting a paid benchmark run.

Output shape:
  == gstack-model-benchmark --dry-run ==
    prompt:     <truncated>
    providers:  claude, gpt, gemini
    workdir:    /tmp/...
    timeout_ms: 300000
    output:     table
    judge:      off

  Adapter availability:
    claude: OK
    gpt:    NOT READY — <reason>
    gemini: NOT READY — <reason>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: lite E2E coverage for benchmark, taste engine, publish

Fills real coverage gaps in v0.19.0.0 primitives. 44 new deterministic
tests (gate tier, ~3s) + 8 live-API tests (periodic tier).

New gate-tier test files (free, <3s total):
- test/taste-engine.test.ts — 24 tests against gstack-taste-update:
  schema shape, Laplace-smoothed confidence, 5%/week decay clamped at 0,
  multi-dimension extraction, case-insensitive matching, session cap,
  legacy profile migration with session truncation, taste-drift conflict
  warning, malformed-JSON recovery, missing-variant exit code.
- test/publish-dry-run.test.ts — 13 tests against gstack-publish --dry-run:
  manifest parsing, missing/malformed JSON, per-skill validation errors
  (missing source file / slug / version / marketplaces), slug filter,
  unknown-skill exit, per-marketplace auth isolation (fake marketplaces
  with always-pass / always-fail / missing-binary CLIs), and a sanity
  check against the real repo manifest.
- test/benchmark-cli.test.ts — 11 tests against gstack-model-benchmark
  --dry-run: provider default, unknown-provider WARN, empty list
  fallback, flag passthrough (timeout/workdir/judge/output), long-prompt
  truncation, prompt resolution (inline vs file vs positional), missing
  prompt exit.

New periodic-tier test file (paid, gated EVALS=1):
- test/skill-e2e-benchmark-providers.test.ts — 8 tests hitting real
  claude, codex, gemini CLIs with a trivial prompt (~$0.001/provider).
  Verifies output parsing, token accounting, cost estimation, timeout
  error.code semantics, Promise.allSettled parallel isolation.
  Per-provider availability gate — unauthed providers skip cleanly.

This suite already caught one real bug (codex adapter missing
--skip-git-repo-check, fixed in 5260987d).

Registered `benchmark-providers-live` in touchfiles.ts (periodic tier,
triggered by changes to bin/gstack-model-benchmark, providers/**,
benchmark-runner.ts, pricing.ts).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(benchmark): dedupe providers in --models

`--models claude,claude,gpt` previously produced a list with a duplicate
entry, meaning the benchmark would run claude twice and bill for two
runs. Surfaced by /review on this branch.

Use a Set internally; return Array.from(seen) to preserve type + order
of first occurrence.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: /review hardening — NOT-READY env isolation, workdir cleanup, perf

Applied from the adversarial subagent pass during /review on this branch:

- test/benchmark-cli.test.ts — new "NOT READY path fires when auth env
  vars are stripped" test. The default dry-run test always showed OK on
  dev machines with auth, hiding regressions in the remediation-hint
  branch. Stripped env (no auth vars, HOME→empty tmpdir) now force-
  exercises gpt + gemini NOT READY paths and asserts every NOT READY
  line includes a concrete remediation hint (install/login/export).
  (claude adapter's os.homedir() call is Bun-cached; the 2-of-3 adapter
  coverage is sufficient to exercise the branch.)

- test/taste-engine.test.ts — session-cap test rewritten to seed the
  profile with 50 entries + one real CLI call, instead of 55 sequential
  subprocess spawns. Same coverage (FIFO eviction at the boundary), ~5s
  faster CI time. Also pins first-casing-wins on the Geist/GEIST merge
  assertion — bumpPref() keeps the first-arrival casing, so the test
  documents that policy.

- test/skill-e2e-benchmark-providers.test.ts — workdir creation moved
  from module-load into beforeAll, cleanup added in afterAll. Previous
  shape leaked a /tmp/bench-e2e-* dir every CI run.

- test/publish-dry-run.test.ts — removed unused empty test/helpers
  mkdirSync from the sandbox setup. The bin doesn't import from there,
  so the empty dir was a footgun for future maintainers.

- test/helpers/providers/gpt.ts — expanded the inline comment on
  `--skip-git-repo-check` to explicitly note that `-s read-only` is now
  load-bearing safety (the trust prompt was the secondary boundary;
  removing read-only while keeping skip-git-repo-check would be unsafe).

Net: 45 passing tests (was 44), session-cap test 5s faster, one real
regression surface covered that didn't exist before.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: surface v0.19 binaries and continuous checkpoint in README

The /review doc-staleness check flagged that v0.19.0.0 ships three new CLIs
(gstack-model-benchmark, gstack-publish, gstack-taste-update) and an opt-in
continuous checkpoint mode, none of which were visible in README's Power
tools section. New users couldn't find them without reading CHANGELOG.

Added:
- "New binaries (v0.19)" subsection with one-row descriptions for each CLI
- "Continuous checkpoint mode (opt-in, local by default)" subsection
  explaining WIP auto-commit + [gstack-context] body + /ship squash +
  /checkpoint resume

CHANGELOG entry already has good voice from /ship; no polish needed.
VERSION already at 0.19.0.0. Other docs (ARCHITECTURE/CONTRIBUTING/BROWSER)
don't reference this surface — scoped intentionally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ship): Step 19.5 — offer gstack-publish for methodology skill changes

Wires the orphaned gstack-publish binary into /ship. When a PR touches
any standalone methodology skill (openclaw/skills/gstack-*/SKILL.md) or
skills.json, /ship now runs gstack-publish --dry-run after PR creation
and asks the user if they want to actually publish.

Previously, the only way to discover gstack-publish was reading the
CHANGELOG or README. Most methodology skill updates landed on main
without ever being pushed to ClawHub / SkillsMP / Vercel Skills.sh,
defeating the whole point of having a marketplace publisher.

The check is conditional — for PRs that don't touch methodology skills
(the common case), this step is a silent no-op. Dry-run runs first so
the user sees the full list of what would publish and which marketplaces
are authed before committing.

Golden fixtures (claude/codex/factory) regenerated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(benchmark-models): new skill wrapping gstack-model-benchmark

Wires the orphaned gstack-model-benchmark binary into a dedicated skill
so users can discover cross-model benchmarking via /benchmark-models or
voice triggers ("compare models", "which model is best").

Deliberately separate from /benchmark (page performance) because the
two surfaces test completely different things — confusing them would
muddy both.

Flow:
  1. Pick a prompt (an existing SKILL.md file, inline text, or file path)
  2. Confirm providers (dry-run shows auth status per provider)
  3. Decide on --judge (adds ~$0.05, scores output quality 0-10)
  4. Run the benchmark — table output
  5. Interpret results (fastest / cheapest / highest quality)
  6. Offer to save to ~/.gstack/benchmarks/<date>.json for trend tracking

Uses gstack-model-benchmark --dry-run as a safety gate — auth status is
visible BEFORE the user spends API calls. If zero providers are authed,
the skill stops cleanly rather than attempting a run that produces no
useful output.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v1.3.0.0 — complete CHANGELOG + bump for post-1.2 scope additions

VERSION 1.2.0.0 → 1.3.0.0. The original 1.2 entry was written before I
added substantial new scope: the /benchmark-models skill, /ship Step 19.5
gstack-publish integration, --dry-run on gstack-model-benchmark, and the
lite E2E test coverage (4 new test files). A minor bump gives those
changes their own version line instead of silently folding them into
1.2's scope.

CHANGELOG additions under 1.3.0.0:
- /benchmark-models skill (new Added)
- /ship Step 19.5 publish check (new Added)
- gstack-model-benchmark --dry-run (new Added)
- Token ceiling 25K → 40K (moved to Changed)
- New Fixed section — codex adapter --skip-git-repo-check, --models
  dedupe, CI Dockerfile xz-utils + nodejs.org tarball
- 4 new test files documented under contributors (taste-engine,
  publish-dry-run, benchmark-cli, skill-e2e-benchmark-providers)
- Ship golden fixtures for claude/codex/factory hosts

Pre-existing 1.2 content preserved verbatim — no entries clobbered or
reordered. Sequence remains contiguous (1.3.0.0 → 1.1.3.0 → 1.1.2.0 →
1.1.1.0 → 1.1.0.0 → 1.0.0.0 → 0.19.0.0 → ...).

package.json and VERSION both at 1.3.0.0. No drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: adopt gbrain's release-summary CHANGELOG format + apply to v1.3

Ported the "release-summary format" rules from ~/git/gbrain/CLAUDE.md
(lines 291-354) into gstack's CLAUDE.md under the existing
"CHANGELOG + VERSION style" section. Every future `## [X.Y.Z]` entry
now needs a verdict-style release summary at the top:
1. Two-line bold headline (10-14 words)
2. Lead paragraph (3-5 sentences)
3. "Numbers that matter" with BEFORE / AFTER / Δ table
4. "What this means for [audience]" closer
5. `### Itemized changes` header
6. Existing itemized subsections below

Rewrote v1.3.0.0 entry to match. Preserved every existing bullet in
Added / Changed / Fixed / For contributors (no content clobbered per
the CLAUDE.md CHANGELOG rule).

Numbers in the v1.3 release summary are verifiable — every row of the
BEFORE / AFTER table has a reproducible command listed in the setup
paragraph (git log, bun test, grep for wiring status). No made-up
metrics.

Also added the gbrain "always credit community contributions" rule to
the itemized-changes section. `Contributed by @username` for every
community PR that lands in a CHANGELOG entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: remove gstack-publish — no real user need

User feedback: "i don't think i would use gstack-publish, i think we
should remove it." Agreed. The CLI + marketplace wiring was an
ambitious but speculative primitive. Zero users, zero validated demand,
and the existing manual `clawhub publish` workflow already covers the
real case (OpenClaw methodology skill publishing).

Deleted:
- bin/gstack-publish (the CLI)
- skills.json (the marketplace manifest)
- test/publish-dry-run.test.ts (13 tests)
- ship/SKILL.md.tmpl Step 19.5 — the methodology-skill publish-on-ship
  check. No target to dispatch to anymore.
- README.md Power tools row for gstack-publish

Updated:
- bin/gstack-model-benchmark doc comment: dropped "matches gstack-publish
  --dry-run semantics" reference (self-describing flag now)
- CHANGELOG 1.3.0.0 entry:
  * Release summary: "three new binaries" → "two new binaries".
    Dropped the /ship publish-check narrative.
  * Numbers table: "1 of 3 → 3 of 3 wired" → "1 of 2 → 2 of 2 wired".
    Deterministic test count: 45 → 32 (removed publish-dry-run's 13).
  * Added section: removed gstack-publish CLI bullet + /ship Step 19.5
    bullet.
  * "What this means for users" closer: replaced the /ship publish
    paragraph with the design-taste-engine learning loop, which IS
    real, wired, and something users hit every week via /design-shotgun.
  * Contributors section: "Four new test files" → "Three new test files"

Retained:
- openclaw/skills/gstack-openclaw-* skill dirs (pre-existed this PR,
  still publishable manually via `clawhub publish`, useful standalone
  for ClawHub installs)
- CLAUDE.md publishing-native-skills section (same rationale)

Regenerated SKILL.md across all hosts. Ship golden fixtures refreshed
for claude/codex/factory. 455 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(CHANGELOG): reorder v1.3 entry around day-to-day user wins

Previous entry led with internal metrics (CLIs wired to skills, preamble
line count, adapter bugs caught in CI). Useful to contributors, invisible
to users. Rewrote the release summary and Added section to lead with
what a day-to-day gstack user actually experiences.

Release summary changes:
- Headline: "Every new CLI wired to a slash command" → "Your design
  skills learn your taste. Your session state survives a laptop close."
- Lead paragraph: shifted from "primitives discoverable from /commands"
  to concrete day-to-day wins (design-shotgun taste memory, design-
  consultation anti-slop gates, continuous checkpoint survival).
- Numbers table: swapped internal metrics (CLI wiring %, test counts,
  preamble line count) for user-visible ones:
    - Design-variant convergence gate (0 → 3 axes required)
    - AI-slop font blacklist (~8 → 10+ fonts)
    - Taste memory across sessions (none → per-project JSON with decay)
    - Session state after crash (lost → auto-WIP with structured body)
    - /context-restore sources (markdown only → + WIP commits)
    - Models with behavioral overlays (1 → 5)
- "Most striking" interpretation: reframed around the mid-session
  crash survival story instead of the codex adapter bug catch.
- "What this means" closer: reframed around /design-shotgun + /design-
  consultation + continuous checkpoint workflow instead of
  /benchmark-models.

Added section — reorganized into six subsections by user value:
  1. Design skills that stop looking like AI
     (anti-slop constraints, taste engine)
  2. Session state that survives a crash
     (continuous checkpoint, /context-restore WIP reading,
     /ship non-destructive squash)
  3. Quality-of-life
     (feature discovery prompt, context health soft directive)
  4. Cross-host support
     (--model flag + 5 overlays)
  5. Config
     (gstack-config list/defaults, checkpoint_mode/push keys)
  6. Power-user / internal
     (gstack-model-benchmark + /benchmark-models skill — grouped and
     pushed to the bottom since it's more of a research tool than a
     daily workflow piece)

Changed / Fixed / For contributors sections unchanged. No content
clobbered per CLAUDE.md CHANGELOG rules — every existing bullet is
preserved, just reordered and grouped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(CHANGELOG): reframe v1.3 entry around transparency vs laptop-close

User feedback: "'closing your laptop' in the changelog is overstated, i
mean claude code does already have session management. i think the use
of the context save restore is mainly just another tool that is more in
your control instead of opaque and a part of CC." Correct. CC handles
session persistence on its own; continuous checkpoint isn't filling a
gap there, it's giving users a parallel, inspectable, portable track.

Reframed every place the old copy overstated:

- Headline: "Your session state survives a laptop close" → "Your
  session state lives in git, not a black box."
- Lead paragraph: dropped the "closing your laptop mid-refactor doesn't
  vaporize your decisions" line. Now frames continuous checkpoint as
  explicitly running alongside CC's built-in session management, not
  replacing it. Emphasizes grep-ability, portability across tools and
  branches.
- Numbers table row: "Session state after mid-refactor crash: lost
  since last manual commit → auto-WIP commits" → "Session state
  format: Claude Code's opaque session store → git commits +
  [gstack-context] bodies + markdown (parallel track)". Honest about
  what's actually changing.
- "Most striking" interpretation: replaced the "used to cost you every
  decision" framing with the real user value — session state stops
  being a black box, `git log --grep "WIP:"` shows the whole thread,
  any tool reading git can see it.
- "What this means" closer: replaced "survives crashes, context
  switches, and forgotten laptops" with accurate framing — parallel
  track alongside CC's own, inspectable, portable, useful when you
  want to review or hand off work.
- Added section: "Session state that survives a crash" subsection
  renamed to "Session state you can see, grep, and move". Lead bullet
  now explicitly notes continuous checkpoint runs alongside CC session
  management, not instead.

No content clobbered. All other bullets and sections unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(CHANGELOG): correct session-state location — home dir by default, git only on opt-in

User correction: "wait is our session management really checked into
git? i don't think that's right, isn't it just saved in your home
dir?" Right. I had the location wrong. The default session-save
mechanism (`/context-save` + `/context-restore`) writes markdown
files to `~/.gstack/projects/$SLUG/checkpoints/` — HOME, not git.
Continuous checkpoint mode (opt-in) is what writes git commits.
Previous copy conflated the two and implied "lives in git" as the
default state, which is wrong.

Every affected location updated:

- Headline: "lives in git, not a black box" → "becomes files you
  can grep, not a black box." Removes the false implication that
  session state lands in git by default.
- Lead paragraph: now explicitly names the two separate mechanisms.
  `/context-save` writes plaintext markdown to `~/.gstack/projects/
  $SLUG/checkpoints/` (the default). Continuous checkpoint mode
  (opt-in) additionally drops WIP: commits into the git log.
- Numbers table row: "Session state format" now reads "markdown in
  `~/.gstack/` by default, plus WIP: git commits if you opt into
  continuous mode (parallel track)." Tells the truth about which
  path is default vs opt-in.
- "Most striking" row interpretation: now names both paths. Default
  path = markdown files in home dir. Opt-in continuous mode = WIP:
  commits in project git log. Either way, plain text the user owns.
- "What this means" closer: similarly names both paths explicitly.
  "markdown files in your home directory by default, plus git
  commits if you opt into continuous mode."
- Continuous checkpoint mode Added bullet: clarifies the commits
  land in "your project's git log" (not implied to be the default),
  and notes it runs alongside BOTH Claude Code's built-in session
  management AND the default `/context-save` markdown flow.

No other bullets or sections touched. No content clobbered.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 17:50:31 +08:00

552 lines
27 KiB
TypeScript

import type { TemplateContext } from './types';
export function generateTestBootstrap(_ctx: TemplateContext): string {
return `## Test Framework Bootstrap
**Detect existing test framework and project runtime:**
\`\`\`bash
setopt +o nomatch 2>/dev/null || true # zsh compat
# Detect project runtime
[ -f Gemfile ] && echo "RUNTIME:ruby"
[ -f package.json ] && echo "RUNTIME:node"
[ -f requirements.txt ] || [ -f pyproject.toml ] && echo "RUNTIME:python"
[ -f go.mod ] && echo "RUNTIME:go"
[ -f Cargo.toml ] && echo "RUNTIME:rust"
[ -f composer.json ] && echo "RUNTIME:php"
[ -f mix.exs ] && echo "RUNTIME:elixir"
# Detect sub-frameworks
[ -f Gemfile ] && grep -q "rails" Gemfile 2>/dev/null && echo "FRAMEWORK:rails"
[ -f package.json ] && grep -q '"next"' package.json 2>/dev/null && echo "FRAMEWORK:nextjs"
# Check for existing test infrastructure
ls jest.config.* vitest.config.* playwright.config.* .rspec pytest.ini pyproject.toml phpunit.xml 2>/dev/null
ls -d test/ tests/ spec/ __tests__/ cypress/ e2e/ 2>/dev/null
# Check opt-out marker
[ -f .gstack/no-test-bootstrap ] && echo "BOOTSTRAP_DECLINED"
\`\`\`
**If test framework detected** (config files or test directories found):
Print "Test framework detected: {name} ({N} existing tests). Skipping bootstrap."
Read 2-3 existing test files to learn conventions (naming, imports, assertion style, setup patterns).
Store conventions as prose context for use in Phase 8e.5 or Step 7. **Skip the rest of bootstrap.**
**If BOOTSTRAP_DECLINED** appears: Print "Test bootstrap previously declined — skipping." **Skip the rest of bootstrap.**
**If NO runtime detected** (no config files found): Use AskUserQuestion:
"I couldn't detect your project's language. What runtime are you using?"
Options: A) Node.js/TypeScript B) Ruby/Rails C) Python D) Go E) Rust F) PHP G) Elixir H) This project doesn't need tests.
If user picks H → write \`.gstack/no-test-bootstrap\` and continue without tests.
**If runtime detected but no test framework — bootstrap:**
### B2. Research best practices
Use WebSearch to find current best practices for the detected runtime:
- \`"[runtime] best test framework 2025 2026"\`
- \`"[framework A] vs [framework B] comparison"\`
If WebSearch is unavailable, use this built-in knowledge table:
| Runtime | Primary recommendation | Alternative |
|---------|----------------------|-------------|
| Ruby/Rails | minitest + fixtures + capybara | rspec + factory_bot + shoulda-matchers |
| Node.js | vitest + @testing-library | jest + @testing-library |
| Next.js | vitest + @testing-library/react + playwright | jest + cypress |
| Python | pytest + pytest-cov | unittest |
| Go | stdlib testing + testify | stdlib only |
| Rust | cargo test (built-in) + mockall | — |
| PHP | phpunit + mockery | pest |
| Elixir | ExUnit (built-in) + ex_machina | — |
### B3. Framework selection
Use AskUserQuestion:
"I detected this is a [Runtime/Framework] project with no test framework. I researched current best practices. Here are the options:
A) [Primary] — [rationale]. Includes: [packages]. Supports: unit, integration, smoke, e2e
B) [Alternative] — [rationale]. Includes: [packages]
C) Skip — don't set up testing right now
RECOMMENDATION: Choose A because [reason based on project context]"
If user picks C → write \`.gstack/no-test-bootstrap\`. Tell user: "If you change your mind later, delete \`.gstack/no-test-bootstrap\` and re-run." Continue without tests.
If multiple runtimes detected (monorepo) → ask which runtime to set up first, with option to do both sequentially.
### B4. Install and configure
1. Install the chosen packages (npm/bun/gem/pip/etc.)
2. Create minimal config file
3. Create directory structure (test/, spec/, etc.)
4. Create one example test matching the project's code to verify setup works
If package installation fails → debug once. If still failing → revert with \`git checkout -- package.json package-lock.json\` (or equivalent for the runtime). Warn user and continue without tests.
### B4.5. First real tests
Generate 3-5 real tests for existing code:
1. **Find recently changed files:** \`git log --since=30.days --name-only --format="" | sort | uniq -c | sort -rn | head -10\`
2. **Prioritize by risk:** Error handlers > business logic with conditionals > API endpoints > pure functions
3. **For each file:** Write one test that tests real behavior with meaningful assertions. Never \`expect(x).toBeDefined()\` — test what the code DOES.
4. Run each test. Passes → keep. Fails → fix once. Still fails → delete silently.
5. Generate at least 1 test, cap at 5.
Never import secrets, API keys, or credentials in test files. Use environment variables or test fixtures.
### B5. Verify
\`\`\`bash
# Run the full test suite to confirm everything works
{detected test command}
\`\`\`
If tests fail → debug once. If still failing → revert all bootstrap changes and warn user.
### B5.5. CI/CD pipeline
\`\`\`bash
# Check CI provider
ls -d .github/ 2>/dev/null && echo "CI:github"
ls .gitlab-ci.yml .circleci/ bitrise.yml 2>/dev/null
\`\`\`
If \`.github/\` exists (or no CI detected — default to GitHub Actions):
Create \`.github/workflows/test.yml\` with:
- \`runs-on: ubuntu-latest\`
- Appropriate setup action for the runtime (setup-node, setup-ruby, setup-python, etc.)
- The same test command verified in B5
- Trigger: push + pull_request
If non-GitHub CI detected → skip CI generation with note: "Detected {provider} — CI pipeline generation supports GitHub Actions only. Add test step to your existing pipeline manually."
### B6. Create TESTING.md
First check: If TESTING.md already exists → read it and update/append rather than overwriting. Never destroy existing content.
Write TESTING.md with:
- Philosophy: "100% test coverage is the key to great vibe coding. Tests let you move fast, trust your instincts, and ship with confidence — without them, vibe coding is just yolo coding. With tests, it's a superpower."
- Framework name and version
- How to run tests (the verified command from B5)
- Test layers: Unit tests (what, where, when), Integration tests, Smoke tests, E2E tests
- Conventions: file naming, assertion style, setup/teardown patterns
### B7. Update CLAUDE.md
First check: If CLAUDE.md already has a \`## Testing\` section → skip. Don't duplicate.
Append a \`## Testing\` section:
- Run command and test directory
- Reference to TESTING.md
- Test expectations:
- 100% test coverage is the goal — tests make vibe coding safe
- When writing new functions, write a corresponding test
- When fixing a bug, write a regression test
- When adding error handling, write a test that triggers the error
- When adding a conditional (if/else, switch), write tests for BOTH paths
- Never commit code that makes existing tests fail
### B8. Commit
\`\`\`bash
git status --porcelain
\`\`\`
Only commit if there are changes. Stage all bootstrap files (config, test directory, TESTING.md, CLAUDE.md, .github/workflows/test.yml if created):
\`git commit -m "chore: bootstrap test framework ({framework name})"\`
---`;
}
// ─── Test Coverage Audit ────────────────────────────────────
//
// Shared methodology for codepath tracing, ASCII diagrams, and test gap analysis.
// Three modes, three placeholders, one inner function:
//
// {{TEST_COVERAGE_AUDIT_PLAN}} → plan-eng-review: adds missing tests to the plan
// {{TEST_COVERAGE_AUDIT_SHIP}} → ship: auto-generates tests, coverage summary
// {{TEST_COVERAGE_AUDIT_REVIEW}} → review: generates tests via Fix-First (ASK)
//
// ┌────────────────────────────────────────────────┐
// │ generateTestCoverageAuditInner(mode) │
// │ │
// │ SHARED: framework detect, codepath trace, │
// │ ASCII diagram, quality rubric, E2E matrix, │
// │ regression rule │
// │ │
// │ plan: edit plan file, write artifact │
// │ ship: auto-generate tests, write artifact │
// │ review: Fix-First ASK, INFORMATIONAL gaps │
// └────────────────────────────────────────────────┘
type CoverageAuditMode = 'plan' | 'ship' | 'review';
function generateTestCoverageAuditInner(mode: CoverageAuditMode): string {
const sections: string[] = [];
// ── Intro (mode-specific) ──
if (mode === 'ship') {
sections.push(`100% coverage is the goal — every untested path is a path where bugs hide and vibe coding becomes yolo coding. Evaluate what was ACTUALLY coded (from the diff), not what was planned.`);
} else if (mode === 'plan') {
sections.push(`100% coverage is the goal. Evaluate every codepath in the plan and ensure the plan includes tests for each one. If the plan is missing tests, add them — the plan should be complete enough that implementation includes full test coverage from the start.`);
} else {
sections.push(`100% coverage is the goal. Evaluate every codepath changed in the diff and identify test gaps. Gaps become INFORMATIONAL findings that follow the Fix-First flow.`);
}
// ── Test framework detection (shared) ──
sections.push(`
### Test Framework Detection
Before analyzing coverage, detect the project's test framework:
1. **Read CLAUDE.md** — look for a \`## Testing\` section with test command and framework name. If found, use that as the authoritative source.
2. **If CLAUDE.md has no testing section, auto-detect:**
\`\`\`bash
setopt +o nomatch 2>/dev/null || true # zsh compat
# Detect project runtime
[ -f Gemfile ] && echo "RUNTIME:ruby"
[ -f package.json ] && echo "RUNTIME:node"
[ -f requirements.txt ] || [ -f pyproject.toml ] && echo "RUNTIME:python"
[ -f go.mod ] && echo "RUNTIME:go"
[ -f Cargo.toml ] && echo "RUNTIME:rust"
# Check for existing test infrastructure
ls jest.config.* vitest.config.* playwright.config.* cypress.config.* .rspec pytest.ini phpunit.xml 2>/dev/null
ls -d test/ tests/ spec/ __tests__/ cypress/ e2e/ 2>/dev/null
\`\`\`
3. **If no framework detected:**${mode === 'ship' ? ' falls through to the Test Framework Bootstrap step (Step 4) which handles full setup.' : ' still produce the coverage diagram, but skip test generation.'}`);
// ── Before/after count (ship only) ──
if (mode === 'ship') {
sections.push(`
**0. Before/after test count:**
\`\`\`bash
# Count test files before any generation
find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' | grep -v node_modules | wc -l
\`\`\`
Store this number for the PR body.`);
}
// ── Codepath tracing methodology (shared, with mode-specific source) ──
const traceSource = mode === 'plan'
? `**Step 1. Trace every codepath in the plan:**
Read the plan document. For each new feature, service, endpoint, or component described, trace how data will flow through the code — don't just list planned functions, actually follow the planned execution:`
: `**${mode === 'ship' ? '1' : 'Step 1'}. Trace every codepath changed** using \`git diff origin/<base>...HEAD\`:
Read every changed file. For each one, trace how data flows through the code — don't just list functions, actually follow the execution:`;
const traceStep1 = mode === 'plan'
? `1. **Read the plan.** For each planned component, understand what it does and how it connects to existing code.`
: `1. **Read the diff.** For each changed file, read the full file (not just the diff hunk) to understand context.`;
sections.push(`
${traceSource}
${traceStep1}
2. **Trace data flow.** Starting from each entry point (route handler, exported function, event listener, component render), follow the data through every branch:
- Where does input come from? (request params, props, database, API call)
- What transforms it? (validation, mapping, computation)
- Where does it go? (database write, API response, rendered output, side effect)
- What can go wrong at each step? (null/undefined, invalid input, network failure, empty collection)
3. **Diagram the execution.** For each changed file, draw an ASCII diagram showing:
- Every function/method that was added or modified
- Every conditional branch (if/else, switch, ternary, guard clause, early return)
- Every error path (try/catch, rescue, error boundary, fallback)
- Every call to another function (trace into it — does IT have untested branches?)
- Every edge: what happens with null input? Empty array? Invalid type?
This is the critical step — you're building a map of every line of code that can execute differently based on input. Every branch in this diagram needs a test.`);
// ── User flow coverage (shared) ──
sections.push(`
**${mode === 'ship' ? '2' : 'Step 2'}. Map user flows, interactions, and error states:**
Code coverage isn't enough — you need to cover how real users interact with the changed code. For each changed feature, think through:
- **User flows:** What sequence of actions does a user take that touches this code? Map the full journey (e.g., "user clicks 'Pay' → form validates → API call → success/failure screen"). Each step in the journey needs a test.
- **Interaction edge cases:** What happens when the user does something unexpected?
- Double-click/rapid resubmit
- Navigate away mid-operation (back button, close tab, click another link)
- Submit with stale data (page sat open for 30 minutes, session expired)
- Slow connection (API takes 10 seconds — what does the user see?)
- Concurrent actions (two tabs, same form)
- **Error states the user can see:** For every error the code handles, what does the user actually experience?
- Is there a clear error message or a silent failure?
- Can the user recover (retry, go back, fix input) or are they stuck?
- What happens with no network? With a 500 from the API? With invalid data from the server?
- **Empty/zero/boundary states:** What does the UI show with zero results? With 10,000 results? With a single character input? With maximum-length input?
Add these to your diagram alongside the code branches. A user flow with no test is just as much a gap as an untested if/else.`);
// ── Check branches against tests + quality rubric (shared) ──
sections.push(`
**${mode === 'ship' ? '3' : 'Step 3'}. Check each branch against existing tests:**
Go through your diagram branch by branch — both code paths AND user flows. For each one, search for a test that exercises it:
- Function \`processPayment()\` → look for \`billing.test.ts\`, \`billing.spec.ts\`, \`test/billing_test.rb\`
- An if/else → look for tests covering BOTH the true AND false path
- An error handler → look for a test that triggers that specific error condition
- A call to \`helperFn()\` that has its own branches → those branches need tests too
- A user flow → look for an integration or E2E test that walks through the journey
- An interaction edge case → look for a test that simulates the unexpected action
Quality scoring rubric:
- ★★★ Tests behavior with edge cases AND error paths
- ★★ Tests correct behavior, happy path only
- ★ Smoke test / existence check / trivial assertion (e.g., "it renders", "it doesn't throw")`);
// ── E2E test decision matrix (shared) ──
sections.push(`
### E2E Test Decision Matrix
When checking each branch, also determine whether a unit test or E2E/integration test is the right tool:
**RECOMMEND E2E (mark as [→E2E] in the diagram):**
- Common user flow spanning 3+ components/services (e.g., signup → verify email → first login)
- Integration point where mocking hides real failures (e.g., API → queue → worker → DB)
- Auth/payment/data-destruction flows — too important to trust unit tests alone
**RECOMMEND EVAL (mark as [→EVAL] in the diagram):**
- Critical LLM call that needs a quality eval (e.g., prompt change → test output still meets quality bar)
- Changes to prompt templates, system instructions, or tool definitions
**STICK WITH UNIT TESTS:**
- Pure function with clear inputs/outputs
- Internal helper with no side effects
- Edge case of a single function (null input, empty array)
- Obscure/rare flow that isn't customer-facing`);
// ── Regression rule (shared) ──
sections.push(`
### REGRESSION RULE (mandatory)
**IRON RULE:** When the coverage audit identifies a REGRESSION — code that previously worked but the diff broke — a regression test is ${mode === 'plan' ? 'added to the plan as a critical requirement' : 'written immediately'}. No AskUserQuestion. No skipping. Regressions are the highest-priority test because they prove something broke.
A regression is when:
- The diff modifies existing behavior (not new code)
- The existing test suite (if any) doesn't cover the changed path
- The change introduces a new failure mode for existing callers
When uncertain whether a change is a regression, err on the side of writing the test.${mode !== 'plan' ? '\n\nFormat: commit as `test: regression test for {what broke}`' : ''}`);
// ── ASCII coverage diagram (shared) ──
sections.push(`
**${mode === 'ship' ? '4' : 'Step 4'}. Output ASCII coverage diagram:**
Include BOTH code paths and user flows in the same diagram. Mark E2E-worthy and eval-worthy paths:
\`\`\`
CODE PATHS USER FLOWS
[+] src/services/billing.ts [+] Payment checkout
├── processPayment() ├── [★★★ TESTED] Complete purchase — checkout.e2e.ts:15
│ ├── [★★★ TESTED] happy + declined + timeout ├── [GAP] [→E2E] Double-click submit
│ ├── [GAP] Network timeout └── [GAP] Navigate away mid-payment
│ └── [GAP] Invalid currency
└── refundPayment() [+] Error states
├── [★★ TESTED] Full refund — :89 ├── [★★ TESTED] Card declined message
└── [★ TESTED] Partial (non-throw only) — :101 └── [GAP] Network timeout UX
LLM integration: [GAP] [→EVAL] Prompt template change — needs eval test
COVERAGE: 5/13 paths tested (38%) | Code paths: 3/5 (60%) | User flows: 2/8 (25%)
QUALITY: ★★★:2 ★★:2 ★:1 | GAPS: 8 (2 E2E, 1 eval)
\`\`\`
Legend: ★★★ behavior + edge + error | ★★ happy path | ★ smoke check
[→E2E] = needs integration test | [→EVAL] = needs LLM eval
**Fast path:** All paths covered → "${mode === 'ship' ? 'Step 7' : mode === 'review' ? 'Step 4.75' : 'Test review'}: All new code paths have test coverage ✓" Continue.`);
// ── Mode-specific action section ──
if (mode === 'plan') {
sections.push(`
**Step 5. Add missing tests to the plan:**
For each GAP identified in the diagram, add a test requirement to the plan. Be specific:
- What test file to create (match existing naming conventions)
- What the test should assert (specific inputs → expected outputs/behavior)
- Whether it's a unit test, E2E test, or eval (use the decision matrix)
- For regressions: flag as **CRITICAL** and explain what broke
The plan should be complete enough that when implementation begins, every test is written alongside the feature code — not deferred to a follow-up.`);
// ── Test plan artifact (plan + ship) ──
sections.push(`
### Test Plan Artifact
After producing the coverage diagram, write a test plan artifact to the project directory so \`/qa\` and \`/qa-only\` can consume it as primary test input:
\`\`\`bash
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gstack/projects/$SLUG
USER=$(whoami)
DATETIME=$(date +%Y%m%d-%H%M%S)
\`\`\`
Write to \`~/.gstack/projects/{slug}/{user}-{branch}-eng-review-test-plan-{datetime}.md\`:
\`\`\`markdown
# Test Plan
Generated by /plan-eng-review on {date}
Branch: {branch}
Repo: {owner/repo}
## Affected Pages/Routes
- {URL path} — {what to test and why}
## Key Interactions to Verify
- {interaction description} on {page}
## Edge Cases
- {edge case} on {page}
## Critical Paths
- {end-to-end flow that must work}
\`\`\`
This file is consumed by \`/qa\` and \`/qa-only\` as primary test input. Include only the information that helps a QA tester know **what to test and where** — not implementation details.`);
} else if (mode === 'ship') {
sections.push(`
**5. Generate tests for uncovered paths:**
If test framework detected (or bootstrapped in Step 4):
- Prioritize error handlers and edge cases first (happy paths are more likely already tested)
- Read 2-3 existing test files to match conventions exactly
- Generate unit tests. Mock all external dependencies (DB, API, Redis).
- For paths marked [→E2E]: generate integration/E2E tests using the project's E2E framework (Playwright, Cypress, Capybara, etc.)
- For paths marked [→EVAL]: generate eval tests using the project's eval framework, or flag for manual eval if none exists
- Write tests that exercise the specific uncovered path with real assertions
- Run each test. Passes → commit as \`test: coverage for {feature}\`
- Fails → fix once. Still fails → revert, note gap in diagram.
Caps: 30 code paths max, 20 tests generated max (code + user flow combined), 2-min per-test exploration cap.
If no test framework AND user declined bootstrap → diagram only, no generation. Note: "Test generation skipped — no test framework configured."
**Diff is test-only changes:** Skip Step 7 entirely: "No new application code paths to audit."
**6. After-count and coverage summary:**
\`\`\`bash
# Count test files after generation
find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' | grep -v node_modules | wc -l
\`\`\`
For PR body: \`Tests: {before} → {after} (+{delta} new)\`
Coverage line: \`Test Coverage Audit: N new code paths. M covered (X%). K tests generated, J committed.\`
**7. Coverage gate:**
Before proceeding, check CLAUDE.md for a \`## Test Coverage\` section with \`Minimum:\` and \`Target:\` fields. If found, use those percentages. Otherwise use defaults: Minimum = 60%, Target = 80%.
Using the coverage percentage from the diagram in substep 4 (the \`COVERAGE: X/Y (Z%)\` line):
- **>= target:** Pass. "Coverage gate: PASS ({X}%)." Continue.
- **>= minimum, < target:** Use AskUserQuestion:
- "AI-assessed coverage is {X}%. {N} code paths are untested. Target is {target}%."
- RECOMMENDATION: Choose A because untested code paths are where production bugs hide.
- Options:
A) Generate more tests for remaining gaps (recommended)
B) Ship anyway — I accept the coverage risk
C) These paths don't need tests — mark as intentionally uncovered
- If A: Loop back to substep 5 (generate tests) targeting the remaining gaps. After second pass, if still below target, present AskUserQuestion again with updated numbers. Maximum 2 generation passes total.
- If B: Continue. Include in PR body: "Coverage gate: {X}% — user accepted risk."
- If C: Continue. Include in PR body: "Coverage gate: {X}% — {N} paths intentionally uncovered."
- **< minimum:** Use AskUserQuestion:
- "AI-assessed coverage is critically low ({X}%). {N} of {M} code paths have no tests. Minimum threshold is {minimum}%."
- RECOMMENDATION: Choose A because less than {minimum}% means more code is untested than tested.
- Options:
A) Generate tests for remaining gaps (recommended)
B) Override — ship with low coverage (I understand the risk)
- If A: Loop back to substep 5. Maximum 2 passes. If still below minimum after 2 passes, present the override choice again.
- If B: Continue. Include in PR body: "Coverage gate: OVERRIDDEN at {X}%."
**Coverage percentage undetermined:** If the coverage diagram doesn't produce a clear numeric percentage (ambiguous output, parse error), **skip the gate** with: "Coverage gate: could not determine percentage — skipping." Do not default to 0% or block.
**Test-only diffs:** Skip the gate (same as the existing fast-path).
**100% coverage:** "Coverage gate: PASS (100%)." Continue.`);
// ── Test plan artifact (ship mode) ──
sections.push(`
### Test Plan Artifact
After producing the coverage diagram, write a test plan artifact so \`/qa\` and \`/qa-only\` can consume it:
\`\`\`bash
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gstack/projects/$SLUG
USER=$(whoami)
DATETIME=$(date +%Y%m%d-%H%M%S)
\`\`\`
Write to \`~/.gstack/projects/{slug}/{user}-{branch}-ship-test-plan-{datetime}.md\`:
\`\`\`markdown
# Test Plan
Generated by /ship on {date}
Branch: {branch}
Repo: {owner/repo}
## Affected Pages/Routes
- {URL path} — {what to test and why}
## Key Interactions to Verify
- {interaction description} on {page}
## Edge Cases
- {edge case} on {page}
## Critical Paths
- {end-to-end flow that must work}
\`\`\``);
} else {
// review mode
sections.push(`
**Step 5. Generate tests for gaps (Fix-First):**
If test framework is detected and gaps were identified:
- Classify each gap as AUTO-FIX or ASK per the Fix-First Heuristic:
- **AUTO-FIX:** Simple unit tests for pure functions, edge cases of existing tested functions
- **ASK:** E2E tests, tests requiring new test infrastructure, tests for ambiguous behavior
- For AUTO-FIX gaps: generate the test, run it, commit as \`test: coverage for {feature}\`
- For ASK gaps: include in the Fix-First batch question with the other review findings
- For paths marked [→E2E]: always ASK (E2E tests are higher-effort and need user confirmation)
- For paths marked [→EVAL]: always ASK (eval tests need user confirmation on quality criteria)
If no test framework detected → include gaps as INFORMATIONAL findings only, no generation.
**Diff is test-only changes:** Skip Step 4.75 entirely: "No new application code paths to audit."
### Coverage Warning
After producing the coverage diagram, check the coverage percentage. Read CLAUDE.md for a \`## Test Coverage\` section with a \`Minimum:\` field. If not found, use default: 60%.
If coverage is below the minimum threshold, output a prominent warning **before** the regular review findings:
\`\`\`
⚠️ COVERAGE WARNING: AI-assessed coverage is {X}%. {N} code paths untested.
Consider writing tests before running /ship.
\`\`\`
This is INFORMATIONAL — does not block /review. But it makes low coverage visible early so the developer can address it before reaching the /ship coverage gate.
If coverage percentage cannot be determined, skip the warning silently.`);
}
return sections.join('\n');
}
export function generateTestCoverageAuditPlan(_ctx: TemplateContext): string {
return generateTestCoverageAuditInner('plan');
}
export function generateTestCoverageAuditShip(_ctx: TemplateContext): string {
return generateTestCoverageAuditInner('ship');
}
export function generateTestCoverageAuditReview(_ctx: TemplateContext): string {
return generateTestCoverageAuditInner('review');
}