mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-01 19:25:10 +02:00
feat(v1.3.0.0): open agents learnings + cross-model benchmark skill (#1040)
* chore: regenerate stale ship golden fixtures
Golden fixtures were missing the VENDORED_GSTACK preamble section that
landed on main. Regression tests failed on all three hosts (claude, codex,
factory). Regenerated from current preamble output.
No code changes, unblocks test suite.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: anti-slop design constraints + delete duplicate constants
Tightens design-consultation and design-shotgun to push back on the
convergence traps every AI design tool falls into.
Changes:
- scripts/resolvers/constants.ts: add "system-ui as primary font" to
AI_SLOP_BLACKLIST. Document Space Grotesk as the new "safe alternative
to Inter" convergence trap alongside the existing overused fonts.
- scripts/gen-skill-docs.ts: delete duplicate AI slop constants block
(dead code — scripts/resolvers/constants.ts is the live source).
Prevents drift between the two definitions.
- design-consultation/SKILL.md.tmpl: add Space Grotesk + system-ui to
overused/slop lists. Add "anti-convergence directive" — vary across
generations in the same project. Add Phase 1 "memorable-thing forcing
question" (what's the one thing someone will remember?). Add Phase 5
"would a human designer be embarrassed by this?" self-gate before
presenting variants.
- design-shotgun/SKILL.md.tmpl: anti-convergence directive — each
variant must use a different font, palette, and layout. If two
variants look like siblings, one of them failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: context health soft directive in preamble (T2+)
Adds a "periodically self-summarize" nudge to long-running skills.
Soft directive only — no thresholds, no enforcement, no auto-commit.
Goal: self-awareness during /qa, /investigate, /cso etc. If you notice
yourself going in circles, STOP and reassess instead of thrashing.
Codex review caught that fake precision thresholds (15/30/45 tool calls)
were unimplementable — SKILL.md is a static prompt, not runtime code.
This ships the soft version only.
Changes:
- scripts/resolvers/preamble.ts: add generateContextHealth(), wire into
T2+ tier. Format: [PROGRESS] ... summary line. Explicit rule that
progress reporting must never mutate git state.
- All T2+ skill SKILL.md files regenerated to include the new section.
- Golden ship fixtures updated (T4 skill, picks up the change).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: model overlays with explicit --model flag (no auto-detect)
Adds a per-model behavioral patch layer orthogonal to the host axis.
Different LLMs have different tendencies (GPT won't stop, Gemini
over-explains, o-series wants structured output). Overlays nudge each
model toward better defaults for gstack workflows.
Codex review caught three landmines the prior reviews missed:
1. Host != model — Claude Code can run any Claude model, Codex runs
GPT/o-series, Cursor fronts multiple providers. Auto-detecting from
host would lie. Dropped auto-detect. --model is explicit (default
claude). Missing overlay file → empty string (graceful).
2. Import cycle — putting Model in resolvers/types.ts would cycle
through hosts/index. Created neutral scripts/models.ts instead.
3. "Final say" is dangerous — overlay at the end of preamble could
override STOP points, AskUserQuestion gates, /ship review gates.
Placed overlay after spawned-session-check but before voice + tier
sections. Wrapper heading adds explicit subordination language on
every overlay: "subordinate to skill workflow, STOP points,
AskUserQuestion gates, plan-mode safety, and /ship review gates."
Changes:
- scripts/models.ts: new neutral module. ALL_MODEL_NAMES, Model type,
resolveModel() for family heuristics (gpt-5.4-mini → gpt-5.4, o3 →
o-series, claude-opus-4-7 → claude), validateModel() helper.
- scripts/resolvers/types.ts: import Model, add ctx.model field.
- scripts/resolvers/model-overlay.ts: new resolver. Reads
model-overlays/{model}.md. Supports {{INHERIT:base}} directive at
top of file for concat (gpt-5.4 inherits gpt). Cycle guard.
- scripts/resolvers/index.ts: register MODEL_OVERLAY resolver.
- scripts/resolvers/preamble.ts: wire generateModelOverlay into
composition before voice. Print MODEL_OVERLAY: {model} in preamble
bash so users can see which overlay is active. Filter empty sections.
- scripts/gen-skill-docs.ts: parse --model CLI flag. Default claude.
Unknown model → throw with list of valid options.
- model-overlays/{claude,gpt,gpt-5.4,gemini,o-series}.md: behavioral
patches per model family. gpt-5.4.md uses {{INHERIT:gpt}} to extend
gpt.md without duplication.
- test/gen-skill-docs.test.ts: fix qa-only guardrail regex scope.
Was matching Edit/Glob/Grep anywhere after `allowed-tools:` in the
whole file. Now scoped to frontmatter only. Body prose (Claude
overlay references Edit as a tool) correctly no longer breaks it.
Verification:
- bun run gen:skill-docs --host all --dry-run → all fresh
- bun run gen:skill-docs --model gpt-5.4 → concat works, gpt.md +
gpt-5.4.md content appears in order
- bun run gen:skill-docs --model unknown → errors with valid list
- All generated skills contain MODEL_OVERLAY: claude in preamble
- Golden ship fixtures regenerated
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: continuous checkpoint mode with non-destructive WIP squash
Adds opt-in auto-commit during long sessions so work survives Claude
Code crashes, Conductor workspace handoffs, and context switches.
Local-only by default — pushing requires explicit opt-in.
Codex review caught multiple landmines that would have shipped:
1. checkpoint_push=true default would push WIP commits to shared
branches, trigger CI/deploys, expose secrets. Now default false.
2. Plan's original /ship squash (git reset --soft to merge base) was
destructive — uncommitted ALL branch commits, not just WIP, and
caused non-fast-forward pushes. Redesigned: rebase --autosquash
scoped to WIP commits only, with explicit fallback for WIP-only
branches and STOP-and-ask for conflicts.
3. gstack-config get returned empty for missing keys with exit 0,
ignoring the annotated defaults in the header comments. Fixed:
get now falls back to a lookup_default() table that is the
canonical source for defaults.
4. Telemetry default mismatched: header said 'anonymous' but runtime
treated empty as 'off'. Aligned: default is 'off' everywhere.
5. /checkpoint resume only read markdown checkpoint files, not the
WIP commit [gstack-context] bodies the plan referenced. Wired up
parsing of [gstack-context] blocks from WIP commits as a second
recovery trail alongside the markdown checkpoints.
Changes:
- bin/gstack-config: add checkpoint_mode (default explicit) and
checkpoint_push (default false) to CONFIG_HEADER. Add lookup_default()
as canonical default source. get() falls back to defaults when key
absent. list now shows value + source (set/default). New 'defaults'
subcommand to inspect the table.
- scripts/resolvers/preamble.ts: preamble bash reads _CHECKPOINT_MODE
and _CHECKPOINT_PUSH, prints CHECKPOINT_MODE: and CHECKPOINT_PUSH: so
the mode is visible. New generateContinuousCheckpoint() section in
T2+ tier describes WIP commit format with [gstack-context] body and
the rules (never git add -A, never commit broken tests, push only
if opted in). Example deliberately shows a clean-state context so
it doesn't contradict the rules.
- ship/SKILL.md.tmpl: new Step 5.75 WIP Commit Squash. Detects WIP
count, exports [gstack-context] blocks before squash (as backup),
uses rebase --autosquash for mixed branches and soft-reset only when
VERIFIED WIP-only. Explicit anti-footgun rules against blind soft-
reset. Aborts with BLOCKED status on conflict instead of destroying
non-WIP commits.
- checkpoint/SKILL.md.tmpl: new Step 1.5 to parse [gstack-context]
blocks from WIP commits via git log --grep="^WIP:". Merges with
markdown checkpoint for fuller session recovery.
- Golden ship fixtures regenerated (ship is T4, preamble change shows up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: feature discovery flow gated by per-feature markers
Extends generateUpgradeCheck() to surface new features once per user
after a just-upgraded session. No more silent features.
Codex review caught: spawned sessions (OpenClaw, etc.) must skip the
discovery prompt entirely — they can't interactively answer. Feature
discovery now checks SPAWNED_SESSION first and is silent in those.
Discovery is per-feature, not per-upgrade. Each feature has its own
marker file at ~/.claude/skills/gstack/.feature-prompted-{name}. Once
the user has been shown a feature (accepted, shown docs, or skipped),
the marker is touched and the prompt never fires again for that
feature. Future features get their own markers.
V1 features surfaced:
- continuous-checkpoint: offer to enable checkpoint_mode=continuous
- model-overlay: inform-only note about --model flag and MODEL_OVERLAY
line in preamble output
Max one prompt per session to avoid nagging. Fires only on JUST_UPGRADED
(not every session), plus spawned-session skip.
Changes:
- scripts/resolvers/preamble.ts: extend generateUpgradeCheck() with
feature discovery rules, per-marker-file semantics, spawned-session
exclusion, and max-one-per-session cap.
- All skill SKILL.md files regenerated to include the new section.
- Golden ship fixtures regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: design taste engine with persistent schema
Adds a cross-session taste profile that learns from design-shotgun
approval/rejection decisions. Biases future design-consultation and
design-shotgun proposals toward the user's demonstrated preferences.
Codex review caught that the plan had "taste engine" as a vague goal
without schema, decay, migration, or placeholder insertion points. This
commit ships the full spec.
Schema v1 at ~/.gstack/projects/$SLUG/taste-profile.json:
- version, updated_at
- dimensions: fonts, colors, layouts, aesthetics — each with approved[]
and rejected[] preference lists
- sessions: last 50 (FIFO truncation), each with ts/action/variant/reason
- Preference: { value, confidence, approved_count, rejected_count, last_seen }
- Confidence: Laplace-smoothed approved/(total+1)
- Decay: 5% per week of inactivity, computed at read time (not write)
Changes:
- bin/gstack-taste-update: new CLI. Subcommands approved/rejected/show/
migrate. Parses reason string for dimension signals (e.g.,
"fonts: Geist; colors: slate; aesthetics: minimal"). Emits taste-drift
NOTE when a new signal contradicts a strong opposing signal. Legacy
approved.json aggregates migrate to v1 on next write.
- scripts/resolvers/design.ts: new generateTasteProfile() resolver.
Produces the prose that skills see: how to read the profile, how to
factor into proposals, conflict handling, schema migration.
- scripts/resolvers/index.ts: register TASTE_PROFILE and a BIN_DIR
resolver (returns ctx.paths.binDir, used by templates that shell out
to gstack-* binaries).
- design-consultation/SKILL.md.tmpl: insert {{TASTE_PROFILE}} placeholder
in Phase 1 right after the memorable-thing forcing question so the
Phase 3 proposal can factor in learned preferences.
- design-shotgun/SKILL.md.tmpl: taste memory section now reads
taste-profile.json via {{TASTE_PROFILE}}, falls back to per-session
approved.json (legacy). Approval flow documented to call
gstack-taste-update after user picks/rejects a variant.
Known gap: v1 extracts dimension signals from a reason string passed
by the caller ("fonts: X; colors: Y"). Future v2 can read EXIF or an
accompanying manifest written by design-shotgun alongside each variant
for automatic dimension extraction without needing the reason argument.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: multi-provider model benchmark (boil the ocean)
Adds the full spec Codex asked for: real provider adapters with auth
detection, normalized RunResult, pricing tables, tool compatibility
maps, parallel execution with error isolation, and table/JSON/markdown
output. Judge stays on Anthropic SDK as the single stable source of
quality scoring, gated behind --judge.
Codex flagged the original plan as massively under-scoped — the
existing runner is Claude-only and the judge is Anthropic-only. You
can't benchmark GPT or Gemini without real provider infrastructure.
This commit ships it.
New architecture:
test/helpers/providers/types.ts ProviderAdapter interface
test/helpers/providers/claude.ts wraps `claude -p --output-format json`
test/helpers/providers/gpt.ts wraps `codex exec --json`
test/helpers/providers/gemini.ts wraps `gemini -p --output-format stream-json --yolo`
test/helpers/pricing.ts per-model USD cost tables (quarterly)
test/helpers/tool-map.ts which tools each CLI exposes
test/helpers/benchmark-runner.ts orchestrator (Promise.allSettled)
test/helpers/benchmark-judge.ts Anthropic SDK quality scorer
bin/gstack-model-benchmark CLI entry
test/benchmark-runner.test.ts 9 unit tests (cost math, formatters, tool-map)
Per-provider error isolation:
- auth → record reason, don't abort batch
- timeout → record reason, don't abort batch
- rate_limit → record reason, don't abort batch
- binary_missing → record in available() check, skip if --skip-unavailable
Pricing correction: cached input tokens are disjoint from uncached
input tokens (Anthropic/OpenAI report them separately). Original
math subtracted them, producing negative costs. Now adds cached at
the 10% discount alongside the full uncached input cost.
CLI:
gstack-model-benchmark --prompt "..." --models claude,gpt,gemini
gstack-model-benchmark ./prompt.txt --output json --judge
gstack-model-benchmark ./prompt.txt --models claude --timeout-ms 60000
Output formats: table (default), json, markdown. Each shows model,
latency, in→out tokens, cost, quality (when --judge used), tool calls,
and any errors.
Known limitations for v1:
- Claude adapter approximates toolCalls as num_turns (stream-json
would give exact counts; v2 can upgrade).
- Live E2E tests (test/providers.e2e.test.ts) not included — they
require CI secrets for all three providers. Unit tests cover the
shape and math.
- Provider CLIs sometimes return non-JSON error text to stdout; the
parsers fall back to treating raw output as plain text in that case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: standalone methodology skill publishing via gstack-publish
Ships the marketplace-distribution half of Item 5 (reframed): publish
the existing standalone OpenClaw methodology skills to multiple
marketplaces with one command.
Codex review caught that the original plan assumed raw generated
multi-host skills could be published directly. They can't — those
depend on gstack binaries, generated host paths, tool names, and
telemetry. The correct artifact class is hand-crafted standalone
skills in openclaw/skills/gstack-openclaw-* (already exist and work
without gstack runtime). This commit adds the wrapper that publishes
them to ClawHub + SkillsMP + Vercel Skills.sh with per-marketplace
error isolation and dry-run validation.
Changes:
- skills.json: root manifest with 4 skills (office-hours, ceo-review,
investigate, retro) each pointing at its openclaw/skills source.
Each skill declares per-marketplace targets with a slug, a publish
flag, and a compatible-hosts list. Marketplace configs include CLI
name, login command, publish command template (with placeholder
substitution), docs URL, and auth_check command.
- bin/gstack-publish: new CLI. Subcommands:
gstack-publish Publish all skills
gstack-publish <slug> Publish one skill
gstack-publish --dry-run Validate + auth-check without publishing
gstack-publish --list List skills + marketplace targets
Features:
* Manifest validation (missing source files, missing slugs, empty
marketplace list all reported).
* Per-marketplace auth check before any publish attempt.
* Per-skill / per-marketplace error isolation: one failure doesn't
abort the batch.
* Idempotent — re-running with the same version is safe; markets
that reject duplicate versions report it as a failure for that
single target without affecting others.
* --dry-run walks the full pipeline but skips execSync; useful in
CI to validate manifest before bumping version.
Tested locally: clawhub auth detected, skillsmp/vercel CLIs not
installed (marked NOT READY and skipped cleanly in dry-run).
Follow-up work (tracked in TODOS.md later):
- Version-bump helper that reads openclaw/skills/*/SKILL.md frontmatter
and updates skills.json in lockstep.
- CI workflow that runs gstack-publish --dry-run on every PR and
gstack-publish on tags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: split preamble.ts into submodules (byte-identical output)
Splits scripts/resolvers/preamble.ts (841 lines, 18 generator functions +
composition root) into one file per generator under
scripts/resolvers/preamble/. Root preamble.ts becomes a thin composition
layer (~80 lines of imports + generatePreamble).
Before:
scripts/resolvers/preamble.ts 841 lines
After:
scripts/resolvers/preamble.ts 83 lines
scripts/resolvers/preamble/generate-preamble-bash.ts 97 lines
scripts/resolvers/preamble/generate-upgrade-check.ts 48 lines
scripts/resolvers/preamble/generate-lake-intro.ts 16 lines
scripts/resolvers/preamble/generate-telemetry-prompt.ts 37 lines
scripts/resolvers/preamble/generate-proactive-prompt.ts 25 lines
scripts/resolvers/preamble/generate-routing-injection.ts 49 lines
scripts/resolvers/preamble/generate-vendoring-deprecation.ts 36 lines
scripts/resolvers/preamble/generate-spawned-session-check.ts 11 lines
scripts/resolvers/preamble/generate-ask-user-format.ts 16 lines
scripts/resolvers/preamble/generate-completeness-section.ts 19 lines
scripts/resolvers/preamble/generate-repo-mode-section.ts 12 lines
scripts/resolvers/preamble/generate-test-failure-triage.ts 108 lines
scripts/resolvers/preamble/generate-search-before-building.ts 14 lines
scripts/resolvers/preamble/generate-completion-status.ts 161 lines
scripts/resolvers/preamble/generate-voice-directive.ts 60 lines
scripts/resolvers/preamble/generate-context-recovery.ts 51 lines
scripts/resolvers/preamble/generate-continuous-checkpoint.ts 48 lines
scripts/resolvers/preamble/generate-context-health.ts 31 lines
Byte-identity verification (the real gate per Codex correction):
- Before refactor: snapshotted 135 generated SKILL.md files via
`find -name SKILL.md -type f | grep -v /gstack/` across all hosts.
- After refactor: regenerated with `bun run gen:skill-docs --host all`
and re-snapshotted.
- `diff -r baseline after` returned zero differences and exit 0.
The `--host all --dry-run` gate passes too. No template or host behavior
changes — purely a code-organization refactor.
Test fix: audit-compliance.test.ts's telemetry check previously grepped
preamble.ts directly for `_TEL != "off"`. After the refactor that logic
lives in preamble/generate-preamble-bash.ts. Test now concatenates all
preamble submodule sources before asserting — tracks the semantic contract,
not the file layout. Doing the minimum rewrite preserves the test's intent
(conditional telemetry) without coupling it to file boundaries.
Why now: we were in-session with full context. Codex had downgraded this
from mandatory to optional, but the preamble had grown to 841 lines and
was getting harder to navigate. User asked "why not?" given the context
was hot. Shipping it as a clean bisectable commit while all the prior
preamble.ts changes are fresh reduces rebase pain later.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.19.0.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: trim verbose preamble + coverage audit prose
Compress without removing behavior or voice. Three targeted cuts:
1. scripts/resolvers/testing.ts coverage diagram example: 40 lines → 14
lines. Two-column ASCII layout instead of stacked sections.
Preserves all required regression-guard phrases (processPayment,
refundPayment, billing.test.ts, checkout.e2e.ts, COVERAGE, QUALITY,
GAPS, Code paths, User flows, ASCII coverage diagram).
2. scripts/resolvers/preamble/generate-completion-status.ts Plan Status
Footer: was 35 lines with embedded markdown table example, now 7
lines that describe the table inline. The footer fires only at
ExitPlanMode time — Claude can construct the placeholder table from
the inline description without copying a literal example.
3. Same file's Plan Mode Safe Operations + Skill Invocation During Plan
Mode sections compressed from ~25 lines combined to ~12. Preserves
all required test phrases (precedence over generic plan mode behavior,
Do not continue the workflow, cancel the skill or leave plan mode,
PLAN MODE EXCEPTION).
NOT touched:
- Voice directive (Garry's voice — protected per CLAUDE.md)
- Office-hours Phase 6 Handoff (Garry's voice + YC pitch)
- Test bootstrap, review army, plan completion (carefully tuned behavior)
Token savings (per skill, system-wide):
ship/SKILL.md 35474 → 34992 tokens (-482)
plan-ceo-review 29436 → 28940 (-496)
office-hours 26700 → 26204 (-496)
Still over the 25K ceiling. Bigger reduction requires restructure
(move large resolvers to externally-referenced docs, split /ship into
ship-quick + ship-full, or refactor the coverage audit + review army
into shorter prose). That's a follow-up — added to TODOS.
Tests: 420/420 pass on gen-skill-docs.test.ts + host-config.test.ts.
Goldens regenerated for claude/codex/factory ship.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): install Node.js from official tarball instead of NodeSource apt setup
The CI Dockerfile's Node install was failing on ubicloud runners. NodeSource's
setup_22.x script runs two internal apt operations that both depend on
archive.ubuntu.com + security.ubuntu.com being reachable:
1. apt-get update (to refresh package lists)
2. apt-get install gnupg (as a prerequisite for its gpg keyring)
Ubicloud's CI runners frequently can't reach those mirrors — last build hit
~2min of connection timeouts to every security.ubuntu.com IP (185.125.190.82,
91.189.91.83, 91.189.92.24, etc.) plus archive.ubuntu.com mirrors. Compounding
this: on Ubuntu 24.04 (noble) "gnupg" was renamed to "gpg" and "gpgconf".
NodeSource's setup script still looks for "gnupg", so even when apt works,
it fails with "Package 'gnupg' has no installation candidate." The subsequent
apt-get install nodejs then fails because the NodeSource repo was never added.
Fix: drop NodeSource entirely. Download Node.js v22.20.0 from nodejs.org as a
tarball, extract to /usr/local. One host, no apt, no script, no keyring.
Before:
RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
&& apt-get install -y --no-install-recommends nodejs ...
After:
ENV NODE_VERSION=22.20.0
RUN curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz" -o /tmp/node.tar.xz \
&& tar -xJ -C /usr/local --strip-components=1 --no-same-owner -f /tmp/node.tar.xz \
&& rm -f /tmp/node.tar.xz \
&& node --version && npm --version
Same installed path (/usr/local/bin/node and npm). Pinned version for
reproducibility. Version is bump-visible in the Dockerfile now.
Does not address the separate apt flakiness that affects the GitHub CLI
install (line 17) or `npx playwright install-deps chromium` (line 33) —
those use apt too. If those fail on a future build we can address then.
Failing job: build-image (71777913820)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: raise skill token ceiling warning from 25K to 40K
The 25K ceiling predated flagship models with 200K-1M windows and assumed
every skill prompt dominates context cost. Modern reality: prompt caching
amortizes the skill load across invocations, and three carefully-tuned
skills (ship, plan-ceo-review, office-hours) legitimately pack 25-35K
tokens of behavior that can't be cut without degrading quality or removing
protected content (Garry's voice, YC pitch, specialist review instructions).
We made the safe prose cuts earlier (coverage diagram, plan status footer,
plan mode operations). The remaining gap is structural — real compression
would require splitting /ship into ship-quick vs ship-full, externalizing
large resolvers to reference docs, or removing detailed skill behavior.
Each is 1-2 days of work. The cost of the warning firing is zero (it's
a warning, not an error). The cost of hitting it is ~15¢ per invocation
at worst, amortized further by prompt caching.
Raising to 40K catches what it's supposed to catch — a runaway 10K+ token
growth in a single release — without crying wolf on legitimately big
skills. Reference doc in CLAUDE.md updated to reflect the new philosophy:
when you hit 40K, ask WHAT grew, don't blindly compress tuned prose.
scripts/gen-skill-docs.ts: TOKEN_CEILING_BYTES 100_000 → 160_000.
CLAUDE.md: document the "watch for feature bloat, not force compression"
intent of the ceiling.
Verification: `bun run gen:skill-docs --host all` shows zero TOKEN
CEILING warnings under the new 40K threshold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): install xz-utils so Node tarball extraction works
The direct-tarball Node install (switched from NodeSource apt in the last
CI fix) failed with "xz: Cannot exec: No such file or directory" because
Ubuntu 24.04 base doesn't include xz-utils. Node ships .tar.xz by default,
and `tar -xJ` shells out to xz, which was missing.
Add xz-utils to the base apt install alongside git/curl/unzip/etc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(benchmark): pass --skip-git-repo-check to codex adapter
The gpt provider adapter spawns `codex exec -C <workdir>` with arbitrary
working directories (benchmark temp dirs, non-git paths). Without
`--skip-git-repo-check`, codex refuses to run and returns "Not inside a
trusted directory" — surfaced as a generic error.code='unknown' that
looks like an API failure.
Benchmarks don't care about codex's git-repo trust model; we just want
the prompt executed. Surfaced by the new provider live E2E test on a
temp workdir.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(benchmark): add --dry-run flag to gstack-model-benchmark
Matches gstack-publish --dry-run semantics. Validates the provider list,
resolves per-adapter auth, echoes the resolved flag values, and exits
without invoking any provider CLI. Zero-cost pre-flight for CI pipelines
and for catching auth drift before starting a paid benchmark run.
Output shape:
== gstack-model-benchmark --dry-run ==
prompt: <truncated>
providers: claude, gpt, gemini
workdir: /tmp/...
timeout_ms: 300000
output: table
judge: off
Adapter availability:
claude: OK
gpt: NOT READY — <reason>
gemini: NOT READY — <reason>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: lite E2E coverage for benchmark, taste engine, publish
Fills real coverage gaps in v0.19.0.0 primitives. 44 new deterministic
tests (gate tier, ~3s) + 8 live-API tests (periodic tier).
New gate-tier test files (free, <3s total):
- test/taste-engine.test.ts — 24 tests against gstack-taste-update:
schema shape, Laplace-smoothed confidence, 5%/week decay clamped at 0,
multi-dimension extraction, case-insensitive matching, session cap,
legacy profile migration with session truncation, taste-drift conflict
warning, malformed-JSON recovery, missing-variant exit code.
- test/publish-dry-run.test.ts — 13 tests against gstack-publish --dry-run:
manifest parsing, missing/malformed JSON, per-skill validation errors
(missing source file / slug / version / marketplaces), slug filter,
unknown-skill exit, per-marketplace auth isolation (fake marketplaces
with always-pass / always-fail / missing-binary CLIs), and a sanity
check against the real repo manifest.
- test/benchmark-cli.test.ts — 11 tests against gstack-model-benchmark
--dry-run: provider default, unknown-provider WARN, empty list
fallback, flag passthrough (timeout/workdir/judge/output), long-prompt
truncation, prompt resolution (inline vs file vs positional), missing
prompt exit.
New periodic-tier test file (paid, gated EVALS=1):
- test/skill-e2e-benchmark-providers.test.ts — 8 tests hitting real
claude, codex, gemini CLIs with a trivial prompt (~$0.001/provider).
Verifies output parsing, token accounting, cost estimation, timeout
error.code semantics, Promise.allSettled parallel isolation.
Per-provider availability gate — unauthed providers skip cleanly.
This suite already caught one real bug (codex adapter missing
--skip-git-repo-check, fixed in 5260987d).
Registered `benchmark-providers-live` in touchfiles.ts (periodic tier,
triggered by changes to bin/gstack-model-benchmark, providers/**,
benchmark-runner.ts, pricing.ts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(benchmark): dedupe providers in --models
`--models claude,claude,gpt` previously produced a list with a duplicate
entry, meaning the benchmark would run claude twice and bill for two
runs. Surfaced by /review on this branch.
Use a Set internally; return Array.from(seen) to preserve type + order
of first occurrence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: /review hardening — NOT-READY env isolation, workdir cleanup, perf
Applied from the adversarial subagent pass during /review on this branch:
- test/benchmark-cli.test.ts — new "NOT READY path fires when auth env
vars are stripped" test. The default dry-run test always showed OK on
dev machines with auth, hiding regressions in the remediation-hint
branch. Stripped env (no auth vars, HOME→empty tmpdir) now force-
exercises gpt + gemini NOT READY paths and asserts every NOT READY
line includes a concrete remediation hint (install/login/export).
(claude adapter's os.homedir() call is Bun-cached; the 2-of-3 adapter
coverage is sufficient to exercise the branch.)
- test/taste-engine.test.ts — session-cap test rewritten to seed the
profile with 50 entries + one real CLI call, instead of 55 sequential
subprocess spawns. Same coverage (FIFO eviction at the boundary), ~5s
faster CI time. Also pins first-casing-wins on the Geist/GEIST merge
assertion — bumpPref() keeps the first-arrival casing, so the test
documents that policy.
- test/skill-e2e-benchmark-providers.test.ts — workdir creation moved
from module-load into beforeAll, cleanup added in afterAll. Previous
shape leaked a /tmp/bench-e2e-* dir every CI run.
- test/publish-dry-run.test.ts — removed unused empty test/helpers
mkdirSync from the sandbox setup. The bin doesn't import from there,
so the empty dir was a footgun for future maintainers.
- test/helpers/providers/gpt.ts — expanded the inline comment on
`--skip-git-repo-check` to explicitly note that `-s read-only` is now
load-bearing safety (the trust prompt was the secondary boundary;
removing read-only while keeping skip-git-repo-check would be unsafe).
Net: 45 passing tests (was 44), session-cap test 5s faster, one real
regression surface covered that didn't exist before.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: surface v0.19 binaries and continuous checkpoint in README
The /review doc-staleness check flagged that v0.19.0.0 ships three new CLIs
(gstack-model-benchmark, gstack-publish, gstack-taste-update) and an opt-in
continuous checkpoint mode, none of which were visible in README's Power
tools section. New users couldn't find them without reading CHANGELOG.
Added:
- "New binaries (v0.19)" subsection with one-row descriptions for each CLI
- "Continuous checkpoint mode (opt-in, local by default)" subsection
explaining WIP auto-commit + [gstack-context] body + /ship squash +
/checkpoint resume
CHANGELOG entry already has good voice from /ship; no polish needed.
VERSION already at 0.19.0.0. Other docs (ARCHITECTURE/CONTRIBUTING/BROWSER)
don't reference this surface — scoped intentionally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ship): Step 19.5 — offer gstack-publish for methodology skill changes
Wires the orphaned gstack-publish binary into /ship. When a PR touches
any standalone methodology skill (openclaw/skills/gstack-*/SKILL.md) or
skills.json, /ship now runs gstack-publish --dry-run after PR creation
and asks the user if they want to actually publish.
Previously, the only way to discover gstack-publish was reading the
CHANGELOG or README. Most methodology skill updates landed on main
without ever being pushed to ClawHub / SkillsMP / Vercel Skills.sh,
defeating the whole point of having a marketplace publisher.
The check is conditional — for PRs that don't touch methodology skills
(the common case), this step is a silent no-op. Dry-run runs first so
the user sees the full list of what would publish and which marketplaces
are authed before committing.
Golden fixtures (claude/codex/factory) regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(benchmark-models): new skill wrapping gstack-model-benchmark
Wires the orphaned gstack-model-benchmark binary into a dedicated skill
so users can discover cross-model benchmarking via /benchmark-models or
voice triggers ("compare models", "which model is best").
Deliberately separate from /benchmark (page performance) because the
two surfaces test completely different things — confusing them would
muddy both.
Flow:
1. Pick a prompt (an existing SKILL.md file, inline text, or file path)
2. Confirm providers (dry-run shows auth status per provider)
3. Decide on --judge (adds ~$0.05, scores output quality 0-10)
4. Run the benchmark — table output
5. Interpret results (fastest / cheapest / highest quality)
6. Offer to save to ~/.gstack/benchmarks/<date>.json for trend tracking
Uses gstack-model-benchmark --dry-run as a safety gate — auth status is
visible BEFORE the user spends API calls. If zero providers are authed,
the skill stops cleanly rather than attempting a run that produces no
useful output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: v1.3.0.0 — complete CHANGELOG + bump for post-1.2 scope additions
VERSION 1.2.0.0 → 1.3.0.0. The original 1.2 entry was written before I
added substantial new scope: the /benchmark-models skill, /ship Step 19.5
gstack-publish integration, --dry-run on gstack-model-benchmark, and the
lite E2E test coverage (4 new test files). A minor bump gives those
changes their own version line instead of silently folding them into
1.2's scope.
CHANGELOG additions under 1.3.0.0:
- /benchmark-models skill (new Added)
- /ship Step 19.5 publish check (new Added)
- gstack-model-benchmark --dry-run (new Added)
- Token ceiling 25K → 40K (moved to Changed)
- New Fixed section — codex adapter --skip-git-repo-check, --models
dedupe, CI Dockerfile xz-utils + nodejs.org tarball
- 4 new test files documented under contributors (taste-engine,
publish-dry-run, benchmark-cli, skill-e2e-benchmark-providers)
- Ship golden fixtures for claude/codex/factory hosts
Pre-existing 1.2 content preserved verbatim — no entries clobbered or
reordered. Sequence remains contiguous (1.3.0.0 → 1.1.3.0 → 1.1.2.0 →
1.1.1.0 → 1.1.0.0 → 1.0.0.0 → 0.19.0.0 → ...).
package.json and VERSION both at 1.3.0.0. No drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: adopt gbrain's release-summary CHANGELOG format + apply to v1.3
Ported the "release-summary format" rules from ~/git/gbrain/CLAUDE.md
(lines 291-354) into gstack's CLAUDE.md under the existing
"CHANGELOG + VERSION style" section. Every future `## [X.Y.Z]` entry
now needs a verdict-style release summary at the top:
1. Two-line bold headline (10-14 words)
2. Lead paragraph (3-5 sentences)
3. "Numbers that matter" with BEFORE / AFTER / Δ table
4. "What this means for [audience]" closer
5. `### Itemized changes` header
6. Existing itemized subsections below
Rewrote v1.3.0.0 entry to match. Preserved every existing bullet in
Added / Changed / Fixed / For contributors (no content clobbered per
the CLAUDE.md CHANGELOG rule).
Numbers in the v1.3 release summary are verifiable — every row of the
BEFORE / AFTER table has a reproducible command listed in the setup
paragraph (git log, bun test, grep for wiring status). No made-up
metrics.
Also added the gbrain "always credit community contributions" rule to
the itemized-changes section. `Contributed by @username` for every
community PR that lands in a CHANGELOG entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove gstack-publish — no real user need
User feedback: "i don't think i would use gstack-publish, i think we
should remove it." Agreed. The CLI + marketplace wiring was an
ambitious but speculative primitive. Zero users, zero validated demand,
and the existing manual `clawhub publish` workflow already covers the
real case (OpenClaw methodology skill publishing).
Deleted:
- bin/gstack-publish (the CLI)
- skills.json (the marketplace manifest)
- test/publish-dry-run.test.ts (13 tests)
- ship/SKILL.md.tmpl Step 19.5 — the methodology-skill publish-on-ship
check. No target to dispatch to anymore.
- README.md Power tools row for gstack-publish
Updated:
- bin/gstack-model-benchmark doc comment: dropped "matches gstack-publish
--dry-run semantics" reference (self-describing flag now)
- CHANGELOG 1.3.0.0 entry:
* Release summary: "three new binaries" → "two new binaries".
Dropped the /ship publish-check narrative.
* Numbers table: "1 of 3 → 3 of 3 wired" → "1 of 2 → 2 of 2 wired".
Deterministic test count: 45 → 32 (removed publish-dry-run's 13).
* Added section: removed gstack-publish CLI bullet + /ship Step 19.5
bullet.
* "What this means for users" closer: replaced the /ship publish
paragraph with the design-taste-engine learning loop, which IS
real, wired, and something users hit every week via /design-shotgun.
* Contributors section: "Four new test files" → "Three new test files"
Retained:
- openclaw/skills/gstack-openclaw-* skill dirs (pre-existed this PR,
still publishable manually via `clawhub publish`, useful standalone
for ClawHub installs)
- CLAUDE.md publishing-native-skills section (same rationale)
Regenerated SKILL.md across all hosts. Ship golden fixtures refreshed
for claude/codex/factory. 455 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(CHANGELOG): reorder v1.3 entry around day-to-day user wins
Previous entry led with internal metrics (CLIs wired to skills, preamble
line count, adapter bugs caught in CI). Useful to contributors, invisible
to users. Rewrote the release summary and Added section to lead with
what a day-to-day gstack user actually experiences.
Release summary changes:
- Headline: "Every new CLI wired to a slash command" → "Your design
skills learn your taste. Your session state survives a laptop close."
- Lead paragraph: shifted from "primitives discoverable from /commands"
to concrete day-to-day wins (design-shotgun taste memory, design-
consultation anti-slop gates, continuous checkpoint survival).
- Numbers table: swapped internal metrics (CLI wiring %, test counts,
preamble line count) for user-visible ones:
- Design-variant convergence gate (0 → 3 axes required)
- AI-slop font blacklist (~8 → 10+ fonts)
- Taste memory across sessions (none → per-project JSON with decay)
- Session state after crash (lost → auto-WIP with structured body)
- /context-restore sources (markdown only → + WIP commits)
- Models with behavioral overlays (1 → 5)
- "Most striking" interpretation: reframed around the mid-session
crash survival story instead of the codex adapter bug catch.
- "What this means" closer: reframed around /design-shotgun + /design-
consultation + continuous checkpoint workflow instead of
/benchmark-models.
Added section — reorganized into six subsections by user value:
1. Design skills that stop looking like AI
(anti-slop constraints, taste engine)
2. Session state that survives a crash
(continuous checkpoint, /context-restore WIP reading,
/ship non-destructive squash)
3. Quality-of-life
(feature discovery prompt, context health soft directive)
4. Cross-host support
(--model flag + 5 overlays)
5. Config
(gstack-config list/defaults, checkpoint_mode/push keys)
6. Power-user / internal
(gstack-model-benchmark + /benchmark-models skill — grouped and
pushed to the bottom since it's more of a research tool than a
daily workflow piece)
Changed / Fixed / For contributors sections unchanged. No content
clobbered per CLAUDE.md CHANGELOG rules — every existing bullet is
preserved, just reordered and grouped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(CHANGELOG): reframe v1.3 entry around transparency vs laptop-close
User feedback: "'closing your laptop' in the changelog is overstated, i
mean claude code does already have session management. i think the use
of the context save restore is mainly just another tool that is more in
your control instead of opaque and a part of CC." Correct. CC handles
session persistence on its own; continuous checkpoint isn't filling a
gap there, it's giving users a parallel, inspectable, portable track.
Reframed every place the old copy overstated:
- Headline: "Your session state survives a laptop close" → "Your
session state lives in git, not a black box."
- Lead paragraph: dropped the "closing your laptop mid-refactor doesn't
vaporize your decisions" line. Now frames continuous checkpoint as
explicitly running alongside CC's built-in session management, not
replacing it. Emphasizes grep-ability, portability across tools and
branches.
- Numbers table row: "Session state after mid-refactor crash: lost
since last manual commit → auto-WIP commits" → "Session state
format: Claude Code's opaque session store → git commits +
[gstack-context] bodies + markdown (parallel track)". Honest about
what's actually changing.
- "Most striking" interpretation: replaced the "used to cost you every
decision" framing with the real user value — session state stops
being a black box, `git log --grep "WIP:"` shows the whole thread,
any tool reading git can see it.
- "What this means" closer: replaced "survives crashes, context
switches, and forgotten laptops" with accurate framing — parallel
track alongside CC's own, inspectable, portable, useful when you
want to review or hand off work.
- Added section: "Session state that survives a crash" subsection
renamed to "Session state you can see, grep, and move". Lead bullet
now explicitly notes continuous checkpoint runs alongside CC session
management, not instead.
No content clobbered. All other bullets and sections unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(CHANGELOG): correct session-state location — home dir by default, git only on opt-in
User correction: "wait is our session management really checked into
git? i don't think that's right, isn't it just saved in your home
dir?" Right. I had the location wrong. The default session-save
mechanism (`/context-save` + `/context-restore`) writes markdown
files to `~/.gstack/projects/$SLUG/checkpoints/` — HOME, not git.
Continuous checkpoint mode (opt-in) is what writes git commits.
Previous copy conflated the two and implied "lives in git" as the
default state, which is wrong.
Every affected location updated:
- Headline: "lives in git, not a black box" → "becomes files you
can grep, not a black box." Removes the false implication that
session state lands in git by default.
- Lead paragraph: now explicitly names the two separate mechanisms.
`/context-save` writes plaintext markdown to `~/.gstack/projects/
$SLUG/checkpoints/` (the default). Continuous checkpoint mode
(opt-in) additionally drops WIP: commits into the git log.
- Numbers table row: "Session state format" now reads "markdown in
`~/.gstack/` by default, plus WIP: git commits if you opt into
continuous mode (parallel track)." Tells the truth about which
path is default vs opt-in.
- "Most striking" row interpretation: now names both paths. Default
path = markdown files in home dir. Opt-in continuous mode = WIP:
commits in project git log. Either way, plain text the user owns.
- "What this means" closer: similarly names both paths explicitly.
"markdown files in your home directory by default, plus git
commits if you opt into continuous mode."
- Continuous checkpoint mode Added bullet: clarifies the commits
land in "your project's git log" (not implied to be the default),
and notes it runs alongside BOTH Claude Code's built-in session
management AND the default `/context-save` markdown flow.
No other bullets or sections touched. No content clobbered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -0,0 +1,579 @@
|
||||
---
|
||||
name: benchmark-models
|
||||
preamble-tier: 1
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Cross-model benchmark for gstack skills. Runs the same prompt through Claude,
|
||||
GPT (via Codex CLI), and Gemini side-by-side — compares latency, tokens, cost,
|
||||
and optionally quality via LLM judge. Answers "which model is actually best
|
||||
for this skill?" with data instead of vibes. Separate from /benchmark, which
|
||||
measures web page performance. Use when: "benchmark models", "compare models",
|
||||
"which model is best for X", "cross-model comparison", "model shootout". (gstack)
|
||||
Voice triggers (speech-to-text aliases): "compare models", "model shootout", "which model is best".
|
||||
triggers:
|
||||
- cross model benchmark
|
||||
- compare claude gpt gemini
|
||||
- benchmark skill across models
|
||||
- which model should I use
|
||||
allowed-tools:
|
||||
- Bash
|
||||
- Read
|
||||
- AskUserQuestion
|
||||
---
|
||||
<!-- AUTO-GENERATED from SKILL.md.tmpl — do not edit directly -->
|
||||
<!-- Regenerate: bun run gen:skill-docs -->
|
||||
|
||||
## Preamble (run first)
|
||||
|
||||
```bash
|
||||
_UPD=$(~/.claude/skills/gstack/bin/gstack-update-check 2>/dev/null || .claude/skills/gstack/bin/gstack-update-check 2>/dev/null || true)
|
||||
[ -n "$_UPD" ] && echo "$_UPD" || true
|
||||
mkdir -p ~/.gstack/sessions
|
||||
touch ~/.gstack/sessions/"$PPID"
|
||||
_SESSIONS=$(find ~/.gstack/sessions -mmin -120 -type f 2>/dev/null | wc -l | tr -d ' ')
|
||||
find ~/.gstack/sessions -mmin +120 -type f -exec rm {} + 2>/dev/null || true
|
||||
_PROACTIVE=$(~/.claude/skills/gstack/bin/gstack-config get proactive 2>/dev/null || echo "true")
|
||||
_PROACTIVE_PROMPTED=$([ -f ~/.gstack/.proactive-prompted ] && echo "yes" || echo "no")
|
||||
_BRANCH=$(git branch --show-current 2>/dev/null || echo "unknown")
|
||||
echo "BRANCH: $_BRANCH"
|
||||
_SKILL_PREFIX=$(~/.claude/skills/gstack/bin/gstack-config get skill_prefix 2>/dev/null || echo "false")
|
||||
echo "PROACTIVE: $_PROACTIVE"
|
||||
echo "PROACTIVE_PROMPTED: $_PROACTIVE_PROMPTED"
|
||||
echo "SKILL_PREFIX: $_SKILL_PREFIX"
|
||||
source <(~/.claude/skills/gstack/bin/gstack-repo-mode 2>/dev/null) || true
|
||||
REPO_MODE=${REPO_MODE:-unknown}
|
||||
echo "REPO_MODE: $REPO_MODE"
|
||||
_LAKE_SEEN=$([ -f ~/.gstack/.completeness-intro-seen ] && echo "yes" || echo "no")
|
||||
echo "LAKE_INTRO: $_LAKE_SEEN"
|
||||
_TEL=$(~/.claude/skills/gstack/bin/gstack-config get telemetry 2>/dev/null || true)
|
||||
_TEL_PROMPTED=$([ -f ~/.gstack/.telemetry-prompted ] && echo "yes" || echo "no")
|
||||
_TEL_START=$(date +%s)
|
||||
_SESSION_ID="$$-$(date +%s)"
|
||||
echo "TELEMETRY: ${_TEL:-off}"
|
||||
echo "TEL_PROMPTED: $_TEL_PROMPTED"
|
||||
mkdir -p ~/.gstack/analytics
|
||||
if [ "$_TEL" != "off" ]; then
|
||||
echo '{"skill":"benchmark-models","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","repo":"'$(basename "$(git rev-parse --show-toplevel 2>/dev/null)" 2>/dev/null || echo "unknown")'"}' >> ~/.gstack/analytics/skill-usage.jsonl 2>/dev/null || true
|
||||
fi
|
||||
# zsh-compatible: use find instead of glob to avoid NOMATCH error
|
||||
for _PF in $(find ~/.gstack/analytics -maxdepth 1 -name '.pending-*' 2>/dev/null); do
|
||||
if [ -f "$_PF" ]; then
|
||||
if [ "$_TEL" != "off" ] && [ -x "~/.claude/skills/gstack/bin/gstack-telemetry-log" ]; then
|
||||
~/.claude/skills/gstack/bin/gstack-telemetry-log --event-type skill_run --skill _pending_finalize --outcome unknown --session-id "$_SESSION_ID" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$_PF" 2>/dev/null || true
|
||||
fi
|
||||
break
|
||||
done
|
||||
# Learnings count
|
||||
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" 2>/dev/null || true
|
||||
_LEARN_FILE="${GSTACK_HOME:-$HOME/.gstack}/projects/${SLUG:-unknown}/learnings.jsonl"
|
||||
if [ -f "$_LEARN_FILE" ]; then
|
||||
_LEARN_COUNT=$(wc -l < "$_LEARN_FILE" 2>/dev/null | tr -d ' ')
|
||||
echo "LEARNINGS: $_LEARN_COUNT entries loaded"
|
||||
if [ "$_LEARN_COUNT" -gt 5 ] 2>/dev/null; then
|
||||
~/.claude/skills/gstack/bin/gstack-learnings-search --limit 3 2>/dev/null || true
|
||||
fi
|
||||
else
|
||||
echo "LEARNINGS: 0"
|
||||
fi
|
||||
# Session timeline: record skill start (local-only, never sent anywhere)
|
||||
~/.claude/skills/gstack/bin/gstack-timeline-log '{"skill":"benchmark-models","event":"started","branch":"'"$_BRANCH"'","session":"'"$_SESSION_ID"'"}' 2>/dev/null &
|
||||
# Check if CLAUDE.md has routing rules
|
||||
_HAS_ROUTING="no"
|
||||
if [ -f CLAUDE.md ] && grep -q "## Skill routing" CLAUDE.md 2>/dev/null; then
|
||||
_HAS_ROUTING="yes"
|
||||
fi
|
||||
_ROUTING_DECLINED=$(~/.claude/skills/gstack/bin/gstack-config get routing_declined 2>/dev/null || echo "false")
|
||||
echo "HAS_ROUTING: $_HAS_ROUTING"
|
||||
echo "ROUTING_DECLINED: $_ROUTING_DECLINED"
|
||||
# Vendoring deprecation: detect if CWD has a vendored gstack copy
|
||||
_VENDORED="no"
|
||||
if [ -d ".claude/skills/gstack" ] && [ ! -L ".claude/skills/gstack" ]; then
|
||||
if [ -f ".claude/skills/gstack/VERSION" ] || [ -d ".claude/skills/gstack/.git" ]; then
|
||||
_VENDORED="yes"
|
||||
fi
|
||||
fi
|
||||
echo "VENDORED_GSTACK: $_VENDORED"
|
||||
echo "MODEL_OVERLAY: claude"
|
||||
# Checkpoint mode (explicit = no auto-commit, continuous = WIP commits as you go)
|
||||
_CHECKPOINT_MODE=$(~/.claude/skills/gstack/bin/gstack-config get checkpoint_mode 2>/dev/null || echo "explicit")
|
||||
_CHECKPOINT_PUSH=$(~/.claude/skills/gstack/bin/gstack-config get checkpoint_push 2>/dev/null || echo "false")
|
||||
echo "CHECKPOINT_MODE: $_CHECKPOINT_MODE"
|
||||
echo "CHECKPOINT_PUSH: $_CHECKPOINT_PUSH"
|
||||
# Detect spawned session (OpenClaw or other orchestrator)
|
||||
[ -n "$OPENCLAW_SESSION" ] && echo "SPAWNED_SESSION: true" || true
|
||||
```
|
||||
|
||||
If `PROACTIVE` is `"false"`, do not proactively suggest gstack skills AND do not
|
||||
auto-invoke skills based on conversation context. Only run skills the user explicitly
|
||||
types (e.g., /qa, /ship). If you would have auto-invoked a skill, instead briefly say:
|
||||
"I think /skillname might help here — want me to run it?" and wait for confirmation.
|
||||
The user opted out of proactive behavior.
|
||||
|
||||
If `SKILL_PREFIX` is `"true"`, the user has namespaced skill names. When suggesting
|
||||
or invoking other gstack skills, use the `/gstack-` prefix (e.g., `/gstack-qa` instead
|
||||
of `/qa`, `/gstack-ship` instead of `/ship`). Disk paths are unaffected — always use
|
||||
`~/.claude/skills/gstack/[skill-name]/SKILL.md` for reading skill files.
|
||||
|
||||
If output shows `UPGRADE_AVAILABLE <old> <new>`: read `~/.claude/skills/gstack/gstack-upgrade/SKILL.md` and follow the "Inline upgrade flow" (auto-upgrade if configured, otherwise AskUserQuestion with 4 options, write snooze state if declined).
|
||||
|
||||
If output shows `JUST_UPGRADED <from> <to>` AND `SPAWNED_SESSION` is NOT set: tell
|
||||
the user "Running gstack v{to} (just updated!)" and then check for new features to
|
||||
surface. For each per-feature marker below, if the marker file is missing AND the
|
||||
feature is plausibly useful for this user, use AskUserQuestion to let them try it.
|
||||
Fire once per feature per user, NOT once per upgrade.
|
||||
|
||||
**In spawned sessions (`SPAWNED_SESSION` = "true"): SKIP feature discovery entirely.**
|
||||
Just print "Running gstack v{to}" and continue. Orchestrators do not want interactive
|
||||
prompts from sub-sessions.
|
||||
|
||||
**Feature discovery markers and prompts** (one at a time, max one per session):
|
||||
|
||||
1. `~/.claude/skills/gstack/.feature-prompted-continuous-checkpoint` →
|
||||
Prompt: "Continuous checkpoint auto-commits your work as you go with `WIP:` prefix
|
||||
so you never lose progress to a crash. Local-only by default — doesn't push
|
||||
anywhere unless you turn that on. Want to try it?"
|
||||
Options: A) Enable continuous mode, B) Show me first (print the section from
|
||||
the preamble Continuous Checkpoint Mode), C) Skip.
|
||||
If A: run `~/.claude/skills/gstack/bin/gstack-config set checkpoint_mode continuous`.
|
||||
Always: `touch ~/.claude/skills/gstack/.feature-prompted-continuous-checkpoint`
|
||||
|
||||
2. `~/.claude/skills/gstack/.feature-prompted-model-overlay` →
|
||||
Inform only (no prompt): "Model overlays are active. `MODEL_OVERLAY: {model}`
|
||||
shown in the preamble output tells you which behavioral patch is applied.
|
||||
Override with `--model` when regenerating skills (e.g., `bun run gen:skill-docs
|
||||
--model gpt-5.4`). Default is claude."
|
||||
Always: `touch ~/.claude/skills/gstack/.feature-prompted-model-overlay`
|
||||
|
||||
After handling JUST_UPGRADED (prompts done or skipped), continue with the skill
|
||||
workflow.
|
||||
|
||||
If `WRITING_STYLE_PENDING` is `yes`: You're on the first skill run after upgrading
|
||||
to gstack v1. Ask the user once about the new default writing style. Use AskUserQuestion:
|
||||
|
||||
> v1 prompts = simpler. Technical terms get a one-sentence gloss on first use,
|
||||
> questions are framed in outcome terms, sentences are shorter.
|
||||
>
|
||||
> Keep the new default, or prefer the older tighter prose?
|
||||
|
||||
Options:
|
||||
- A) Keep the new default (recommended — good writing helps everyone)
|
||||
- B) Restore V0 prose — set `explain_level: terse`
|
||||
|
||||
If A: leave `explain_level` unset (defaults to `default`).
|
||||
If B: run `~/.claude/skills/gstack/bin/gstack-config set explain_level terse`.
|
||||
|
||||
Always run (regardless of choice):
|
||||
```bash
|
||||
rm -f ~/.gstack/.writing-style-prompt-pending
|
||||
touch ~/.gstack/.writing-style-prompted
|
||||
```
|
||||
|
||||
This only happens once. If `WRITING_STYLE_PENDING` is `no`, skip this entirely.
|
||||
|
||||
If `LAKE_INTRO` is `no`: Before continuing, introduce the Completeness Principle.
|
||||
Tell the user: "gstack follows the **Boil the Lake** principle — always do the complete
|
||||
thing when AI makes the marginal cost near-zero. Read more: https://garryslist.org/posts/boil-the-ocean"
|
||||
Then offer to open the essay in their default browser:
|
||||
|
||||
```bash
|
||||
open https://garryslist.org/posts/boil-the-ocean
|
||||
touch ~/.gstack/.completeness-intro-seen
|
||||
```
|
||||
|
||||
Only run `open` if the user says yes. Always run `touch` to mark as seen. This only happens once.
|
||||
|
||||
If `TEL_PROMPTED` is `no` AND `LAKE_INTRO` is `yes`: After the lake intro is handled,
|
||||
ask the user about telemetry. Use AskUserQuestion:
|
||||
|
||||
> Help gstack get better! Community mode shares usage data (which skills you use, how long
|
||||
> they take, crash info) with a stable device ID so we can track trends and fix bugs faster.
|
||||
> No code, file paths, or repo names are ever sent.
|
||||
> Change anytime with `gstack-config set telemetry off`.
|
||||
|
||||
Options:
|
||||
- A) Help gstack get better! (recommended)
|
||||
- B) No thanks
|
||||
|
||||
If A: run `~/.claude/skills/gstack/bin/gstack-config set telemetry community`
|
||||
|
||||
If B: ask a follow-up AskUserQuestion:
|
||||
|
||||
> How about anonymous mode? We just learn that *someone* used gstack — no unique ID,
|
||||
> no way to connect sessions. Just a counter that helps us know if anyone's out there.
|
||||
|
||||
Options:
|
||||
- A) Sure, anonymous is fine
|
||||
- B) No thanks, fully off
|
||||
|
||||
If B→A: run `~/.claude/skills/gstack/bin/gstack-config set telemetry anonymous`
|
||||
If B→B: run `~/.claude/skills/gstack/bin/gstack-config set telemetry off`
|
||||
|
||||
Always run:
|
||||
```bash
|
||||
touch ~/.gstack/.telemetry-prompted
|
||||
```
|
||||
|
||||
This only happens once. If `TEL_PROMPTED` is `yes`, skip this entirely.
|
||||
|
||||
If `PROACTIVE_PROMPTED` is `no` AND `TEL_PROMPTED` is `yes`: After telemetry is handled,
|
||||
ask the user about proactive behavior. Use AskUserQuestion:
|
||||
|
||||
> gstack can proactively figure out when you might need a skill while you work —
|
||||
> like suggesting /qa when you say "does this work?" or /investigate when you hit
|
||||
> a bug. We recommend keeping this on — it speeds up every part of your workflow.
|
||||
|
||||
Options:
|
||||
- A) Keep it on (recommended)
|
||||
- B) Turn it off — I'll type /commands myself
|
||||
|
||||
If A: run `~/.claude/skills/gstack/bin/gstack-config set proactive true`
|
||||
If B: run `~/.claude/skills/gstack/bin/gstack-config set proactive false`
|
||||
|
||||
Always run:
|
||||
```bash
|
||||
touch ~/.gstack/.proactive-prompted
|
||||
```
|
||||
|
||||
This only happens once. If `PROACTIVE_PROMPTED` is `yes`, skip this entirely.
|
||||
|
||||
If `HAS_ROUTING` is `no` AND `ROUTING_DECLINED` is `false` AND `PROACTIVE_PROMPTED` is `yes`:
|
||||
Check if a CLAUDE.md file exists in the project root. If it does not exist, create it.
|
||||
|
||||
Use AskUserQuestion:
|
||||
|
||||
> gstack works best when your project's CLAUDE.md includes skill routing rules.
|
||||
> This tells Claude to use specialized workflows (like /ship, /investigate, /qa)
|
||||
> instead of answering directly. It's a one-time addition, about 15 lines.
|
||||
|
||||
Options:
|
||||
- A) Add routing rules to CLAUDE.md (recommended)
|
||||
- B) No thanks, I'll invoke skills manually
|
||||
|
||||
If A: Append this section to the end of CLAUDE.md:
|
||||
|
||||
```markdown
|
||||
|
||||
## Skill routing
|
||||
|
||||
When the user's request matches an available skill, ALWAYS invoke it using the Skill
|
||||
tool as your FIRST action. Do NOT answer directly, do NOT use other tools first.
|
||||
The skill has specialized workflows that produce better results than ad-hoc answers.
|
||||
|
||||
Key routing rules:
|
||||
- Product ideas, "is this worth building", brainstorming → invoke office-hours
|
||||
- Bugs, errors, "why is this broken", 500 errors → invoke investigate
|
||||
- Ship, deploy, push, create PR → invoke ship
|
||||
- QA, test the site, find bugs → invoke qa
|
||||
- Code review, check my diff → invoke review
|
||||
- Update docs after shipping → invoke document-release
|
||||
- Weekly retro → invoke retro
|
||||
- Design system, brand → invoke design-consultation
|
||||
- Visual audit, design polish → invoke design-review
|
||||
- Architecture review → invoke plan-eng-review
|
||||
- Save progress, checkpoint, resume → invoke checkpoint
|
||||
- Code quality, health check → invoke health
|
||||
```
|
||||
|
||||
Then commit the change: `git add CLAUDE.md && git commit -m "chore: add gstack skill routing rules to CLAUDE.md"`
|
||||
|
||||
If B: run `~/.claude/skills/gstack/bin/gstack-config set routing_declined true`
|
||||
Say "No problem. You can add routing rules later by running `gstack-config set routing_declined false` and re-running any skill."
|
||||
|
||||
This only happens once per project. If `HAS_ROUTING` is `yes` or `ROUTING_DECLINED` is `true`, skip this entirely.
|
||||
|
||||
If `VENDORED_GSTACK` is `yes`: This project has a vendored copy of gstack at
|
||||
`.claude/skills/gstack/`. Vendoring is deprecated. We will not keep vendored copies
|
||||
up to date, so this project's gstack will fall behind.
|
||||
|
||||
Use AskUserQuestion (one-time per project, check for `~/.gstack/.vendoring-warned-$SLUG` marker):
|
||||
|
||||
> This project has gstack vendored in `.claude/skills/gstack/`. Vendoring is deprecated.
|
||||
> We won't keep this copy up to date, so you'll fall behind on new features and fixes.
|
||||
>
|
||||
> Want to migrate to team mode? It takes about 30 seconds.
|
||||
|
||||
Options:
|
||||
- A) Yes, migrate to team mode now
|
||||
- B) No, I'll handle it myself
|
||||
|
||||
If A:
|
||||
1. Run `git rm -r .claude/skills/gstack/`
|
||||
2. Run `echo '.claude/skills/gstack/' >> .gitignore`
|
||||
3. Run `~/.claude/skills/gstack/bin/gstack-team-init required` (or `optional`)
|
||||
4. Run `git add .claude/ .gitignore CLAUDE.md && git commit -m "chore: migrate gstack from vendored to team mode"`
|
||||
5. Tell the user: "Done. Each developer now runs: `cd ~/.claude/skills/gstack && ./setup --team`"
|
||||
|
||||
If B: say "OK, you're on your own to keep the vendored copy up to date."
|
||||
|
||||
Always run (regardless of choice):
|
||||
```bash
|
||||
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" 2>/dev/null || true
|
||||
touch ~/.gstack/.vendoring-warned-${SLUG:-unknown}
|
||||
```
|
||||
|
||||
This only happens once per project. If the marker file exists, skip entirely.
|
||||
|
||||
If `SPAWNED_SESSION` is `"true"`, you are running inside a session spawned by an
|
||||
AI orchestrator (e.g., OpenClaw). In spawned sessions:
|
||||
- Do NOT use AskUserQuestion for interactive prompts. Auto-choose the recommended option.
|
||||
- Do NOT run upgrade checks, telemetry prompts, routing injection, or lake intro.
|
||||
- Focus on completing the task and reporting results via prose output.
|
||||
- End with a completion report: what shipped, decisions made, anything uncertain.
|
||||
|
||||
## Model-Specific Behavioral Patch (claude)
|
||||
|
||||
The following nudges are tuned for the claude model family. They are
|
||||
**subordinate** to skill workflow, STOP points, AskUserQuestion gates, plan-mode
|
||||
safety, and /ship review gates. If a nudge below conflicts with skill instructions,
|
||||
the skill wins. Treat these as preferences, not rules.
|
||||
|
||||
**Todo-list discipline.** When working through a multi-step plan, mark each task
|
||||
complete individually as you finish it. Do not batch-complete at the end. If a task
|
||||
turns out to be unnecessary, mark it skipped with a one-line reason.
|
||||
|
||||
**Think before heavy actions.** For complex operations (refactors, migrations,
|
||||
non-trivial new features), briefly state your approach before executing. This lets
|
||||
the user course-correct cheaply instead of mid-flight.
|
||||
|
||||
**Dedicated tools over Bash.** Prefer Read, Edit, Write, Glob, Grep over shell
|
||||
equivalents (cat, sed, find, grep). The dedicated tools are cheaper and clearer.
|
||||
|
||||
## Voice
|
||||
|
||||
**Tone:** direct, concrete, sharp, never corporate, never academic. Sound like a builder, not a consultant. Name the file, the function, the command. No filler, no throat-clearing.
|
||||
|
||||
**Writing rules:** No em dashes (use commas, periods, "..."). No AI vocabulary (delve, crucial, robust, comprehensive, nuanced, etc.). Short paragraphs. End with what to do.
|
||||
|
||||
The user always has context you don't. Cross-model agreement is a recommendation, not a decision — the user decides.
|
||||
|
||||
## Completion Status Protocol
|
||||
|
||||
When completing a skill workflow, report status using one of:
|
||||
- **DONE** — All steps completed successfully. Evidence provided for each claim.
|
||||
- **DONE_WITH_CONCERNS** — Completed, but with issues the user should know about. List each concern.
|
||||
- **BLOCKED** — Cannot proceed. State what is blocking and what was tried.
|
||||
- **NEEDS_CONTEXT** — Missing information required to continue. State exactly what you need.
|
||||
|
||||
### Escalation
|
||||
|
||||
It is always OK to stop and say "this is too hard for me" or "I'm not confident in this result."
|
||||
|
||||
Bad work is worse than no work. You will not be penalized for escalating.
|
||||
- If you have attempted a task 3 times without success, STOP and escalate.
|
||||
- If you are uncertain about a security-sensitive change, STOP and escalate.
|
||||
- If the scope of work exceeds what you can verify, STOP and escalate.
|
||||
|
||||
Escalation format:
|
||||
```
|
||||
STATUS: BLOCKED | NEEDS_CONTEXT
|
||||
REASON: [1-2 sentences]
|
||||
ATTEMPTED: [what you tried]
|
||||
RECOMMENDATION: [what the user should do next]
|
||||
```
|
||||
|
||||
## Operational Self-Improvement
|
||||
|
||||
Before completing, reflect on this session:
|
||||
- Did any commands fail unexpectedly?
|
||||
- Did you take a wrong approach and have to backtrack?
|
||||
- Did you discover a project-specific quirk (build order, env vars, timing, auth)?
|
||||
- Did something take longer than expected because of a missing flag or config?
|
||||
|
||||
If yes, log an operational learning for future sessions:
|
||||
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-learnings-log '{"skill":"SKILL_NAME","type":"operational","key":"SHORT_KEY","insight":"DESCRIPTION","confidence":N,"source":"observed"}'
|
||||
```
|
||||
|
||||
Replace SKILL_NAME with the current skill name. Only log genuine operational discoveries.
|
||||
Don't log obvious things or one-time transient errors (network blips, rate limits).
|
||||
A good test: would knowing this save 5+ minutes in a future session? If yes, log it.
|
||||
|
||||
## Telemetry (run last)
|
||||
|
||||
After the skill workflow completes (success, error, or abort), log the telemetry event.
|
||||
Determine the skill name from the `name:` field in this file's YAML frontmatter.
|
||||
Determine the outcome from the workflow result (success if completed normally, error
|
||||
if it failed, abort if the user interrupted).
|
||||
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** This command writes telemetry to
|
||||
`~/.gstack/analytics/` (user config directory, not project files). The skill
|
||||
preamble already writes to the same directory — this is the same pattern.
|
||||
Skipping this command loses session duration and outcome data.
|
||||
|
||||
Run this bash:
|
||||
|
||||
```bash
|
||||
_TEL_END=$(date +%s)
|
||||
_TEL_DUR=$(( _TEL_END - _TEL_START ))
|
||||
rm -f ~/.gstack/analytics/.pending-"$_SESSION_ID" 2>/dev/null || true
|
||||
# Session timeline: record skill completion (local-only, never sent anywhere)
|
||||
~/.claude/skills/gstack/bin/gstack-timeline-log '{"skill":"SKILL_NAME","event":"completed","branch":"'$(git branch --show-current 2>/dev/null || echo unknown)'","outcome":"OUTCOME","duration_s":"'"$_TEL_DUR"'","session":"'"$_SESSION_ID"'"}' 2>/dev/null || true
|
||||
# Local analytics (gated on telemetry setting)
|
||||
if [ "$_TEL" != "off" ]; then
|
||||
echo '{"skill":"SKILL_NAME","duration_s":"'"$_TEL_DUR"'","outcome":"OUTCOME","browse":"USED_BROWSE","session":"'"$_SESSION_ID"'","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}' >> ~/.gstack/analytics/skill-usage.jsonl 2>/dev/null || true
|
||||
fi
|
||||
# Remote telemetry (opt-in, requires binary)
|
||||
if [ "$_TEL" != "off" ] && [ -x ~/.claude/skills/gstack/bin/gstack-telemetry-log ]; then
|
||||
~/.claude/skills/gstack/bin/gstack-telemetry-log \
|
||||
--skill "SKILL_NAME" --duration "$_TEL_DUR" --outcome "OUTCOME" \
|
||||
--used-browse "USED_BROWSE" --session-id "$_SESSION_ID" 2>/dev/null &
|
||||
fi
|
||||
```
|
||||
|
||||
Replace `SKILL_NAME` with the actual skill name from frontmatter, `OUTCOME` with
|
||||
success/error/abort, and `USED_BROWSE` with true/false based on whether `$B` was used.
|
||||
If you cannot determine the outcome, use "unknown". The local JSONL always logs. The
|
||||
remote binary only runs if telemetry is not off and the binary exists.
|
||||
|
||||
## Plan Mode Safe Operations
|
||||
|
||||
In plan mode, these are always allowed (they inform the plan, don't modify source):
|
||||
`$B` (browse), `$D` (design), `codex exec`/`codex review`, writes to `~/.gstack/`,
|
||||
writes to the plan file, `open` for generated artifacts.
|
||||
|
||||
## Skill Invocation During Plan Mode
|
||||
|
||||
If the user invokes a skill in plan mode, that skill takes precedence over generic plan mode behavior. Treat it as executable instructions, not reference. Follow step
|
||||
by step. AskUserQuestion calls satisfy plan mode's end-of-turn requirement. At a STOP
|
||||
point, stop immediately. Do not continue the workflow past a STOP point and do not call ExitPlanMode there. Commands marked "PLAN
|
||||
MODE EXCEPTION — ALWAYS RUN" execute. Other writes need to be already permitted
|
||||
above or explicitly exception-marked. Call ExitPlanMode only after the skill
|
||||
workflow completes — only then call ExitPlanMode (or if the user tells you to cancel the skill or leave plan mode).
|
||||
|
||||
## Plan Status Footer
|
||||
|
||||
In plan mode, before ExitPlanMode: if the plan file lacks a `## GSTACK REVIEW REPORT`
|
||||
section, run `~/.claude/skills/gstack/bin/gstack-review-read` and append a report.
|
||||
With JSONL entries (before `---CONFIG---`), format the standard runs/status/findings
|
||||
table. With `NO_REVIEWS` or empty, append a 5-row placeholder table (CEO/Codex/Eng/
|
||||
Design/DX Review) with all zeros and verdict "NO REVIEWS YET — run `/autoplan`".
|
||||
If a richer review report already exists, skip — review skills wrote it.
|
||||
|
||||
PLAN MODE EXCEPTION — always allowed (it's the plan file).
|
||||
|
||||
# /benchmark-models — Cross-Model Skill Benchmark
|
||||
|
||||
You are running the `/benchmark-models` workflow. Wraps the `gstack-model-benchmark` binary with an interactive flow that picks a prompt, confirms providers, previews auth, and runs the benchmark.
|
||||
|
||||
Different from `/benchmark` — that skill measures web page performance (Core Web Vitals, load times). This skill measures AI model performance on gstack skills or arbitrary prompts.
|
||||
|
||||
---
|
||||
|
||||
## Step 0: Locate the binary
|
||||
|
||||
```bash
|
||||
BIN="$HOME/.claude/skills/gstack/bin/gstack-model-benchmark"
|
||||
[ -x "$BIN" ] || BIN=".claude/skills/gstack/bin/gstack-model-benchmark"
|
||||
[ -x "$BIN" ] || { echo "ERROR: gstack-model-benchmark not found. Run ./setup in the gstack install dir." >&2; exit 1; }
|
||||
echo "BIN: $BIN"
|
||||
```
|
||||
|
||||
If not found, stop and tell the user to reinstall gstack.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Choose a prompt
|
||||
|
||||
Use AskUserQuestion with the preamble format:
|
||||
- **Re-ground:** current project + branch.
|
||||
- **Simplify:** "A cross-model benchmark runs the same prompt through 2-3 AI models and shows you how they compare on speed, cost, and output quality. What prompt should we use?"
|
||||
- **RECOMMENDATION:** A because benchmarking against a real skill exposes tool-use differences, not just raw generation.
|
||||
- **Options:**
|
||||
- A) Benchmark one of my gstack skills (we'll pick which skill next). Completeness: 10/10.
|
||||
- B) Use an inline prompt — type it on the next turn. Completeness: 8/10.
|
||||
- C) Point at a prompt file on disk — specify path on the next turn. Completeness: 8/10.
|
||||
|
||||
If A: list top-level gstack skills that have SKILL.md files (from `find . -maxdepth 2 -name SKILL.md -not -path './.*'`), ask the user to pick one via a second AskUserQuestion. Use the picked SKILL.md path as the prompt file.
|
||||
|
||||
If B: ask the user for the inline prompt. Use it verbatim via `--prompt "<text>"`.
|
||||
|
||||
If C: ask for the path. Verify it exists. Use as positional argument.
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Choose providers
|
||||
|
||||
```bash
|
||||
"$BIN" --prompt "unused, dry-run" --models claude,gpt,gemini --dry-run
|
||||
```
|
||||
|
||||
Show the dry-run output. The "Adapter availability" section tells the user which providers will actually run (OK) vs skip (NOT READY — remediation hint included).
|
||||
|
||||
If ALL three show NOT READY: stop with a clear message — benchmark can't run without at least one authed provider. Suggest `claude login`, `codex login`, or `gemini login` / `export GOOGLE_API_KEY`.
|
||||
|
||||
If at least one is OK: AskUserQuestion:
|
||||
- **Simplify:** "Which models should we include? The dry-run above showed which are authed. Unauthed ones will be skipped cleanly — they won't abort the batch."
|
||||
- **RECOMMENDATION:** A (all authed providers) because running as many as possible gives the richest comparison.
|
||||
- **Options:**
|
||||
- A) All authed providers. Completeness: 10/10.
|
||||
- B) Only Claude. Completeness: 6/10 (no cross-model signal — use /ship's review for solo claude benchmarks instead).
|
||||
- C) Pick two — specify on next turn. Completeness: 8/10.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Decide on judge
|
||||
|
||||
```bash
|
||||
[ -n "$ANTHROPIC_API_KEY" ] || grep -q 'ANTHROPIC' "$HOME/.claude/.credentials.json" 2>/dev/null && echo "JUDGE_AVAILABLE" || echo "JUDGE_UNAVAILABLE"
|
||||
```
|
||||
|
||||
If judge is available, AskUserQuestion:
|
||||
- **Simplify:** "The quality judge scores each model's output on a 0-10 scale using Anthropic's Claude as a tiebreaker. Adds ~$0.05/run. Recommended if you care about output quality, not just latency and cost."
|
||||
- **RECOMMENDATION:** A — the whole point is comparing quality, not just speed.
|
||||
- **Options:**
|
||||
- A) Enable judge (adds ~$0.05). Completeness: 10/10.
|
||||
- B) Skip judge — speed/cost/tokens only. Completeness: 7/10.
|
||||
|
||||
If judge is NOT available, skip this question and omit the `--judge` flag.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Run the benchmark
|
||||
|
||||
Construct the command from Step 1, 2, 3 decisions:
|
||||
|
||||
```bash
|
||||
"$BIN" <prompt-spec> --models <picked-models> [--judge] --output table
|
||||
```
|
||||
|
||||
Where `<prompt-spec>` is either `--prompt "<text>"` (Step 1B), a file path (Step 1A or 1C), and `<picked-models>` is the comma-separated list from Step 2.
|
||||
|
||||
Stream the output as it arrives. This is slow — each provider runs the prompt fully. Expect 30s-5min depending on prompt complexity and whether `--judge` is on.
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Interpret results
|
||||
|
||||
After the table prints, summarize for the user:
|
||||
- **Fastest** — provider with lowest latency.
|
||||
- **Cheapest** — provider with lowest cost.
|
||||
- **Highest quality** (if `--judge` ran) — provider with highest score.
|
||||
- **Best overall** — use judgment. If judge ran: quality-weighted. Otherwise: note the tradeoff the user needs to make.
|
||||
|
||||
If any provider hit an error (auth/timeout/rate_limit), call it out with the remediation path.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Offer to save results
|
||||
|
||||
AskUserQuestion:
|
||||
- **Simplify:** "Save this benchmark as JSON so you can compare future runs against it?"
|
||||
- **RECOMMENDATION:** A — skill performance drifts as providers update their models; a saved baseline catches quality regressions.
|
||||
- **Options:**
|
||||
- A) Save to `~/.gstack/benchmarks/<date>-<skill-or-prompt-slug>.json`. Completeness: 10/10.
|
||||
- B) Just print, don't save. Completeness: 5/10 (loses trend data).
|
||||
|
||||
If A: re-run with `--output json` and tee to the dated file. Print the path so the user can diff future runs against it.
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
- **Never run a real benchmark without Step 2's dry-run first.** Users need to see auth status before spending API calls.
|
||||
- **Never hardcode model names.** Always pass providers from user's Step 2 choice — the binary handles the rest.
|
||||
- **Never auto-include `--judge`.** It adds real cost; user must opt in.
|
||||
- **If zero providers are authed, STOP.** Don't attempt the benchmark — it produces no useful output.
|
||||
- **Cost is visible.** Every run shows per-provider cost in the table. Users should see it before the next run.
|
||||
@@ -0,0 +1,151 @@
|
||||
---
|
||||
name: benchmark-models
|
||||
preamble-tier: 1
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Cross-model benchmark for gstack skills. Runs the same prompt through Claude,
|
||||
GPT (via Codex CLI), and Gemini side-by-side — compares latency, tokens, cost,
|
||||
and optionally quality via LLM judge. Answers "which model is actually best
|
||||
for this skill?" with data instead of vibes. Separate from /benchmark, which
|
||||
measures web page performance. Use when: "benchmark models", "compare models",
|
||||
"which model is best for X", "cross-model comparison", "model shootout". (gstack)
|
||||
voice-triggers:
|
||||
- "compare models"
|
||||
- "model shootout"
|
||||
- "which model is best"
|
||||
triggers:
|
||||
- cross model benchmark
|
||||
- compare claude gpt gemini
|
||||
- benchmark skill across models
|
||||
- which model should I use
|
||||
allowed-tools:
|
||||
- Bash
|
||||
- Read
|
||||
- AskUserQuestion
|
||||
---
|
||||
|
||||
{{PREAMBLE}}
|
||||
|
||||
# /benchmark-models — Cross-Model Skill Benchmark
|
||||
|
||||
You are running the `/benchmark-models` workflow. Wraps the `gstack-model-benchmark` binary with an interactive flow that picks a prompt, confirms providers, previews auth, and runs the benchmark.
|
||||
|
||||
Different from `/benchmark` — that skill measures web page performance (Core Web Vitals, load times). This skill measures AI model performance on gstack skills or arbitrary prompts.
|
||||
|
||||
---
|
||||
|
||||
## Step 0: Locate the binary
|
||||
|
||||
```bash
|
||||
BIN="$HOME/.claude/skills/gstack/bin/gstack-model-benchmark"
|
||||
[ -x "$BIN" ] || BIN=".claude/skills/gstack/bin/gstack-model-benchmark"
|
||||
[ -x "$BIN" ] || { echo "ERROR: gstack-model-benchmark not found. Run ./setup in the gstack install dir." >&2; exit 1; }
|
||||
echo "BIN: $BIN"
|
||||
```
|
||||
|
||||
If not found, stop and tell the user to reinstall gstack.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Choose a prompt
|
||||
|
||||
Use AskUserQuestion with the preamble format:
|
||||
- **Re-ground:** current project + branch.
|
||||
- **Simplify:** "A cross-model benchmark runs the same prompt through 2-3 AI models and shows you how they compare on speed, cost, and output quality. What prompt should we use?"
|
||||
- **RECOMMENDATION:** A because benchmarking against a real skill exposes tool-use differences, not just raw generation.
|
||||
- **Options:**
|
||||
- A) Benchmark one of my gstack skills (we'll pick which skill next). Completeness: 10/10.
|
||||
- B) Use an inline prompt — type it on the next turn. Completeness: 8/10.
|
||||
- C) Point at a prompt file on disk — specify path on the next turn. Completeness: 8/10.
|
||||
|
||||
If A: list top-level gstack skills that have SKILL.md files (from `find . -maxdepth 2 -name SKILL.md -not -path './.*'`), ask the user to pick one via a second AskUserQuestion. Use the picked SKILL.md path as the prompt file.
|
||||
|
||||
If B: ask the user for the inline prompt. Use it verbatim via `--prompt "<text>"`.
|
||||
|
||||
If C: ask for the path. Verify it exists. Use as positional argument.
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Choose providers
|
||||
|
||||
```bash
|
||||
"$BIN" --prompt "unused, dry-run" --models claude,gpt,gemini --dry-run
|
||||
```
|
||||
|
||||
Show the dry-run output. The "Adapter availability" section tells the user which providers will actually run (OK) vs skip (NOT READY — remediation hint included).
|
||||
|
||||
If ALL three show NOT READY: stop with a clear message — benchmark can't run without at least one authed provider. Suggest `claude login`, `codex login`, or `gemini login` / `export GOOGLE_API_KEY`.
|
||||
|
||||
If at least one is OK: AskUserQuestion:
|
||||
- **Simplify:** "Which models should we include? The dry-run above showed which are authed. Unauthed ones will be skipped cleanly — they won't abort the batch."
|
||||
- **RECOMMENDATION:** A (all authed providers) because running as many as possible gives the richest comparison.
|
||||
- **Options:**
|
||||
- A) All authed providers. Completeness: 10/10.
|
||||
- B) Only Claude. Completeness: 6/10 (no cross-model signal — use /ship's review for solo claude benchmarks instead).
|
||||
- C) Pick two — specify on next turn. Completeness: 8/10.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Decide on judge
|
||||
|
||||
```bash
|
||||
[ -n "$ANTHROPIC_API_KEY" ] || grep -q 'ANTHROPIC' "$HOME/.claude/.credentials.json" 2>/dev/null && echo "JUDGE_AVAILABLE" || echo "JUDGE_UNAVAILABLE"
|
||||
```
|
||||
|
||||
If judge is available, AskUserQuestion:
|
||||
- **Simplify:** "The quality judge scores each model's output on a 0-10 scale using Anthropic's Claude as a tiebreaker. Adds ~$0.05/run. Recommended if you care about output quality, not just latency and cost."
|
||||
- **RECOMMENDATION:** A — the whole point is comparing quality, not just speed.
|
||||
- **Options:**
|
||||
- A) Enable judge (adds ~$0.05). Completeness: 10/10.
|
||||
- B) Skip judge — speed/cost/tokens only. Completeness: 7/10.
|
||||
|
||||
If judge is NOT available, skip this question and omit the `--judge` flag.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Run the benchmark
|
||||
|
||||
Construct the command from Step 1, 2, 3 decisions:
|
||||
|
||||
```bash
|
||||
"$BIN" <prompt-spec> --models <picked-models> [--judge] --output table
|
||||
```
|
||||
|
||||
Where `<prompt-spec>` is either `--prompt "<text>"` (Step 1B), a file path (Step 1A or 1C), and `<picked-models>` is the comma-separated list from Step 2.
|
||||
|
||||
Stream the output as it arrives. This is slow — each provider runs the prompt fully. Expect 30s-5min depending on prompt complexity and whether `--judge` is on.
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Interpret results
|
||||
|
||||
After the table prints, summarize for the user:
|
||||
- **Fastest** — provider with lowest latency.
|
||||
- **Cheapest** — provider with lowest cost.
|
||||
- **Highest quality** (if `--judge` ran) — provider with highest score.
|
||||
- **Best overall** — use judgment. If judge ran: quality-weighted. Otherwise: note the tradeoff the user needs to make.
|
||||
|
||||
If any provider hit an error (auth/timeout/rate_limit), call it out with the remediation path.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Offer to save results
|
||||
|
||||
AskUserQuestion:
|
||||
- **Simplify:** "Save this benchmark as JSON so you can compare future runs against it?"
|
||||
- **RECOMMENDATION:** A — skill performance drifts as providers update their models; a saved baseline catches quality regressions.
|
||||
- **Options:**
|
||||
- A) Save to `~/.gstack/benchmarks/<date>-<skill-or-prompt-slug>.json`. Completeness: 10/10.
|
||||
- B) Just print, don't save. Completeness: 5/10 (loses trend data).
|
||||
|
||||
If A: re-run with `--output json` and tee to the dated file. Print the path so the user can diff future runs against it.
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
- **Never run a real benchmark without Step 2's dry-run first.** Users need to see auth status before spending API calls.
|
||||
- **Never hardcode model names.** Always pass providers from user's Step 2 choice — the binary handles the rest.
|
||||
- **Never auto-include `--judge`.** It adds real cost; user must opt in.
|
||||
- **If zero providers are authed, STOP.** Don't attempt the benchmark — it produces no useful output.
|
||||
- **Cost is visible.** Every run shows per-provider cost in the table. Users should see it before the next run.
|
||||
Reference in New Issue
Block a user