* feat(retro): factory scorecard bin (lead time, gate cost, red-train and flaky rates) and the recommended-harness adoption doc Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(scorecard): register the recorded gh fixture with its free consumer Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(scorecard): cover --gate-event any, an empty window, a failed PR read and a bad --now Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes Review pass 1 (core checklist, testing, maintainability, security, performance, simplification, state-machines, red team) over the whole diff: - the flaky line and cost-per-merge depend on every input they read, so a failed free-run or PR read prints unavailable (...) on its half instead of 0 - a PR verdict-comment cost backs one gate run per PR; a READY comment after the merge is ignored - one unreadable artifact leaves its run without cost or flake data instead of dropping the gate section - --repo defaults to the origin owner/name and is required without one (no built-in repository) - --max-artifact-runs, --pass-cap and --pass-marker are validated; --csv creates its directory - one artifact listing per gate run; PR comments are paged; issues use the exact window instant - every run appends its rows to trend.csv in --out-dir - new row: review passes per PR (median, p90, PRs at the cap) from the ship receipt or a comment marker - /retro: report format gains the Factory scorecard slot; the Step 7.5 fence quotes its placeholder - tests for each; <window-days> joins the placeholder allowlist Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: review pass 2 (delta): usage text, pass-marker capture group, GitHub-host origin parsing, idempotent trend.csv - usage() strips the header comment decoration again; --help text asserted - passCount takes the first all-digit capture group; --pass-marker without a capture group exits 2 - --repo falls back to the origin remote only on github.com or GH_HOST; a local-path origin exits 2 - trend.csv keeps one row set per repo and window (a rerun replaces its rows) and always carries its header - /retro Step 7.5 points at SCORECARD_TREND for the weeks; the doc cites the per-PR baseline CSV Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(host-conformance): delete a named section in the C3 broken-link case The case removed the first .md readdirSync returned under gstack-qa/sections. qa-patterns.md and system-functional.md are referenced only in table form, which setup --status does not count as section links, so on an ext4 directory order that listed one of them first the deleted file broke nothing and --status exited 0: deterministic red on remote-gate VMs, green on Actions. Delete scope.md, which every host's rendered SKILL.md reads. Closes #3479 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.252.0 stamp Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
12 KiB
Standalone binaries
Beyond the slash-command skills, gstack ships standalone CLIs for workflows that don't belong inside a session:
| Command | What it does |
|---|---|
gstack-model-benchmark |
Cross-model benchmark — run the same prompt through Claude, GPT (via Codex CLI), and Gemini; compare latency, tokens, cost, and (optionally) LLM-judge quality score. Auth detected per provider, unavailable providers skip cleanly. Output as table, JSON, or markdown. --dry-run validates flags + auth without spending API calls. |
gstack-taste-update |
Design taste learning — writes approvals and rejections from /design-shotgun into a persistent per-project taste profile. Decays 5%/week. Feeds back into future variant generation so the system learns what you actually pick. |
gstack-factory-scorecard |
Factory scorecard — one window (default 7 days) of a repository's shipping numbers, read from GitHub through gh: merged PRs (crew vs outside), lead time (created → merged, first READY comment → merged), gate runs with their known cost, red-train rate, flaky events (paid reruns and free-suite retries), issues opened vs closed, releases cut. Markdown table plus CSV, every line with its counting rule and what one of the best factories would show; /retro Step 7.5 runs it. Read-only, no paid work. Guide. |
gstack-egress |
Egress receipt auditor — every gstack-initiated off-machine send writes a tamper-evident, hash-chained receipt to ~/.gstack/security/egress.jsonl before the send. list shows what gstack attempted to send and to which host, grants shows the standing consent settings plus the exact command that revokes each, verify recomputes the hash chain and exits 3 on tamper (catches edits, reordering, and mid-chain deletion; truncating or deleting the ledger itself is out of scope — it's a forensic log, not tamper-proof storage). |
gstack-context-bill |
Token bill-of-materials — read-only, offline audit of what an installed skills tree costs in tokens: always-on frontmatter every session pays vs per-invocation SKILL.md + forced references. --diff compares two trees, --budget enforces a ceiling, --exact opts into Anthropic count_tokens (sends the text of the billed .md files inside the tree, the linked gstack checkout or gstack's render folder off-machine, names and estimates anything else a symlink reaches, labels a part-estimated bill mixed; writes an egress receipt first, degrades to the offline estimate if the receipt can't be written). |
gstack-code-intelligence |
Code-intelligence provider picker — wraps GBrain, Sourcebot, and Graphify behind one interface: options/status to see what's available, select to pick one, index/search to use it, suggest to check whether the one-time indexing offer should fire here. The offer triggers on large repos (1,000+ tracked files; a decline is persisted). Non-local providers refuse to index or search until you record per-repo consent (consent <repo> yes|no — the query text is repo-derived content), the per-repo trust policy's deny and read-only tiers veto write-class operations regardless of consent, and every off-machine send writes an egress receipt. Fully optional — with nothing selected, gstack falls back to grep. |
gstack-verify-gate |
Verification stop hook (opt-in) — blocks a Claude Code turn from ending until the project's declared verify command passes (after 3 blocked re-entries it yields with a loud still-RED warning instead of looping forever). Declare it on one line in CLAUDE.md: <!-- gstack:verify: bun test -->. Hooks bypass the permission system, so a declared command never runs until you trust it once per repo (gstack-verify-gate --trust); editing the command invalidates trust until re-granted, and every grant is audit-logged. ./setup never registers it for you — opt in with gstack-settings-hook add-event --event Stop --command ~/.claude/skills/gstack/bin/gstack-verify-gate --source verify-gate, remove with gstack-settings-hook remove-source --source verify-gate. |
gstack-memorable |
Memorable recall bridge (opt-in, third party, Claude Code only) — connects Claude Code to the external Memorable CLI through gstack instead of the vendor's own installer, so the hook gets gstack's guarantees: an explicit consent key (memorable_recall, off by default, listed by gstack-egress grants), a fail-closed egress receipt for every prompt handed over (gstack-egress list --sink memorable-recall), a HIGH-tier secret pre-scan, a trust envelope and 8 KiB cap on whatever comes back, an allowlisted environment and process-group containment for the vendor process, and clean removal. enable registers the hook at the stable install with a 5 s timeout and never runs the vendor's own consent command; disable revokes the gate first and removes the entry by identity even after Claude Code strips the tag; status is read-only. gstack never installs Memorable, and what its binary sends is the vendor's claim, not gstack's. Not available on Windows yet. Full guide. |
gstack-wtree |
Working-tree fingerprint — prints a content hash of what's actually on disk (temp index seeded from the stat cache, ~40x cheaper than a full re-hash; untracked source counts, gitignored scratch doesn't). Identical content fingerprints identically through commits, rebases, amends, and squashes — it's what binds reviews and test evidence to content instead of commit SHAs. |
gstack-review-log |
Review-pass receipts — --start <skill> captures the working-tree fingerprint before a diff review; '<JSON>' --finish <token> consumes that single-use, repository/branch/skill-scoped receipt. Binding requires matching start/end content and reviewer-reported completed:true and converged:true; it is not independent proof that a model read the source. --check-shared-libs <token> reads a current finding as JSON on stdin and checks prior Skip decisions against the actual branch, capture and eligible raw source blobs. It returns reusable:false when proof is missing or unsafe, including older records without logger-versioned coverage. Final review logging computes shared-code fingerprints and coverage rather than trusting supplied proof. |
gstack-review-read |
Review freshness — emits review records with computed review_freshness.status and reason: CURRENT, STALE, or UNVERIFIED for diff reviews. /ship and /land-and-deploy use the same grade; a matching commit alone never certifies a diff review. Dashboard rules. |
gstack-evidence |
Verification-evidence ledger — run --label <lane> -- <cmd> transparently wraps any test command (the child's exit code always passes through) and records what ran against which working-tree fingerprint; check grades each label FRESH/STALE/MISSING with --expect-cmd, --max-age, and --allow-paths binding. /ship and /land-and-deploy cite fresh evidence instead of re-running suites. Per-run logs are 0600, capped at 2MB, pruned after 30 days; the ledger and logs stay machine-local by design. |
gstack-issue-guard |
Tracker-text trust envelope — fetches GitHub issue/PR text (issue <n>, pr-body, pr-comments, or --stdin) and wraps it in a labeled envelope so agents treat it as data: injection-shaped lines get labeled even through fullwidth and invisible-character evasion, and forged envelope banners are defused. Every tracker-text ingress in gstack routes through it, enforced by a CI scanner. |
gstack-ios-qa-daemon |
iOS QA daemon — Mac-side broker between an agent and a connected iPhone or iPad over USB CoreDevice. Loopback by default; --tailnet opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on ~/.gstack/ios-qa-daemon.pid. See docs/howto-ios-testing-with-gstack.md. |
gstack-ios-qa-mint |
iOS allowlist manager — owner-grant CLI for the tailnet allowlist. grant/revoke/list against ~/.gstack/ios-qa-allowlist.json (mode 0600). Remote agents never auto-allowlist; this is the explicit-intent path. |
gstack-ios-qa-regen |
iOS bridge regenerator — deterministically installs the canonical DebugBridge package, generates typed state accessors, and records the installed gstack version. Safe to rerun after source changes or upgrades. |
The private paid CSO evaluation producer is a packaging contract, not an ordinary bun run build artifact. On macOS or Linux, a release operator compiles cso-eval-producer with the documented hardened Bun flags in the same clean build session as bun run build:cso, then installs it beside gstack-cso-launcher, gstack-cso-core, gstack-cso-watchdog, and the hidden .gstack-cso-generation manifest as one root-owned, nonwritable five-artifact unit. The producer rejects root execution, writable/symlinked/incomplete installations, and unreviewed provider CLI versions; each receipt binds all five artifact hashes, and collection rejects receipts from different unit identities. See the clean producer procedure. Paid producer evaluation remains unavailable on Windows because the detached watchdog has no Windows build.
Each producer gets a curated one-cell source copy with directories sealed to 0555 and files to 0444, plus pre/post content, Git, and mode checks. Claude receives no source add-directory, because an add-directory is a sandbox write root where Claude Code's bubblewrap leaves an empty .mcp.json mount point. Its Bash runs inside Claude Code's sandbox with one exception: the exact absolute launcher path that --allowed-tools names, which runs outside it as it does for users. The launcher is the boundary there. It replaces its environment with a fixed allowlist, uses only the local Docker Unix socket, and runs every target in hardened containers. The producer passes that policy through --settings (sandbox enabled, fail if unavailable, no unsandboxed fallback, no auto-allow, and excludedCommands naming only the launcher). It refuses to start the agent (CLAUDE_PRODUCER_SANDBOX_POLICY_MISMATCH) unless the generated arguments and environment carry exactly that policy. CLAUDE_CODE_SUBPROCESS_ENV_SCRUB stays unset, because it would force the launcher back into the sandbox. Every other Bash command is denied by --allowed-tools, and Write is confined to the provider work directory and cso-home. Gemini receives no source working directory or include-directory. Codex technically receives read-only filesystem access to the exact curated source root because the trusted helper inherits the Codex permission profile; its private provider work directory and exact cso-home artifact directory are the only write roots, every other root path remains denied, and the producer prompt requires source access through the helper. This is evaluation containment for an immutable public fixture, not a claim that Codex cannot directly read that fixture.
Copy each completed cell's receipt together with state/cso-home/security/cso/ to the trusted adjudication host. Receipt entries are sorted paths relative to that directory and bind every retained file's size and SHA-256 plus an aggregate inventory hash. Provider homes, settings, sessions, and credentials live under separate disposable directories and are removed after the cell; they are never part of the retained artifact tree. The adjudicator must re-hash the transported tree against the receipt before trusting reports or repair bundles.
Paid producer qualification uses this explicit host matrix:
| Producer host | Daily/static cells | Comprehensive target execution |
|---|---|---|
| Codex CLI 0.153.4 | Supported under the custom permission profile | Fails closed because Docker/socket access is not granted to the model command sandbox; report the setup gap as partial |
| Claude Code 2.1.263 | Supported under restricted safe mode | Fails closed when the restricted launcher child cannot reach Docker; report the setup gap as partial |
| Gemini CLI 0.59.0 | Supported with isolated home/settings | Private release qualification host: the exact run_shell_command(<launcher>) policy can run the trusted helper without exposing a general shell |
Every setup-blocked comprehensive cell remains a miss in release-gate denominators. The evaluator does not silently count a host or workflow as supported when its containment policy prevents required setup.