* Attribute every paid trial's cost: bind label-named records by testName, price aborted SDK sessions from streamed usage, model-free cases cost $0
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: case cost adds streamed sessions to record sums and keeps record cost_known
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: the paid report headline counts ledger-only session cost once
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: a cost-only record never sets the trial model
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: recordE2E writes cost_case_id only for a registered label-named case
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: CHANGELOG for paid-report cost attribution (Unreleased)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: the headline adds only streamed session cost; cache writes without a 5m key
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: how the paid report prices a trial and what stays cost-unknown
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Price PTY sessions from their transcripts: pin a session id, sum the session's and its subagents' usage into the ledger's streamed_cost_usd
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: transcript pricing never throws from a PTY ledger writer
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore: pin transcript-cost.ts as non-behavioral at HARNESS_VERSION 6
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: recognise --session-id= and --resume= forms
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: CHANGELOG for PTY transcript pricing (Unreleased)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: PTY sessions priced from their transcripts
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: price PTY transcripts without adding a launch argument; native AUQ capture opts out
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: harness notes and CHANGELOG match the transcript claim
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): gstack-flaky run reruns a failed JUnit test once and ledgers the flake (GSTA-158)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): stream output, keep unreported runner failures red, fail-closed duplicates and nonzero reruns
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: pre-landing review fixes
gstack-flaky never greens an incomplete run: a targeted rerun needs the
runner's summary proving the first run complete, otherwise (and always for
generic runners) the rerun repeats the whole command. A rerun clears a test
only when it exits 0 and the test passes once on its first attempt. Killed
runners exit 3 (FLAKY_COMMAND_KILLED) and signals are forwarded. The first
report is restored at --junit with reruns kept beside it; ids are redacted at
every output; the ledger gets its own GSTACK_FLAKE_LEDGER_JUNIT; fixtures are
registered in FREE_FIXTURES.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: latch cancellation and check whole-command rerun reports
A cancel forwarded to a runner that traps the signal now counts as killed for
every later attempt, so a cancelled job is never rerun or greened. A
whole-command rerun clears nothing when its report has a failed test, misses a
first-run test, or its runner counted failures outside the report.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: whole-command rerun when coverage or runner-flaky tests can fail the run
A coverage threshold or a fail-on-flaky policy fails a run for a reason a
narrowed rerun cannot reproduce, so either forces a whole-command rerun.
--junit refuses an existing file that is not a JUnit report instead of
deleting it.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): rerun the whole command by default; targeted rerun is opt-in
A rerun counts only on the runner's own full green verdict: exit 0, no failed
or missing test, nothing counted outside the report. That covers every way a
run can fail, which a narrowed rerun cannot (a coverage threshold with an lcov
reporter, a fail-on-flaky policy, a test never reached). --rerun-scope test
reruns the failed test alone, with its preconditions documented and the
complete-run checks kept as a safety net. An unreadable existing --junit file
is a usage error, not a crash.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: gstack-flaky CHANGELOG entry and standalone-binaries row (GSTA-158)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(flaky): quarantine registry for repeat flakes (GSTA-158 P2)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): bound renewal dates; match non-ASCII paths in check --base
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(flaky): reword the FLAKY_QUARANTINE_BASE hint so the egress scanner does not read it as a network op
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(agent-sdk): every SDK caller passes the claude_code preset or names why not
The SDK's default system prompt is a bare stub that drops the line telling the
model to run /skill through the Skill tool. runAgentSdkTest now refuses a
missing, empty or bare-string prompt; non-preset callers name a reason; a free
tripwire scans test/ and scripts/ for the same mistake.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: record the SDK prompt guard as a non-behavioral harness change and baseline its tripwire
harness-version: runAgentSdkTest validates systemPrompt before starting; every
live caller already passed the claude_code preset, so no verdict can change.
test-of-test ratchet: the tripwire's subject is the eval harness by design.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: name the preset literal callers pass and the unnamed-prompt refusal
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* feat(eval-plan): frontier models under test (R54)
Haiku and Sonnet are not models under test. dry-run refuses a plan whose every
arm is Haiku or Sonnet unless --tier-question names what it asks about that
tier, warns on a mix, and re-costs the plan on Opus 5.5 and GPT-6.1 Sol.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(eval-plan): trim R54 skill prose and raise the eval-plan context ceiling
The R54 paragraph and CHECK: frontier bullet were cut to the minimum; the
exemptions and verdict rule stay in docs/TESTING_INTERNALS.md. The remaining
growth is the feature, so only the eval-plan eager ceiling moves (4887 -> 5458,
the capture helper's value); other skills' ceilings are left as they were.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(eval-plan): pilot start keeps the R54 refusal; --tier-question rejects a flag
Review pass 1 (full depth) found the frontier refusal could be skipped by
starting a pilot on Haiku or Sonnet arms behind a dry run made on other arms,
or behind a dry-run.json written before the check existed. startPilot now
refuses a pilot whose every arm is Haiku or Sonnet unless its dry run recorded
a tier question. dry-run --tier-question no longer takes the next flag as its
answer. The testing doc now says the frontier re-cost is printed for
comparison with the brief cap, not enforced.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(support-tiers): Windows CI in the present tense now that windows-smoke is required (#3806)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore: empty commit for a clean check set at the same tree (cancelled orphan free-suite check at 97487df)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(skills): /eval-plan entry names the R54 frontier check
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
18 KiB
Standalone binaries
Beyond the slash-command skills, gstack ships standalone CLIs for workflows that don't belong inside a session:
| Command | What it does |
|---|---|
gstack-model-benchmark |
Cross-model benchmark — run the same prompt through Claude, GPT (via Codex CLI), and Gemini; compare latency, tokens, cost, and (optionally) LLM-judge quality score. Auth detected per provider, unavailable providers skip cleanly. Output as table, JSON, or markdown. --dry-run validates flags + auth without spending API calls. |
gstack-taste-update |
Design taste learning — writes approvals and rejections from /design-shotgun into a persistent per-project taste profile. Decays 5%/week. Feeds back into future variant generation so the system learns what you actually pick. |
gstack-factory-scorecard |
Factory scorecard — one window (default 7 days) of a repository's shipping numbers, read from GitHub through gh: merged PRs (crew vs outside), lead time (created → merged, first READY comment → merged), gate runs with their known cost, red-train rate, flaky events (paid reruns and free-suite retries), issues opened vs closed, releases cut. Markdown table plus CSV, every line with its counting rule and what one of the best factories would show; /retro Step 7.5 runs it. Read-only, no paid work. Guide. |
gstack-egress |
Egress receipt auditor — every gstack-initiated off-machine send writes a tamper-evident, hash-chained receipt to ~/.gstack/security/egress.jsonl before the send. list shows what gstack attempted to send and to which host, grants shows the standing consent settings plus the exact command that revokes each, verify recomputes the hash chain and exits 3 on tamper (catches edits, reordering, and mid-chain deletion; truncating or deleting the ledger itself is out of scope — it's a forensic log, not tamper-proof storage). |
gstack-context-bill |
Token bill-of-materials — read-only, offline audit of what an installed skills tree costs in tokens: always-on frontmatter every session pays vs per-invocation SKILL.md + forced references. --diff compares two trees, --budget enforces a ceiling, --exact opts into Anthropic count_tokens (sends the text of the billed .md files inside the tree, the linked gstack checkout or gstack's render folder off-machine, names and estimates anything else a symlink reaches, labels a part-estimated bill mixed; writes an egress receipt first, degrades to the offline estimate if the receipt can't be written). |
gstack-code-intelligence |
Code-intelligence provider picker — wraps GBrain, Sourcebot, and Graphify behind one interface: options/status to see what's available, select to pick one, index/search to use it, suggest to check whether the one-time indexing offer should fire here. The offer triggers on large repos (1,000+ tracked files; a decline is persisted). Non-local providers refuse to index or search until you record per-repo consent (consent <repo> yes|no — the query text is repo-derived content), the per-repo trust policy's deny and read-only tiers veto write-class operations regardless of consent, and every off-machine send writes an egress receipt. Fully optional — with nothing selected, gstack falls back to grep. |
gstack-verify-gate |
Verification stop hook (opt-in) — blocks a Claude Code turn from ending until the project's declared verify command passes (after 3 blocked re-entries it yields with a loud still-RED warning instead of looping forever). Declare it on one line in CLAUDE.md: <!-- gstack:verify: bun test -->. Hooks bypass the permission system, so a declared command never runs until you trust it once per repo (gstack-verify-gate --trust); editing the command invalidates trust until re-granted, and every grant is audit-logged. ./setup never registers it for you — opt in with gstack-settings-hook add-event --event Stop --command ~/.claude/skills/gstack/bin/gstack-verify-gate --source verify-gate, remove with gstack-settings-hook remove-source --source verify-gate. |
gstack-memorable |
Memorable recall bridge (opt-in, third party, Claude Code only) — connects Claude Code to the external Memorable CLI through gstack instead of the vendor's own installer, so the hook gets gstack's guarantees: an explicit consent key (memorable_recall, off by default, listed by gstack-egress grants), a fail-closed egress receipt for every prompt handed over (gstack-egress list --sink memorable-recall), a HIGH-tier secret pre-scan, a trust envelope and 8 KiB cap on whatever comes back, an allowlisted environment and process-group containment for the vendor process, and clean removal. enable registers the hook at the stable install with a 5 s timeout and never runs the vendor's own consent command; disable revokes the gate first and removes the entry by identity even after Claude Code strips the tag; status is read-only. gstack never installs Memorable, and what its binary sends is the vendor's claim, not gstack's. Not available on Windows yet. Full guide. |
gstack-decision-trail |
Decision trail — append one row (ts, phase, decision, why, evidence, result) to a run's TSV trail so a reviewer reads what was chosen and why without the transcript. Header once, append-only, cells flattened to one line, and a cell starting with =, +, - or @ quoted so a spreadsheet cannot run it. start <file> <run-id> opens a run's stretch of a shared trail (idempotent; start is reserved as a phase name), and --run <id> refuses an append until that start row is the newest. Ported from pstack's log.sh (MIT, NOTICE.md). Rule. |
gstack-worktree-audit |
Worktree audit — read-only table of every git worktree beside the main one, largest first: size, age, merged into the base or not, clean/clean,ignored:<n>/wip:<n>/scratch:<n>, pushed/ahead<n>/no-remote/detached, the PR on its branch through gh (fork PRs ignored), the newest Claude Code chat in it, and a bucket (prunable for an unlocked registered worktree whose directory is gone, review for a locked or broken one, hold-wip, hold-open-pr, verify-recent-chat, safe only for a clean tree that is merged into the base or whose PR is closed or merged with every commit pushed, else review). -b origin/<branch> sets the base, --no-fetch skips the best-effort fetch of the base ref (its one write). Never deletes. Ported from pstack's worktree-audit.sh (MIT, NOTICE.md). Guide. |
gstack-clean |
Per-repo .gstack/ cleanup — run inside a repository to see what it would reclaim from that repository's .gstack/ folder; --apply does it. It removes tmp/codex-* scratch files older than 7 days, cuts the browse logs (browse-console.log, browse-network.log, browse-dialog.log, browse-daemon.log, browse-startup-error.log) to their last 256 KB once they pass 1 MB, and trims browse-audit.jsonl only with --audit-days N. It never touches browse.json, session-state.json, chromium-profile/, browser-skills/ or browse-states/, skips browse files while the daemon runs and tmp/ while codex runs, and refuses (exit 3) outside git, when .gstack is a symlink, when BROWSE_STATE_FILE names another folder, and when the folder is, holds or sits inside the gstack state root. --json prints one versioned object. Exit 0 done, 1 an action failed (rerunning is safe), 2 usage, 3 refused. Error codes. |
gstack-wtree |
Working-tree fingerprint — prints a content hash of what's actually on disk (temp index seeded from the stat cache, ~40x cheaper than a full re-hash; untracked source counts, gitignored scratch doesn't). Identical content fingerprints identically through commits, rebases, amends, and squashes — it's what binds reviews and test evidence to content instead of commit SHAs. |
gstack-review-log |
Review-pass receipts — --start <skill> captures the working-tree fingerprint before a diff review; '<JSON>' --finish <token> consumes that single-use, repository/branch/skill-scoped receipt. Binding requires matching start/end content and reviewer-reported completed:true and converged:true; it is not independent proof that a model read the source. --check-shared-libs <token> reads a current finding as JSON on stdin and checks prior Skip decisions against the actual branch, capture and eligible raw source blobs. It returns reusable:false when proof is missing or unsafe, including older records without logger-versioned coverage. Final review logging computes shared-code fingerprints and coverage rather than trusting supplied proof. |
gstack-review-read |
Review freshness — emits review records with computed review_freshness.status and reason: CURRENT, STALE, or UNVERIFIED for diff reviews. /ship and /land-and-deploy use the same grade; a matching commit alone never certifies a diff review. Dashboard rules. |
gstack-evidence |
Verification-evidence ledger — run --label <lane> -- <cmd> transparently wraps any test command (the child's exit code always passes through) and records what ran against which working-tree fingerprint (tracked and untracked files that are not gitignored, minus an untracked .bun/ folder at the repo root, which Bun can create during a run); a run whose files changed while it ran is recorded without a fingerprint and with up to five of the changed paths, which check names in its STALE reason. check grades each label FRESH/STALE/MISSING with --expect-cmd, --max-age, and --allow-paths binding. /ship and /land-and-deploy cite fresh evidence instead of re-running suites. Per-run logs are 0600, capped at 2MB, pruned after 30 days; the ledger and logs stay machine-local by design. |
gstack-flaky |
Flaky-test rerun for any JUnit runner — run --junit <file.xml> -- <command> runs your test command (Playwright, bun test, jest, vitest, pytest) and, when at most --max-reruns (3) tests failed and the runner made no retries of its own, reruns the whole command once (--rerun-scope test opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (GSTACK_FLAKE_LEDGER_JUNIT, default <state root>/flake-ledger-junit.jsonl). A missing or malformed report, a crash or a killed runner never passes. `quarantine add |
gstack-pr-wait |
PR event watcher — gstack-pr-wait <pr> [--repo owner/name] [--until green|ready|any-change] [--interval 20] [--timeout 60] [--exclude CHECK,...] polls one pull request through gh and prints one stdout line per real event (a check failed with its link, all checks finished, a comment or review from someone other than the author, a review bot finished on the head, the base moved and its new commits touch this PR's files, BEHIND or DIRTY, merged or closed), with progress on stderr, so a monitor wakes the agent once per event instead of on a timer. green gates on every check on the head (an advisory bot's review is never waited for, its checks gate like any other), as gstack-ci-gate does; a check the base branch requires (rulesets and classic branch protection) is pending until it appears, never green by omission; two identical polls confirm a check set is complete; --exclude leaves a named non-required check out of the verdict as gstack-ci-gate --exclude does (a required name is refused and gates). Exit 0 when the condition is met, 1 when it needs action or --timeout passes (PR_WAIT_BLOCKED), 2 on bad input, a second watcher for the same PR or a local error (PR_WAIT_REFUSED). State under <state root>/pr-wait/, baseline discarded after 6 hours. Review bots are configuration: gstack-config set pr_wait_gating_bots greptile,macroscope, pr_wait_advisory_bots coderabbit (advisory bots are reported, never waited on). /land-and-deploy Step 3 is one call of it. Design ported from tstack's pr-wait (MIT, Theo Browne; NOTICE.md). Exit codes. |
gstack-issue-guard |
Tracker-text trust envelope — fetches GitHub issue/PR text (issue <n>, pr-body, pr-comments, or --stdin) and wraps it in a labeled envelope so agents treat it as data: injection-shaped lines get labeled even through fullwidth and invisible-character evasion, and forged envelope banners are defused. Every tracker-text ingress in gstack routes through it, enforced by a CI scanner. |
gstack-ios-qa-daemon |
iOS QA daemon — Mac-side broker between an agent and a connected iPhone or iPad over USB CoreDevice. Loopback by default; --tailnet opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on ~/.gstack/ios-qa-daemon.pid. See docs/howto-ios-testing-with-gstack.md. |
gstack-ios-qa-mint |
iOS allowlist manager — owner-grant CLI for the tailnet allowlist. grant/revoke/list against ~/.gstack/ios-qa-allowlist.json (mode 0600). Remote agents never auto-allowlist; this is the explicit-intent path. |
gstack-ios-qa-regen |
iOS bridge regenerator — deterministically installs the canonical DebugBridge package, generates typed state accessors, and records the installed gstack version. Safe to rerun after source changes or upgrades. |
The private paid CSO evaluation producer is a packaging contract, not an ordinary bun run build artifact. On macOS or Linux, a release operator compiles cso-eval-producer with the documented hardened Bun flags in the same clean build session as bun run build:cso, then installs it beside gstack-cso-launcher, gstack-cso-core, gstack-cso-watchdog, and the hidden .gstack-cso-generation manifest as one root-owned, nonwritable five-artifact unit. The producer rejects root execution, writable/symlinked/incomplete installations, and unreviewed provider CLI versions; each receipt binds all five artifact hashes, and collection rejects receipts from different unit identities. See the clean producer procedure. Paid producer evaluation remains unavailable on Windows because the detached watchdog has no Windows build.
Each producer gets a curated one-cell source copy with directories sealed to 0555 and files to 0444, plus pre/post content, Git, and mode checks. Claude receives no source add-directory, because an add-directory is a sandbox write root where Claude Code's bubblewrap leaves an empty .mcp.json mount point. Its Bash runs inside Claude Code's sandbox with one exception: the exact absolute launcher path that --allowed-tools names, which runs outside it as it does for users. The launcher is the boundary there. It replaces its environment with a fixed allowlist, uses only the local Docker Unix socket, and runs every target in hardened containers. The producer passes that policy through --settings (sandbox enabled, fail if unavailable, no unsandboxed fallback, no auto-allow, and excludedCommands naming only the launcher). It refuses to start the agent (CLAUDE_PRODUCER_SANDBOX_POLICY_MISMATCH) unless the generated arguments and environment carry exactly that policy. CLAUDE_CODE_SUBPROCESS_ENV_SCRUB stays unset, because it would force the launcher back into the sandbox. Every other Bash command is denied by --allowed-tools, and Write is confined to the provider work directory and cso-home. Gemini receives no source working directory or include-directory. Codex technically receives read-only filesystem access to the exact curated source root because the trusted helper inherits the Codex permission profile; its private provider work directory and exact cso-home artifact directory are the only write roots, every other root path remains denied, and the producer prompt requires source access through the helper. This is evaluation containment for an immutable public fixture, not a claim that Codex cannot directly read that fixture.
Copy each completed cell's receipt together with state/cso-home/security/cso/ to the trusted adjudication host. Receipt entries are sorted paths relative to that directory and bind every retained file's size and SHA-256 plus an aggregate inventory hash. Provider homes, settings, sessions, and credentials live under separate disposable directories and are removed after the cell; they are never part of the retained artifact tree. The adjudicator must re-hash the transported tree against the receipt before trusting reports or repair bundles.
Paid producer qualification uses this explicit host matrix:
| Producer host | Daily/static cells | Comprehensive target execution |
|---|---|---|
| Codex CLI 0.153.4 | Supported under the custom permission profile | Fails closed because Docker/socket access is not granted to the model command sandbox; report the setup gap as partial |
| Claude Code 2.1.263 | Supported under restricted safe mode | Fails closed when the restricted launcher child cannot reach Docker; report the setup gap as partial |
| Gemini CLI 0.59.0 | Supported with isolated home/settings | Private release qualification host: the exact run_shell_command(<launcher>) policy can run the trusted helper without exposing a general shell |
Every setup-blocked comprehensive cell remains a miss in release-gate denominators. The evaluator does not silently count a host or workflow as supported when its containment policy prevents required setup.