mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
v1.91.16.0 fix: safe installs and upgrades, skills start everywhere, truthful completion (parallel wave) (#3014)
* fix(make-pdf,design): real TOC page numbers and anchors; design models move to gpt-5.5 + gpt-image-2 (#2903, #2807) * ci: pin every action to a SHA, Scorecard, checksummed Bun, scoped Codex CLI in the CI image, macOS named-regressions job (#1948, #1706, #1997) * fix(cso,browse): Windows/macOS lease identity, retention skips bad runs, truthful 'not assessed' reports, env.kv false positive, Windows browse spawn fixes (#2894, #2912, #2048, #2637, #2805) * fix(outside-voice): one Codex model precedence for probe and dispatch, bounded timeouts with KILL escalation, probe cache identity, guarded mktemp, skills hidden in nested Codex (#2776, #2914, #2787, #2881, #2847) * feat(setup): install registry, ./setup --status, staged activation, per-scope migrations, host tiers and conformance kit, OpenCode/Hermes fixes, Copilot CLI (experimental), disabled_skills, slimmer CLAUDE.md (#1925, #2879, #2347, #2338, #2626, #2629, #2825, #2015, #393, #1206, #2096) Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com> Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com> Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com> * fix(preamble): skills start in worktree-isolated Claude Code sessions — one literal skill-start line, context-recovery helper, no eval in skill blocks, $N token tripwire, honest plan-mode precedence (#2763, #2896, #2851) * fix(setup): per-install renders for renamed, vendored and custom-config installs, space-safe alias roots, per-host /cso launcher path (#1882, #2906) * test: free tests never write the real HOME (home-write tripwire), no orphaned browse daemons, symlinked-TMPDIR (macOS) fixes, gbrain-on-PATH isolation (#2895, #2816, #2829, #2841, #2982, #2911, #2909, #2908, #2910) * chore: integrate wave follow-ups — regenerate all hosts, README model docs, TODOS, touchfiles, duration seeds, Bun 1.3 telemetry test fix, replay fixtures pinned to their recorded bytes * v1.91.14.0: release metadata * test(browse): group-kill cleanup falls back to the leader when macOS returns EPERM for a live group * ci: Codex login steps run under bash (the CI container's default sh rejects pipefail) * fix: full-suite repairs — private state roots for 7 real-HOME writers, truthful home-guard attribution with --attribute-home, gstack-paths shim in ship-hook fixtures, Copilot in team-init probes, test-of-test detector, cookie review fixture pinned to the approved excerpt * fix(windows): LF for Dockerfiles, normalized home-guard exclusions, path.relative fixture containment, ship push test spawn timeouts * test(design-detect): widen the whole-scan budget test's timing margins so a loaded macOS runner can't time out the single-batch call * fix(evals): qa protocol names defined for the judge, async PTY judge so overlapping sessions keep their cleanup deadline, bws setup cases use the standard JUDGE_MS tier * test(pty): the non-blocking judge case is POSIX-only (its fake judge is a shebang script, like the PTY harness it covers) * test(evals): ship-coverage-audit names absolute skill paths, uses Read, and skips the Step 7 artifact nothing consumes (removes the mechanism behind 3 of 4 historical timeouts) * test: deflake terminal-agent concurrent startup (write the placeholder record before spawning, like the product) and qa-evidence resumed pipe (barrier on a full high-water mark instead of a sleep) * fix(design): daemon log stream handles open errors instead of crashing the daemon; shutdown test awaits the stubbed exit * test(evals): skillify checks the product's resolved state root (no GSTACK_HOME bypass), negative cases check every root; tpa shims live outside the agent's cwd and answer consistently * test(evals): coverage-audit detector splits parallel columns at │ rails and [+] groups; coverage prompts name absolute skill paths; TODOs from the gate eval census * fix(plan-eng-review): scope gate always sends the menu (no headless 'Scope pending' exit); PTY harness ends idle turn-ends and API-error panels promptly with a precise diagnostic * test(evals): one position-bound coverage-read rule credits read-only && chains (labels, context ranges, git displays in either order) and still rejects forged output * fix(ship): documentation section has one saved source inserted by cat (never retyped); spawned-marker evidence is the child output; refreshed codex/factory ship goldens * test(docsync): the byte-exact insert positive is POSIX-only (the bash-command observer refuses backslash paths by design) * test(evals): plan-design-with-ui focus-title rule accepts the comma-less 'dimensions or focus' wording (replayed on the captured CI run, negative controls kept) * fix(qa-callers): exploratory preflight says the scope/method Reads come after exploratory.md; design-review scan check credits an executed scan proven by one engine run over the changed files --------- Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com> Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com> Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com>
This commit is contained in:
392 files changed
+10217
-3784
No files matched your search
+194
-4
@@ -3,8 +3,196 @@
|
||||
gstack uses a declarative host config system. Each supported AI coding agent
|
||||
(Claude, Codex, Factory, Kiro, OpenCode, Slate, Cursor, OpenClaw, Hermes,
|
||||
GBrain) is defined as a typed TypeScript config object built by the
|
||||
`defineHost()` factory. Adding a new host means creating one file and
|
||||
re-exporting it. Zero code changes to the generator, setup, or tooling.
|
||||
`defineHost()` factory. Rendering a new host means creating one file and
|
||||
re-exporting it: the generator, tests, and validation pick it up. Installing
|
||||
it is not free: setup still has one install arm per installable host (see
|
||||
"What setup changes on disk" below), so an installable host also needs a setup
|
||||
arm, an entry in the install registry tier table, and a certification run
|
||||
before it is labelled `full`.
|
||||
|
||||
## Install ownership rules
|
||||
|
||||
These rules hold for every host, every scope, and every upgrade path
|
||||
(`./setup`, `./setup --refresh-registered`, `/gstack-upgrade`, the team-mode
|
||||
auto-update). Each refusal names what was left alone, the rule that applied,
|
||||
and the exact command to proceed.
|
||||
|
||||
1. **Project-local setup never silently replaces a global install.** Setup
|
||||
run from a project-vendored copy, or from any checkout other than the one a
|
||||
global install links to, leaves that global install alone and prints
|
||||
`./setup --global` as the way to replace it on purpose. (Codex:
|
||||
`setup:186-207`, #2879. Claude: the global `~/.claude/skills/gstack` link.)
|
||||
2. **An explicit `--host X` never mutates another host.** `--host codex`
|
||||
writes only Codex's skills directory and the checkout's own `.agents/`
|
||||
render. It does not heal, register, or remove Claude Code hooks in
|
||||
`settings.json`, and it does not touch the Claude gbrain render (#2347).
|
||||
One deliberate exception: the one-time `/claude` → `/claude-code` rename
|
||||
repair (`bin/gstack-migrate-claude-code`) renames gstack-owned entries in
|
||||
every install that shares the checkout.
|
||||
3. **Upgrades refresh exactly the registered installs.** Every install setup
|
||||
activates is recorded in the install registry. Upgrades run
|
||||
`./setup --refresh-registered`, which runs setup once per registered host
|
||||
of that checkout (printing the source first) and nothing else. Installs
|
||||
owned by another checkout, and project installs of another project, are
|
||||
listed with the command that refreshes them there; they are never repointed
|
||||
(#1925).
|
||||
4. **User-owned files survive.** Setup, `gstack-relink`, and
|
||||
`gstack-uninstall` delete or link over an entry only with proof that gstack
|
||||
created it (the #2119 ownership gate); anything else is reported and left.
|
||||
|
||||
### The install registry
|
||||
|
||||
`$GSTACK_STATE_ROOT/installs.tsv` (mode 0600) is the one record of installs.
|
||||
Only `bin/gstack-install-registry.sh` writes it, under a lock directory with an
|
||||
atomic rename. Columns: host, scope (`global`/`project`), project, destination
|
||||
(the skills directory the host discovers), install root, source realpath,
|
||||
version, prefix setting, render, updated-at. A row publishes only after the
|
||||
host arm finished activating. Rows whose install root is gone are dropped
|
||||
(`gstack_install_registry_reconcile`) by every refresh and by
|
||||
`gstack-uninstall`, so an upgrade never resurrects an uninstalled host.
|
||||
|
||||
`./setup --status` prints one row per install (host, tier, scope, result,
|
||||
version, destination, source) without building or writing anything, plus any
|
||||
pre-registry install it finds on disk, labelled `unregistered` with the
|
||||
command that registers it. A Codex install captured by a project-vendored copy
|
||||
(#2879) is reported with both choices instead of being repointed. The setup
|
||||
summary, the `/gstack-upgrade` summary, and `--status` share one renderer
|
||||
(`gstack_install_registry_render`).
|
||||
|
||||
### Migrations
|
||||
|
||||
`gstack-upgrade/migrations/v*.sh` are state-root migrations. Their marker is
|
||||
`$GSTACK_STATE_ROOT/.last-setup-version` (read from `~/.gstack` once for
|
||||
installs that predate the move). It advances only past migrations that
|
||||
succeeded; a failed migration stops the sequence, prints a `failed` row with
|
||||
the retry command, and runs again on the next `./setup`. Install and host
|
||||
migrations are tracked per install by the registry row's version.
|
||||
|
||||
### What setup changes on disk
|
||||
|
||||
Inventory of every setup side effect, by phase (line numbers are approximate):
|
||||
|
||||
| Phase | Writes | Scope |
|
||||
|---|---|---|
|
||||
| Build | `browse/dist`, `design/dist`, `make-pdf/dist`, `bin/gstack-cso-*` in the source checkout | source |
|
||||
| Generation | `.agents/`, `.kiro/`, `.factory/`, `.opencode/`, `.cursor/` renders in the source checkout; `gstack-patch-names` rewrites Claude `name:` fields in place | source |
|
||||
| Chromium | Playwright cache (`~/.cache/ms-playwright`), lock under the state root | machine |
|
||||
| Config | `skill_prefix`, `timeline_stop_hook`, team-mode keys in `$GSTACK_STATE_ROOT/config.yaml` | state root |
|
||||
| Claude arm | `~/.claude/skills/gstack` link, one dir per skill with a `SKILL.md` link, alias copies | selected host |
|
||||
| Codex arm | `${CODEX_HOME:-~/.codex}/skills/gstack` runtime root, `gstack-*` links into `.agents/skills`, `.agents/skills/gstack` sidecar | selected host |
|
||||
| Kiro, Factory, OpenCode, Cursor, Copilot arms | the host's `skills/gstack` runtime root (built beside the live one and swapped in) and `gstack-*` links | selected host |
|
||||
| OpenCode commands | `~/.config/opencode/commands/gstack-*.md` with a managed marker (#2629); user command files are never touched | selected host |
|
||||
| Registry | one row per activated install in `installs.tsv` | state root |
|
||||
| Migrations | whatever each `v*.sh` repairs; marker in the state root | state root |
|
||||
| Claude hooks | `~/.claude/settings.json` (heal, SessionStart, plan-tune, Stop) — only when Claude is selected | Claude |
|
||||
| gbrain | `gbrain-detection.json`; the Claude render in `$GSTACK_STATE_ROOT/render/claude` — only when Claude is selected | state root / Claude |
|
||||
|
||||
Activation today: on macOS and Linux every install is a set of symlinks into
|
||||
the source checkout, so a `git pull` changes the active payload before setup
|
||||
runs and "keep the prior payload" means "keep the prior checkout" — which the
|
||||
upgrade skill does on the vendored path (`.bak` restore) and records on the
|
||||
git path (`PRE_UPGRADE_COMMIT`). On Windows, where setup copies instead of
|
||||
linking, every copy is staged as `<dst>.gstack-new.<pid>` and swapped in only
|
||||
when complete; an interrupted swap is repaired on the next run (a missing
|
||||
destination is restored from `<dst>.gstack-old.<pid>`). Versioned payload
|
||||
directories for the symlink platforms are not built yet.
|
||||
|
||||
### Install-context render contract
|
||||
|
||||
Committed renders name each host's default install root. A render for any
|
||||
other root (project install, `CLAUDE_CONFIG_DIR`, renamed checkout) is made
|
||||
with `gen-skill-docs --host <h> --install-root <abs path> --out-dir <tmp>
|
||||
--link-root <dir>`: every default-root path becomes the given root, and
|
||||
resolvers see it as `TemplateContext.installRoot` (null keeps committed bytes).
|
||||
Per-install renders live in `$GSTACK_STATE_ROOT/render/installs/<host>-<id>/`
|
||||
(`<id>` keyed by the root's realpath) and are recorded in the registry's render
|
||||
column. Setup makes one for a Claude install outside `~/.claude/skills/gstack`
|
||||
(renamed checkout, project-vendored copy, `CLAUDE_CONFIG_DIR`) and a global Codex
|
||||
install outside `~/.codex/skills/gstack` (`CODEX_HOME`); repo-local Codex installs
|
||||
resolve their root at run time. A root that is not a plain path (whitespace,
|
||||
shell metacharacters) is named through a `<host>-<id>.root` alias symlink:
|
||||
worktree-isolated Claude Code refuses a command path containing a space in any
|
||||
quoting.
|
||||
|
||||
## Host tiers and capabilities
|
||||
|
||||
Every host declares `tier` and `capabilities` (`scripts/host-config.ts`).
|
||||
One vocabulary is used everywhere: `./setup --help`, setup and `--status`
|
||||
rows, host errors, and the README host matrix, all checked against
|
||||
`hosts/index.ts` by `test/host-config.test.ts`.
|
||||
|
||||
| Tier | Meaning |
|
||||
|---|---|
|
||||
| `full` | Installs, renders, passes the conformance kit, and has a dated certification record below. |
|
||||
| `experimental` | Installs and renders and passes the conformance kit; no certification record yet. |
|
||||
| `instruction-only` | No install arm. `./setup --host X` prints what to copy and changes nothing. |
|
||||
|
||||
Rendering alone never certifies `full`.
|
||||
|
||||
| Capability | Values | Used for |
|
||||
|---|---|---|
|
||||
| `toolExecution` | boolean | preambles and `bin/` helpers run |
|
||||
| `questions` | `native` / `prose` | structured question tool vs plain-text questions |
|
||||
| `planMode` | boolean | plan-mode transitions |
|
||||
| `delegation` | boolean | sub-agents |
|
||||
| `browser` | boolean (needs `toolExecution`) | gstack's browser |
|
||||
| `safetyHooks` | `enforced` / `advisory` | `/careful`, `/freeze`, `/guard` block (Claude) or only warn |
|
||||
|
||||
Hosts with `safetyHooks: 'advisory'` get one line at the top of `/careful`,
|
||||
`/freeze` and `/guard`: "not enforced on <host>: advisory, not blocked".
|
||||
|
||||
## Certify your host
|
||||
|
||||
1. Pass the conformance kit: `bun test test/host-conformance.test.ts` installs
|
||||
every installable host into a temp HOME, checks discovery directories, no
|
||||
other host's variant, no duplicates, `./setup --status`, and an upgrade from
|
||||
the previous release's install layout.
|
||||
2. Run one representative workflow for real on the host (for example
|
||||
`/review` on a small diff), and an upgrade from an existing install.
|
||||
3. Add a row below with the date, host version, gstack version, and where the
|
||||
transcript lives, then set `tier: 'full'`.
|
||||
|
||||
| Host | Date | Host version | gstack version | Evidence |
|
||||
|---|---|---|---|---|
|
||||
| claude | 2026-10-02 | Claude Code 2.1.284 (CI image pin) | v1.91.13.0 (1c54555) | gate-tier evals, real Claude Code sessions: https://github.com/garrytan/gstack/actions/runs/37033755821 |
|
||||
|
||||
Codex stays `experimental` until its periodic cases (`codex-discover-skill`,
|
||||
`codex-review`) pass on the CI image's pinned Codex CLI and a row is added.
|
||||
|
||||
## GitHub Copilot CLI (#393)
|
||||
|
||||
Shipped as `experimental` (`hosts/copilot.ts`, `./setup --host copilot`):
|
||||
skills in `~/.copilot/skills/gstack-*`, runtime root `~/.copilot/skills/gstack`
|
||||
with a `.source-path` marker so `/gstack-upgrade` finds the checkout, names
|
||||
prefixed `gstack-` (Copilot's built-in `/review` would win otherwise),
|
||||
`disable-model-invocation: true` on sensitive skills, and a one-paragraph
|
||||
tool-name glossary (`ask_user`, `exit_plan_mode`, `task`, `view`, `skill`)
|
||||
inserted ahead of the preamble's STATUS rules. A shared directory was not
|
||||
enough: Copilot CLI stopped reading `~/.claude/skills` in 1.0.36, gstack never
|
||||
writes `~/.agents/skills`, and a Codex render carries Codex paths and the
|
||||
Codex model profile.
|
||||
|
||||
Disposition of the community attempts:
|
||||
|
||||
| PR | Decision | Why |
|
||||
|---|---|---|
|
||||
| #2323 (@andrey-esipov) | Selectively reused, with credit | Kept: `~/.copilot/skills` and `.github/skills` roots, `copilot`-binary detection, `.source-path`, the `gstack-` prefix, the 1024-char description limit, uninstall coverage. Not kept: new schema fields and generator edits (predate `defineHost()`), an unguarded `rm -rf` of the runtime root (#2142), links to `setup`/`scripts`/Claude sources inside a recursively scanned dir, a test-only install backdoor in setup, and a blanket AskUserQuestion → `ask_user` rewrite. |
|
||||
| #396 (@ridermw) | Rejected | Sed-rewrites Codex output into a shared `.agents/skills`; its E2E runner copies real `~/.copilot` credentials into a temp HOME. Its session scanner is a candidate for a later `/retro global` change. |
|
||||
| #487 (@ridermw) | Rejected | Relies on `~/.claude/skills`, which Copilot CLI no longer reads. |
|
||||
| #1852 (@lolisaigao1234) | Rejected | Detects `gh` instead of `copilot`, installs to a directory Copilot does not scan, links `bin` without `lib`. |
|
||||
|
||||
Unverified without a real Copilot CLI (why it is not `full`): autopilot may
|
||||
answer `ask_user` itself, plan mode blocks mutating shell commands, symlinked
|
||||
skill dirs need Copilot CLI 1.0.62 or later, and `COPILOT_HOME` other than
|
||||
`~/.copilot` is refused for now.
|
||||
|
||||
## Instruction-only hosts
|
||||
|
||||
OpenClaw, Hermes, Slate and GBrain have no install arm. Hermes stays
|
||||
instruction-only until someone certifies a native install (#2826 measured one
|
||||
on Hermes 0.21.0): `./setup --host hermes` prints the digest path and the
|
||||
generator command, and the Hermes render writes `name: gstack-<skill>` so
|
||||
Hermes, which indexes skills by frontmatter name, cannot shadow them (#2825).
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -140,8 +328,7 @@ bun test test/gen-skill-docs.test.ts
|
||||
bun test test/host-config.test.ts
|
||||
```
|
||||
|
||||
The parameterized smoke tests automatically pick up the new host. Zero test
|
||||
code to write. They verify: output exists, no path leakage, valid frontmatter,
|
||||
The parameterized smoke tests automatically pick up the new host. They verify: output exists, no path leakage, valid frontmatter,
|
||||
freshness check passes, and outside-review skills match each host's exclusions.
|
||||
|
||||
### 6. Update README.md
|
||||
@@ -157,6 +344,9 @@ Key fields:
|
||||
|
||||
| Field | Purpose |
|
||||
|-------|---------|
|
||||
| `tier` | `full`, `experimental`, or `instruction-only` (see Host tiers) |
|
||||
| `capabilities` | What the host runtime offers; rendering and docs must not claim more |
|
||||
| `frontmatter.nameMatchesDirectory` | Write `gstack-<skill>` into `name:` for hosts that index by frontmatter name |
|
||||
| `defaultModel` | Model overlay rendered when generation gets no explicit `--model` (validated against `ALL_MODEL_NAMES` in `scripts/models.ts`) |
|
||||
| `frontmatter.mode` | `allowlist` (keep only listed) or `denylist` (strip listed) |
|
||||
| `frontmatter.descriptionLimit` | Max chars, `null` for no limit |
|
||||
|
||||
+125
-1
@@ -1,8 +1,132 @@
|
||||
# CHANGELOG entry format
|
||||
# CHANGELOG + VERSION style
|
||||
|
||||
Moved verbatim from CLAUDE.md (token-load reduction). Read this BEFORE
|
||||
writing any `## [X.Y.Z]` CHANGELOG entry.
|
||||
|
||||
## Versioning and branch-scoped entries
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit). The autonomous version
|
||||
policy (choose versions without asking; default to PATCH) stays in CLAUDE.md
|
||||
as the single source.
|
||||
|
||||
**Versioning invariant (workspace-aware ship).** VERSION is a monotonic ordered
|
||||
release identifier, not a strict semver commitment. The bump level
|
||||
(major/minor/patch/micro) expresses intent at ship time. Queue-advancing past a
|
||||
claimed version within the same bump level is explicitly permitted — if branch A
|
||||
claims v1.7.0.0 as a MINOR and branch B is also a MINOR, B lands at v1.8.0.0
|
||||
(still a MINOR relative to main). Downstream consumers must NOT rely on
|
||||
"MINOR = feature-only, PATCH = fix-only" as a strict contract. This is why
|
||||
`bin/gstack-next-version` advances within the chosen bump level rather than
|
||||
repicking the level when collisions happen.
|
||||
|
||||
**package.json carries the npm-valid translation, not VERSION verbatim.**
|
||||
VERSION stays the 4-digit source of truth (e.g. `1.67.0.0`); package.json and
|
||||
any subdirectory manifests with a `version` field get the 3-digit npm-valid
|
||||
translation (`1.67.0`), and lockfile `version` fields sync only when the
|
||||
lockfile already exists. `bin/gstack-version-bump` (via `lib/version-source.ts`)
|
||||
owns the translation and judges drift on translated forms — do NOT "fix" the
|
||||
apparent mismatch by hand, and do not write a 4-digit version into
|
||||
package.json (npm rejects it). Rationale and translation rules live in the
|
||||
`lib/version-source.ts` header; `test/gstack-version-bump.test.ts` pins the
|
||||
contract.
|
||||
|
||||
**VERSION and CHANGELOG are branch-scoped.** Every feature branch that ships gets its
|
||||
own version bump and CHANGELOG entry. The entry describes what THIS branch adds —
|
||||
not what was already on main.
|
||||
|
||||
**The CHANGELOG entry is the diff between main and the shipping branch — what users
|
||||
get when they upgrade. NOT how the branch got there.** A reader landing on the entry
|
||||
should learn what they can do now that they couldn't before; they should not learn
|
||||
about the branch's internal version bumps, the bugs we caught and fixed mid-branch,
|
||||
the plan reviews we ran, or the commits we squashed. That is branch development
|
||||
narrative. It belongs in PR descriptions and commit messages, not CHANGELOG.
|
||||
|
||||
**Never reference branch-internal versions in a CHANGELOG entry.** If your branch
|
||||
bumped VERSION from v1.5.0.0 → v1.5.1.0 → v1.6.0.0 during development and only the
|
||||
final v1.6.0.0 ships to main, the entry must read as if v1.5.1.0 never existed.
|
||||
Concretely, NEVER write:
|
||||
- "v1.5.1.0 had a bug that v1.6.0.0 fixes" — readers don't know about v1.5.1.0; it's
|
||||
a branch-internal artifact.
|
||||
- "The shipping headline of v1.5.1.0 was broken because..." — same reason. From main's
|
||||
perspective, v1.5.1.0 was never released.
|
||||
- "Pre-fix tests encoded the broken behavior" — that's a contributor's victory lap,
|
||||
not a user benefit.
|
||||
- "Two surgical edits, both in the dispatch path" — micro-narrative of the patch.
|
||||
|
||||
Instead, describe the released system: "Browser-skills run end-to-end with the
|
||||
expected tab-access semantics." If a property of the shipped system is worth calling
|
||||
out (e.g., "skill spawns get permissive tab access; pair-agent tunnel tokens require
|
||||
ownership"), document it as a property, not as a fix. The shipped system is what
|
||||
the user gets; the path to that system is invisible to them.
|
||||
|
||||
**When to write the CHANGELOG entry:**
|
||||
- At `/ship` time (Step 13), not during development or mid-branch.
|
||||
- The entry covers ALL commits on this branch vs the base branch.
|
||||
- Never fold new work into an existing CHANGELOG entry from a prior version that
|
||||
already landed on main. If main has v0.10.0.0 and your branch adds features,
|
||||
bump to v0.10.1.0 with a new entry — don't edit the v0.10.0.0 entry.
|
||||
|
||||
**Key questions before writing:**
|
||||
1. What branch am I on? What did THIS branch change?
|
||||
2. Is the base branch version already released? (If yes, bump and create new entry.)
|
||||
3. Does an existing entry on this branch already cover earlier work? (If yes, replace
|
||||
it with one unified entry for the final version.)
|
||||
|
||||
**Merging main does NOT mean adopting main's version.** When you merge origin/main into
|
||||
a feature branch, main may bring new CHANGELOG entries and a higher VERSION. Your branch
|
||||
still needs its OWN version bump on top. If main is at v0.13.8.0 and your branch adds
|
||||
features, bump to v0.13.9.0 with a new entry. Never jam your changes into an entry that
|
||||
already landed on main. Your entry goes on top because your branch lands next.
|
||||
|
||||
**After merging main, always check:**
|
||||
- Does CHANGELOG have your branch's own entry separate from main's entries?
|
||||
- Is VERSION higher than main's VERSION?
|
||||
- Is your entry the topmost entry in CHANGELOG (above main's latest)?
|
||||
If any answer is no, fix it before continuing.
|
||||
|
||||
**After any CHANGELOG edit that moves, adds, or removes entries,** immediately run
|
||||
`grep "^## \[" CHANGELOG.md` to verify no duplicates and a sensible reverse-chronological
|
||||
order. Gaps between version numbers are fine. A branch that ships at v1.6.4.0 without
|
||||
a prior v1.5.2.0 or v1.5.3.0 entry on main is correct — those were branch-internal
|
||||
version numbers that never landed. Do not back-fill gaps with placeholder entries.
|
||||
|
||||
**Never orphan branch-internal versions.** If your branch bumped VERSION several times
|
||||
during development (v1.5.1.0 → v1.5.2.0 → v1.6.4.0, say) and those earlier entries were
|
||||
never released to main, the final ship consolidates ALL of them into a single entry at
|
||||
the final version (v1.6.4.0). Collapse them — delete the old entries and move their
|
||||
content into the final entry, re-version table columns accordingly. Readers see one
|
||||
release, not a branch diary. Gaps are fine (v1.6.3.0 → v1.6.4.0 with no v1.5.x
|
||||
in between on main is correct).
|
||||
|
||||
CHANGELOG.md is **for users**, not contributors. Write it like product release notes:
|
||||
|
||||
- Lead with what the user can now **do** that they couldn't before. Sell the feature.
|
||||
- Use plain language, not implementation details. "You can now..." not "Refactored the..."
|
||||
- **Never mention TODOS.md, internal tracking, eval infrastructure, or contributor-facing
|
||||
details.** These are invisible to users and meaningless to them.
|
||||
- Put contributor/internal changes in a separate "For contributors" section at the bottom.
|
||||
- Every entry should make someone think "oh nice, I want to try that."
|
||||
- No jargon: say "every question now tells you which project and branch you're in" not
|
||||
"AskUserQuestion format standardized across skill templates via preamble resolver."
|
||||
|
||||
**Only document what shipped between main and this change.** Readers do not care how
|
||||
we got here. Keep out of the CHANGELOG, always:
|
||||
|
||||
- Branch resyncs, merge commits with main, rebase activity.
|
||||
- Plan approvals, review outcomes (CEO / eng / design / outside-voice / codex findings),
|
||||
AskUserQuestion decisions, scope negotiations.
|
||||
- "Work queued," "plan approved," "in-progress," "will ship later" — the CHANGELOG
|
||||
documents what DID ship, not what MIGHT ship.
|
||||
- Version-bump housekeeping when no user-facing work actually landed.
|
||||
|
||||
If the diff between the base branch version and this version has no user-facing change
|
||||
(only merges, only CHANGELOG edits, only placeholder work), the honest entry is one
|
||||
sentence: "Version bump for branch-ahead discipline. No user-facing changes yet." Stop
|
||||
there. Do not pad. Do not explain the plan that will ship eventually. Do not narrate
|
||||
the branch's history. When real work lands, the entry will replace this at /ship time.
|
||||
|
||||
## Entry format
|
||||
|
||||
### Release-summary format (every `## [X.Y.Z]` entry)
|
||||
|
||||
Every version entry in `CHANGELOG.md` MUST start with a release-summary section in
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
# Contributor reference
|
||||
|
||||
Contributor-only detail moved verbatim from CLAUDE.md, which Claude Code
|
||||
loads into every session and caps at 40,000 characters (#2096,
|
||||
`test/claude-md-size.test.ts`). CLAUDE.md keeps a one-line pointer to each
|
||||
section here.
|
||||
|
||||
## Skill size budgets
|
||||
|
||||
**Token ceiling:** Generated SKILL.md files trip a warning above 160KB (~40K tokens).
|
||||
This is a "watch for feature bloat" guardrail, not a hard gate. Modern flagship
|
||||
models have 200K-1M context windows, so 40K is 4-20% of window, and prompt caching
|
||||
makes the marginal cost of larger skills small. The ceiling exists to catch runaway
|
||||
preamble/resolver growth, not to force compression on carefully-tuned big skills
|
||||
(`ship`, `plan-ceo-review`, `office-hours` legitimately pack 25-35K tokens of
|
||||
behavior). If you blow past 40K, the right fix is usually: (1) look at WHAT grew,
|
||||
(2) if one resolver added 10K+ in a single PR, question whether it belongs inline
|
||||
or as a reference doc, (3) only compress carefully-tuned prose as a last resort —
|
||||
cuts to the coverage audit, review army, or voice directive have real quality cost.
|
||||
|
||||
A second, harder ceiling guards the DISCOVERY surface: `test/catalog-budget.test.ts`
|
||||
caps the aggregate frontmatter `name` + `description` across all skills at
|
||||
`CATALOG_BUDGET_TOKEN_EQUIVALENTS` (1,194 today; each new skill ratchets it)
|
||||
token-equivalents (260-byte per-skill sub-cap), counted through the shared census
|
||||
in `test/helpers/skill-census.ts`. This one is enforced, not a warning — every
|
||||
host loads the full catalog every session, so growth here taxes every
|
||||
conversation. The failure message carries the re-measure + ratchet protocol.
|
||||
`bin/gstack-context-bill` shows the full token bill-of-materials for a skills
|
||||
tree (always-on vs per-invocation, `--diff`, `--budget`; `--exact` opts into the
|
||||
real tokenizer and POSTs file text to api.anthropic.com with an egress receipt).
|
||||
|
||||
The context-budget ratchet (`test/context-budget-ratchet.test.ts`, free, runs
|
||||
in `bun run test`) pins ABSOLUTE ceilings on two more ledgers: the always-on
|
||||
FULL-frontmatter aggregate (catalog-budget counts only name+description) and
|
||||
each skill's per-invocation eager tokens (SKILL.md + forced-read references —
|
||||
size floors and parity ratios guard these relatively, not absolutely), graded
|
||||
against `test/fixtures/context-budget.json`. A skill that grows past its
|
||||
ceiling fails; a new skill fails until it's consciously budgeted. For
|
||||
legitimate growth or a landed reduction, re-run
|
||||
`bun test/helpers/capture-context-budget.ts` and commit the refreshed fixture
|
||||
in the same commit, so ceilings ratchet down and every win is locked.
|
||||
|
||||
## Egress receipts
|
||||
|
||||
**Egress receipts at every off-machine sink** (v1.63.0.0+). Every gstack-initiated
|
||||
send off the machine MUST write a hash-chained receipt to
|
||||
`~/.gstack/security/egress.jsonl` BEFORE the send: TypeScript callers use
|
||||
`writeReceipt` from `lib/egress-receipt.ts`; shell scripts source
|
||||
`bin/gstack-egress-lib.sh` and use `_receipted_curl` / `_receipted_git`. Failure
|
||||
polarity is per-class: fail-closed for sensitive sinks (brain-sync, memory-ingest,
|
||||
gbrain-sync, telemetry, ngrok tunnels, mcp-verify, supabase-provision, and the
|
||||
Memorable bridge's per-prompt memorable-recall hand-off), fail-open
|
||||
+ stderr warning for user-facing ones (design OpenAI calls, update-check,
|
||||
dashboards, git-class ops). The new-sink scanner in
|
||||
`test/egress-receipt-wiring.test.ts` fails CI on an unreceipted `curl` /
|
||||
`git push` / `fetch` to a non-loopback host unless the file carries a reasoned
|
||||
entry in its `SCANNER_EXEMPT` list (user-directed page fetches, reachability
|
||||
probes, instruction strings, skill prose) — if you add a new off-machine sink,
|
||||
wire it through the helpers and add it to the enumerated sink list. `aside exec`
|
||||
(a gstack-composed prompt sent to Aside's agent) is a fail-open user-facing
|
||||
sink: skills call it through the `_aside_exec` wrapper that
|
||||
`scripts/resolvers/aside.ts` renders, never bare. Inspect with
|
||||
`bin/gstack-egress` (`list` | `verify`, exit 3 on tamper | `grants`). Threat
|
||||
model: forensic observability of ATTEMPTED egress, not an exfiltration control.
|
||||
|
||||
## Skill linking and ownership
|
||||
|
||||
**Prefix setting:** Setup creates real directories (not symlinks) at the top level
|
||||
with a SKILL.md symlink inside (e.g., `qa/SKILL.md -> gstack/qa/SKILL.md`), plus
|
||||
links to each skill's runtime assets (sections/, templates, checklists — everything
|
||||
except SKILL.md, tests, build output, and `.tmpl` sources). Alias skills
|
||||
(`_gstack-command`, `connect-chrome`) install as rewritten copies, never symlinks.
|
||||
This ensures Claude discovers them as top-level skills, not nested under `gstack/`.
|
||||
Names are either short (`qa`) or namespaced (`gstack-qa`), controlled by
|
||||
`skill_prefix` in `~/.gstack/config.yaml`. Pass `--no-prefix` or `--prefix` to
|
||||
skip the interactive prompt.
|
||||
|
||||
**Ownership gate (#2119):** `setup` writes a `.gstack-owned` marker into every
|
||||
skill directory it creates, and `setup` (the linker, the alias installer, both
|
||||
prefix-flip cleanups, and the retired-skill prune) and `bin/gstack-relink` only
|
||||
delete or link over an entry they can prove is gstack's. Strong proof (a
|
||||
symlink resolving into gstack, or the marker) allows deleting or refreshing the
|
||||
whole directory. Weak proof (a
|
||||
real SKILL.md byte-identical to the source, or carrying gen-skill-docs' two-line
|
||||
banner) covers only that one file, and a weakly-proven file that differs is
|
||||
moved to `~/.gstack/backups/skills/<ts>/<skill>/SKILL.md` before gstack links
|
||||
over it. Anything else is a foreign skill: skipped, and named in setup's final
|
||||
summary. The rule lives in two copies (`setup` and `bin/gstack-relink`); keep
|
||||
them in sync until the shared helper filed in TODOS.md lands. The retired-skill
|
||||
prune (`_prune_stale_generated`) applies the same strong/weak split to renders
|
||||
of skills that no longer exist, through its own gate
|
||||
(`_owned_for_windows_refresh`: a real host directory is a candidate only when
|
||||
its SKILL.md carries the generated banner; the marker and byte identity are not
|
||||
consulted): it scans the render tree and every host skills dir,
|
||||
deletes a real render directory, removes a host symlink only when it resolves
|
||||
into gstack, cleans a bannered real directory through `_cleanup_weak_dir`,
|
||||
never follows a symlink inside the render tree, and recognizes a skill renamed
|
||||
through its frontmatter `name:`. Pinned by `test/setup-link-ownership.test.ts`,
|
||||
`test/setup-cleanup-orphans.test.ts`, `test/setup-prune-stale-generated.test.ts`,
|
||||
and `test/relink.test.ts`.
|
||||
|
||||
## Redaction guard
|
||||
|
||||
Shared redaction engine catches credentials, PII, and legal/damaging content
|
||||
before it reaches an external sink (codex dispatch, GitHub issue/PR body, pushed
|
||||
commit). It is a **guardrail, not airtight enforcement** — `git push --no-verify`,
|
||||
direct `gh issue create`, and `GSTACK_REDACT_PREPUSH=skip` all bypass it. It
|
||||
catches accidents and carelessness, the 99% case. Do not claim it stops a
|
||||
determined leaker (a CHANGELOG line that does would fail a hostile screenshotter).
|
||||
|
||||
- **Engine + taxonomy:** `lib/redact-patterns.ts` (the single source of truth —
|
||||
3 tiers; HIGH = genuinely-secret credentials that block, MEDIUM = PII/legal/
|
||||
internal + high-FP credential shapes that confirm via AskUserQuestion, LOW =
|
||||
FYI) and `lib/redact-engine.ts` (pure `scan()` + `applyRedactions()`).
|
||||
Calibration matters: a gate that cries wolf gets ignored, so context-variable
|
||||
shapes (Stripe `pk_live_`, Google `AIza`, JWT, env `*_KEY=`) sit at MEDIUM.
|
||||
- **CLI:** `bin/gstack-redact` (exit 0 clean / 2 MEDIUM / 3 HIGH; `--json`,
|
||||
`--auto-redact`, `--repo-visibility`, `--from-file`). `bin/gstack-redact-prepush`
|
||||
is the opt-in git hook.
|
||||
- **Skill docs are generated** from `scripts/resolvers/redact-doc.ts`
|
||||
(`{{REDACT_INVOCATION_BLOCK:<sink>}}`) so /spec,
|
||||
/cso, /ship, /document-release, /document-generate never drift from the engine.
|
||||
- **Scan-at-sink:** always scan the EXACT bytes that will be sent — write to a
|
||||
temp file, scan that file, pass the SAME file to `gh`/`git`. Never scan a string
|
||||
then re-render (that reopens a scan-vs-send gap).
|
||||
- **Visibility (no tier promotion):** resolve once per run, order = local config
|
||||
(`gstack-config get redact_repo_visibility`, ~/.gstack so never committed) → gh
|
||||
→ glab → unknown(=public-strict). Public repos get STERNER per-finding
|
||||
confirmation (no batch-acknowledge, no silent-proceed); MEDIUM is never
|
||||
auto-promoted to HIGH.
|
||||
- **Tool-attributed fences:** wrap Codex/Greptile/eval output in ` ```codex-review `
|
||||
/ ` ```greptile ` fences so example credentials those tools quote WARN-degrade
|
||||
instead of blocking. A live-format credential inside the fence still blocks.
|
||||
- **Config keys:** `redact_repo_visibility` (public|private|unknown, local-only
|
||||
override for repos gh/glab can't read), `redact_prepush_hook` (true|false).
|
||||
There is intentionally NO key to disable HIGH blocking.
|
||||
- **Audit:** the /spec semantic pass appends a content-free record (categories +
|
||||
body sha256, no spec text) to `~/.gstack/security/semantic-reviews.jsonl` (0600).
|
||||
|
||||
## Checking out PRs from garrytan-agents
|
||||
|
||||
When the user says "check out <PR link>" and the PR is from `garrytan-agents/gstack`
|
||||
(or any other fork that is NOT a collaborator on `garrytan/gstack`), do NOT just
|
||||
`gh pr checkout`. Fork PRs don't receive base-repo secrets (`ANTHROPIC_API_KEY`,
|
||||
`OPENAI_API_KEY`, etc.), so the eval/E2E CI jobs fail with empty-env auth errors
|
||||
regardless of what's set on the base repo.
|
||||
|
||||
**Workflow:** push the branch to `garrytan/gstack` (the base repo) and re-target
|
||||
the PR from there.
|
||||
|
||||
Concretely, after `gh pr checkout <N>`:
|
||||
|
||||
1. Note the original PR number and head branch name.
|
||||
2. Push the same branch to the base repo: `git push origin HEAD:<branch-name>`
|
||||
(origin = `garrytan/gstack`, since the worktree is set up with that remote).
|
||||
3. Close the fork PR (`gh pr close <N> --comment "moving to base-repo branch for secret access"`).
|
||||
4. Open a new PR from the base-repo branch: `gh pr create --base main --head <branch-name>`.
|
||||
5. New PR's workflows will get secrets automatically.
|
||||
|
||||
Why not fix it on the fork side? `garrytan-agents` isn't a collaborator on
|
||||
`garrytan/gstack`. Adding it as a collaborator (option A) or flipping the
|
||||
repo-wide "send secrets to fork PRs" toggle (option B) would let secrets reach
|
||||
fork PRs from anyone — broader blast radius than just moving this one branch.
|
||||
Option C (this section) keeps secret-distribution scope tight.
|
||||
|
||||
If the user asks you to skip the move (e.g., "just leave it as a fork PR"),
|
||||
respect that — eval CI will fail with empty-env auth, but check-freshness,
|
||||
workflow-lint, and windows-tests will still pass on the fork PR.
|
||||
@@ -647,3 +647,140 @@ a broken tree can't masquerade as flaky). The required CI free lane and the
|
||||
Windows lane set the retry knob too, appending every flaky pass to the JSONL ledger it uploads
|
||||
(`GSTACK_FLAKE_LEDGER`) — a flaky pass never reds the lane, but it never
|
||||
disappears either.
|
||||
|
||||
## Test selection and tiers
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit).
|
||||
|
||||
**Diff-based test selection:** `test:evals` and `test:e2e` auto-select tests based
|
||||
on `git diff` against the base branch. Each test declares its file dependencies in
|
||||
`test/helpers/touchfiles.ts`. Changes to global touchfiles (session-runner, eval-store,
|
||||
touchfiles.ts itself) trigger all tests. Use `EVALS_ALL=1` or the `:all` script
|
||||
variants to force all tests. Run `eval:select` to preview which tests would run.
|
||||
|
||||
**Two-tier system:** Tests are classified as `gate` or `periodic` in `E2E_TIERS`
|
||||
(in `test/helpers/touchfiles.ts` — a facade over `touchfiles-data.ts` +
|
||||
`test-selection.ts`). CI runs the changed fast PR profile and selected judges
|
||||
per PR via evals.yml's sliced lane
|
||||
(planner manifest → executors → fail-closed report; engine =
|
||||
scripts/test-paid-shards.ts, the same runner as local eval:bg:pr); the free
|
||||
suite runs on every PR via `.github/workflows/free-tests.yml` (a REQUIRED
|
||||
check, secretless — fork PRs get real signal); ALL periodic tests run weekly
|
||||
via evals-periodic.yml (EVALS_ALL, minus the reasoned exclusions in
|
||||
`test/helpers/periodic-exclude-data.ts` — reason + tracking required per
|
||||
entry), plus a weekly EVALS_ALL gate census. Use `EVALS_TIER=gate` or
|
||||
`EVALS_TIER=periodic` to filter locally. When adding new E2E tests, classify them:
|
||||
1. Safety guardrail or deterministic functional test? -> `gate`
|
||||
2. Quality benchmark, Opus model test, or non-deterministic? -> `periodic`
|
||||
3. Requires external service (Codex, Gemini)? -> `periodic`
|
||||
|
||||
Tier declarations are enforced by `test/e2e-tier-alignment.test.ts` (free, runs
|
||||
in `bun test`): a `skill-e2e-*` file named in a touchfiles dep list whose
|
||||
`EVALS_TIER` self-gate disagrees with its declared tier in `E2E_TIERS` fails the
|
||||
suite. Files not named in any dep list are reported, not enforced — keep both
|
||||
in sync.
|
||||
|
||||
## Free suite runner, judge reuse and engine skips
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit).
|
||||
|
||||
`bun run test` routes through `scripts/test-free-shards.ts` (N concurrent
|
||||
shard processes, serial within each, packed by recorded per-file durations
|
||||
when `scripts/free-test-durations.json` exists — refresh occasionally with
|
||||
`bun run test:free --record-durations`; strict-output classification per
|
||||
shard: a shard without bun's terminal summary line FAILS — silent truncation
|
||||
cannot report green). `TREE_MUTATING` lists the files that still run in their
|
||||
own trailing serial shard (today only `test/bootstrap-retention.test.ts`, for
|
||||
host-wide procfs visibility); gen-skill-docs tests no longer need it (main()
|
||||
guard, and `--out-dir` renders every host into mkdtemps — see
|
||||
docs/TESTING_INTERNALS.md). Never type bare `bun test` for the suite: it
|
||||
walks the whole repo, loading paid eval files and missing the strict
|
||||
classifier.
|
||||
It covers skill validation, gen-skill-docs quality checks, browse
|
||||
integration tests, the Aside contract pins, and the render-wrapper pins.
|
||||
`bun run test:pr` runs the selected short live behaviors and quality judges.
|
||||
It reports deferred broad coverage; unknown dependencies restore the full gate,
|
||||
and an unmapped prompt without registered coverage blocks planning. Full free
|
||||
acceptance and required PR checks must pass before publishing. CI can reuse the
|
||||
16 workflow-judge passes for 24 hours when their complete consumed inputs and
|
||||
runtime match; records preserve original provenance. The cookie workflow's custom
|
||||
input, the other 11 judge cases, dynamic agent tests, and local runs without
|
||||
scoped cache configuration stay fresh.
|
||||
Scheduled/manual full coverage and `test:release` always run fresh.
|
||||
See [testing policy](CONTRIBUTING.md#test-tiers) for commands and measured targets.
|
||||
Anything that needs Aside
|
||||
itself (`test/skill-e2e-aside.test.ts`, the Aside qa/design E2E cases, the
|
||||
live render in `test/aside-render.test.ts`) runs only on a Mac with the Aside
|
||||
app open and self-skips elsewhere (`asideAvailable()`). make-pdf's render
|
||||
gates and `test/skill-e2e-diagram.test.ts` run through whichever engine
|
||||
resolves (`browserAvailable()` — Aside, or the browse binary CI builds with
|
||||
`bun run build:gates`) and skip only when neither exists; the fallback
|
||||
engine's own tests run everywhere.
|
||||
|
||||
### Real-home tripwire
|
||||
|
||||
No free test may write the developer's real home. `scripts/lib/free-home-guard.ts`
|
||||
snapshots `~/.gstack`, `~/.claude`, `~/.codex`, `~/.agents` and `~/.config/gstack`
|
||||
(live session logs excluded) and fails the run when an entry changes.
|
||||
|
||||
- A shard that runs alone (`--shard`, CI, the exclusive host-state shard, the
|
||||
flaky retry) owns its window, so the failure names its files.
|
||||
- Full-suite shards run concurrently on one HOME and cannot tell whose write a
|
||||
change was. The runner guards that whole phase once and names no shard.
|
||||
- `bun run scripts/test-free-shards.ts --attribute-home` names the writer: it
|
||||
runs every file alone with a private HOME (browser cache and git identity
|
||||
still come from the real home) and reports each file that wrote a watched
|
||||
surface. The retained shard directory keeps the written files as evidence.
|
||||
|
||||
Fix a writer with `usePrivateStateRoot()` (`test/helpers/private-state-root.ts`)
|
||||
or a child HOME/GSTACK_HOME, and resolve product state paths at write time,
|
||||
never at import.
|
||||
|
||||
## Running evals as an agent: detach
|
||||
|
||||
Moved verbatim from CLAUDE.md (#2096 size limit). The short rule stays in
|
||||
CLAUDE.md; this is the full mechanism.
|
||||
|
||||
When **you (an agent/harness)** launch a long eval/benchmark run, run it through
|
||||
`bin/gstack-detach` — NEVER as a plain backgrounded Bash task. A plain background
|
||||
task lives in the harness's process group, so a SIGTERM ("polite quit") on a turn
|
||||
boundary, a stopped Monitor, or an interruption kills the run mid-flight (observed:
|
||||
`script "test:gate" was terminated by signal SIGTERM` ~40 min into a run). On macOS
|
||||
the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
|
||||
(escapes the group SIGTERM) wrapped in `caffeinate -i` (blocks idle-sleep).
|
||||
|
||||
- Use the `eval:bg*` scripts (`eval:bg`, `eval:bg:all`, `eval:bg:gate`,
|
||||
`eval:bg:periodic`) — they wrap the eval command in `gstack-detach` with the
|
||||
machine-wide `gstack-evals` lock (concurrent worktrees serialize instead of
|
||||
saturating the shared model API), a per-tier watchdog, and a **run-scoped** log
|
||||
under `~/.gstack-dev/eval-runs/` (no shared-`/tmp` collision). Each prints its
|
||||
log path. `eval:bg:gate` / `eval:bg:periodic` run their tier through the
|
||||
sharded paid runner (`scripts/test-paid-shards.ts`, also exposed as
|
||||
`test:gate:sharded` / `test:periodic:sharded`): one Bun process per test
|
||||
file, an external wall-clock timeout that kills the shard's process GROUP
|
||||
(stray `claude`/`codex` grandchildren included), a per-shard
|
||||
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
|
||||
constructor, and an aggregate that separates failed vs timed-out vs
|
||||
never-started shards — the detach timeouts (the `--timeout` values on
|
||||
package.json's `eval:bg:gate` / `eval:bg:periodic`;
|
||||
floor enforced against the live shard census by
|
||||
test/eval-detach-timeout-floor.test.ts)
|
||||
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
|
||||
process count (default 8); `EVALS_CONCURRENCY` is bun's --max-concurrency
|
||||
WITHIN a shard (default 2) — they are deliberately separate knobs. `eval:list` / `eval:compare` /
|
||||
`eval:summary` / `eval:flake-rank` read the shard dirs too. Or call
|
||||
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] --
|
||||
<cmd>` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never
|
||||
pass keys in argv).
|
||||
- Then **poll the printed logfile** with a death-aware watcher: break on the
|
||||
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (success AND failure are
|
||||
both marked, so silence is never mistaken for success). The detached run survives
|
||||
even if your watcher gets reaped, so re-checking the log always works. Keep
|
||||
checking until the sentinel appears or the user tells you to stop; a long run is
|
||||
expected, and a promise to check later is not a result. At each check, report
|
||||
which tests passed, which are still running, and any failures so far.
|
||||
- Why the lock: a shared dev box with several Conductor worktrees will rate-limit
|
||||
the model API if two eval suites run at once (15-way concurrency each), which
|
||||
mass-times-out E2E tests. The lock makes the second run WAIT, not collide.
|
||||
- Humans running `bun run test:evals` foreground in their own terminal don't need
|
||||
this — Ctrl-C is intended there. Detachment is for agent-launched runs only.
|
||||
+3
-1
@@ -153,9 +153,11 @@ applied per run and never cached.
|
||||
Skill blocks resolve the root with
|
||||
|
||||
```bash
|
||||
eval "$(~/.claude/skills/gstack/bin/gstack-paths)"; : "${GSTACK_STATE_ROOT:?gstack-paths failed; reinstall with ./setup or /gstack-upgrade}"
|
||||
GSTACK_STATE_ROOT=$(~/.claude/skills/gstack/bin/gstack-paths --get GSTACK_STATE_ROOT); : "${GSTACK_STATE_ROOT:?gstack-paths failed; reinstall with ./setup or /gstack-upgrade}"
|
||||
```
|
||||
|
||||
(No `eval`: worktree-isolated Claude Code sessions refuse it.)
|
||||
|
||||
The second half stops the block instead of writing under `/` if the resolver
|
||||
is broken. If you see `gstack-paths failed` or
|
||||
`cannot resolve the gstack state root`, reinstall: run `./setup` in the gstack
|
||||
|
||||
Reference in new issue
Block a user