From 8011b51ae6540f7cdb7263a2d66aa05f75afa572 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Tue, 18 Aug 2026 16:41:22 -0700 Subject: [PATCH] docs: post-ship documentation sync (v1.67.2.0) Sol exact-match and near-miss warning documented in README; CODEX_HOME-aware uninstall and troubleshooting paths; hermetic auth.json-only detail and the build-clobber gotcha in CLAUDE.md; eval-store location corrected in ARCHITECTURE.md; defaultModel row in the ADDING_A_HOST field reference; resolver test count corrected in the CHANGELOG entry. --- ARCHITECTURE.md | 4 ++-- CHANGELOG.md | 2 +- CLAUDE.md | 12 +++++++++--- README.md | 8 +++++--- TODOS.md | 2 ++ docs/ADDING_A_HOST.md | 1 + 6 files changed, 20 insertions(+), 9 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 4ff31eb91..c426aa1a5 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -415,7 +415,7 @@ The `parseNDJSON()` function is pure — no I/O, no side effects — making it i **Machine-readable diagnostics:** Each test result includes `exit_reason` (success, timeout, error_max_turns, error_api, exit_code_N), `timeout_at_turn`, and `last_tool_call`. This enables `jq` queries like: ```bash -jq '.tests[] | select(.exit_reason == "timeout") | .last_tool_call' ~/.gstack-dev/evals/_partial-e2e.json +jq '.tests[] | select(.exit_reason == "timeout") | .last_tool_call' ~/.gstack/projects//evals/_partial-e2e.json ``` ### Eval persistence (`test/helpers/eval-store.ts`) @@ -425,7 +425,7 @@ The `EvalCollector` accumulates test results and writes them in two ways: 1. **Incremental:** `savePartial()` writes `_partial-e2e.json` after each test (atomic: write `.tmp`, `fs.renameSync`). Survives kills. 2. **Final:** `finalize()` writes a timestamped eval file (e.g. `e2e-20260314-143022.json`). The partial file is never cleaned up — it persists alongside the final file for observability. -`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack-dev/evals/`. Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `/shards//` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards//` subdirectories. Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline. +`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack/projects//evals/` (legacy fallback `~/.gstack-dev/evals/`). Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `/shards//` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards//` subdirectories. Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline. ### Test tiers diff --git a/CHANGELOG.md b/CHANGELOG.md index a28b4e869..c2f874242 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,7 +44,7 @@ If you run Codex on `gpt-5.6-sol`, rerun `./setup --host codex` once. Your skill - `setup` resolves the Codex generation model on every run (a read-only TOML lookup), so any install path preserves a Sol user's rendered profile; the codex install summary prints the active profile and its source. #### For contributors -- New free-tier coverage: every resolver branch including hostile-config shapes (12 tests), overlay content pins, the explicit `--model` override through the real generation CLI, real-file periodic-tier classification for both codex E2E files, and invalid `defaultModel` validation. +- New free-tier coverage: every resolver branch including hostile-config shapes (10 tests), overlay content pins, the explicit `--model` override through the real generation CLI, real-file periodic-tier classification for both codex E2E files, and invalid `defaultModel` validation. - Static pins in `test/setup-codex-model.test.ts` hold the load-bearing setup properties: unconditional resolver, quoted `--explicit` argv, fail-closed empty-resolver exit, the Kiro claude-render sandwich, and the `--host kiro` rewrite. - The Sol E2E snapshots the exact prior `.agents` render and restores it in `beforeAll`, so the shared tree never stays Sol-flavored for goldens, parallel shards, or symlinked installs. Fixture commits disable gpg signing so the eval runs under any global git config. diff --git a/CLAUDE.md b/CLAUDE.md index 477aaff58..ced80711a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,8 +26,10 @@ bun run slop # full slop-scan report (all files) bun run slop:diff # slop findings in files changed on this branch only ``` -`test:evals` requires `ANTHROPIC_API_KEY`. Codex E2E tests (`test/codex-e2e.test.ts`) -use Codex's own auth from `~/.codex/` config — no `OPENAI_API_KEY` env var needed. +`test:evals` requires `ANTHROPIC_API_KEY`. Codex E2E tests (`test/codex-e2e.test.ts`, +`test/codex-e2e-sol-scope.test.ts`) use Codex's own auth — the hermetic runner copies +only `auth.json` from `${CODEX_HOME:-~/.codex}` and pins `CODEX_HOME` in the child +env — no `OPENAI_API_KEY` env var needed. **Env keys in Conductor workspaces.** The `GSTACK_*` env-shim (v1.39.2.0+, `lib/conductor-env-shim.ts`) promotes `GSTACK_ANTHROPIC_API_KEY` / @@ -194,7 +196,11 @@ SKILL.md files are **generated** from `.tmpl` templates. To update docs: Generation uses each host's `defaultModel` (`claude` for existing hosts, `gpt` for Codex) unless `--model` is explicit. Codex installs additionally read the top-level model from `${CODEX_HOME:-~/.codex}/config.toml`; rerun -`./setup --host codex` after changing that model. +`./setup --host codex` after changing that model. Note: `bun run build` and a +bare `gen:skill-docs --host codex` render the host default (gpt) — if your +Codex config.toml pins a different model, rerun `./setup --host codex` +afterwards to restore your profile (single-owner persistence is filed in +TODOS.md). To add a new browse command: add it to `browse/src/commands.ts` and rebuild. To add a snapshot flag: add it to `SNAPSHOT_FLAGS` in `browse/src/snapshot.ts` and rebuild. diff --git a/README.md b/README.md index 12d6cb77a..f284f8c65 100644 --- a/README.md +++ b/README.md @@ -126,7 +126,9 @@ For Codex, setup reads the top-level `model` from `${CODEX_HOME:-~/.codex}/config.toml` and generates the matching behavioral profile. `gpt-5.6-sol` automatically receives bounded-scope instructions that finish the requested lake without expanding into adjacent cleanup or speculative -hardening. Override detection with `./setup --host codex --model ` — the +hardening. The Sol profile is exact-match only: dated snapshots and other 5.6 +variants get the generic GPT profile, and setup warns on near-misses like +`gpt-5.6-sol-2026-08-01`. Override detection with `./setup --host codex --model ` — the override applies to that run only; set `model` in your Codex `config.toml` to make it stick across upgrades. After changing your Codex model, rerun `./setup --host codex` to regenerate the skills. @@ -382,7 +384,7 @@ rm -rf ~/.claude/skills/gstack rm -rf ~/.gstack # 5. Remove integrations (skip any you never installed) -rm -rf ~/.codex/skills/gstack* 2>/dev/null +rm -rf "${CODEX_HOME:-$HOME/.codex}/skills/gstack"* 2>/dev/null rm -rf ~/.factory/skills/gstack* 2>/dev/null rm -rf ~/.kiro/skills/gstack* 2>/dev/null rm -rf ~/.openclaw/skills/gstack* 2>/dev/null @@ -499,7 +501,7 @@ Data is stored in [Supabase](https://supabase.com) (open source Firebase alterna **Want namespaced commands?** `cd ~/.claude/skills/gstack && ./setup --prefix` — switches from `/qa` to `/gstack-qa`. Useful if you run other skill packs alongside gstack. -**Codex says "Skipped loading skill(s) due to invalid SKILL.md"?** Your Codex skill descriptions are stale. Fix: `cd ~/.codex/skills/gstack && git pull && ./setup --host codex` — or for repo-local installs: `cd "$(readlink -f .agents/skills/gstack)" && git pull && ./setup --host codex` +**Codex says "Skipped loading skill(s) due to invalid SKILL.md"?** Your Codex skill descriptions are stale. Fix: `cd "${CODEX_HOME:-$HOME/.codex}/skills/gstack" && git pull && ./setup --host codex` — or for repo-local installs: `cd "$(readlink -f .agents/skills/gstack)" && git pull && ./setup --host codex` **Windows users:** gstack works on Windows 11 via Git Bash or WSL. Node.js is required in addition to Bun — Bun has a known bug with Playwright's pipe transport on Windows ([bun#4253](https://github.com/oven-sh/bun/issues/4253)). The browse server automatically falls back to Node.js. Make sure both `bun` and `node` are on your PATH. diff --git a/TODOS.md b/TODOS.md index 7655395fe..76c0f67e4 100644 --- a/TODOS.md +++ b/TODOS.md @@ -815,6 +815,8 @@ or reject the value with guidance like openclaw/hermes/gbrain get. **Why:** Silent success with zero effect is the worst failure shape. **Priority:** P4. **Effort:** S. +--- + ## browse server: terminal-agent teardown follow-ups (filed v1.41 via /plan-eng-review) ### ✅ DONE (v1.44.0.0): Identity-based terminal-agent kill (replace pkill regex with PID) diff --git a/docs/ADDING_A_HOST.md b/docs/ADDING_A_HOST.md index 533c57489..d699816df 100644 --- a/docs/ADDING_A_HOST.md +++ b/docs/ADDING_A_HOST.md @@ -157,6 +157,7 @@ Key fields: | Field | Purpose | |-------|---------| +| `defaultModel` | Model overlay rendered when generation gets no explicit `--model` (validated against `ALL_MODEL_NAMES` in `scripts/models.ts`) | | `frontmatter.mode` | `allowlist` (keep only listed) or `denylist` (strip listed) | | `frontmatter.descriptionLimit` | Max chars, `null` for no limit | | `frontmatter.descriptionLimitBehavior` | `error` (fail build), `truncate`, `warn` |