docs: update project documentation for v1.77.0.0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-31 06:10:04 +00:00
co-authored by Claude Fable 5
parent 7cb0f2d93a
commit 71b2a0168b
4 changed files with 32 additions and 8 deletions
+4 -3
View File
@@ -22,6 +22,7 @@ bun run dev:skill # watch mode: auto-regen + validate on change
bun run eval:list # list all eval runs from ~/.gstack/projects/<slug>/evals/ bun run eval:list # list all eval runs from ~/.gstack/projects/<slug>/evals/
bun run eval:compare # compare two eval runs (auto-picks most recent) bun run eval:compare # compare two eval runs (auto-picks most recent)
bun run eval:summary # aggregate stats across all eval runs bun run eval:summary # aggregate stats across all eval runs
bun run eval:flake-rank # rank tests by flake signal (retried passes first; --json, --dir, --since-days)
bun run slop # full slop-scan report (all files) bun run slop # full slop-scan report (all files)
bun run slop:diff # slop findings in files changed on this branch only bun run slop:diff # slop findings in files changed on this branch only
``` ```
@@ -647,9 +648,9 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
floor enforced against the live shard census by floor enforced against the live shard census by
test/eval-detach-timeout-floor.test.ts) test/eval-detach-timeout-floor.test.ts)
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
process count (default 4); `EVALS_CONCURRENCY` is bun's --max-concurrency process count (default 8); `EVALS_CONCURRENCY` is bun's --max-concurrency
WITHIN a shard (default 4) — they are deliberately separate knobs. `eval:list` / `eval:compare` / WITHIN a shard (default 2) — they are deliberately separate knobs. `eval:list` / `eval:compare` /
`eval:summary` read the shard dirs too. Or call `eval:summary` / `eval:flake-rank` read the shard dirs too. Or call
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] -- `gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] --
<cmd>` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never <cmd>` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never
pass keys in argv). pass keys in argv).
+4 -3
View File
@@ -236,6 +236,7 @@ When E2E tests run, they produce machine-readable artifacts in `~/.gstack-dev/`:
bun run eval:list # list all eval runs (turns, duration, cost per run) bun run eval:list # list all eval runs (turns, duration, cost per run)
bun run eval:compare # compare two runs — shows per-test deltas + Takeaway commentary bun run eval:compare # compare two runs — shows per-test deltas + Takeaway commentary
bun run eval:summary # aggregate stats + per-test efficiency averages across runs bun run eval:summary # aggregate stats + per-test efficiency averages across runs
bun run eval:flake-rank # rank tests by flake signal: retried passes first, then failure rate (--json, --dir, --since-days)
``` ```
**Detached runs for agents and long suites.** When an agent (or you, for a run **Detached runs for agents and long suites.** When an agent (or you, for a run
@@ -264,9 +265,9 @@ distinguishes failed vs timed-out vs never-started shards. The runner also
selects by diff: shards untouched by your branch are reported as selects by diff: shards untouched by your branch are reported as
skipped-by-diff, with a selection banner naming the reason (`EVALS_ALL=1` skipped-by-diff, with a selection banner naming the reason (`EVALS_ALL=1`
forces everything). `EVALS_JOBS` sets how many shard processes run at once forces everything). `EVALS_JOBS` sets how many shard processes run at once
(default 4); `EVALS_CONCURRENCY` is bun's concurrency WITHIN a shard — they (default 8); `EVALS_CONCURRENCY` is bun's concurrency WITHIN a shard
are deliberately separate knobs. `eval:list`, (default 2) — they are deliberately separate knobs. `eval:list`,
`eval:compare`, and `eval:summary` are shard-aware. Humans running `eval:compare`, `eval:summary`, and `eval:flake-rank` are shard-aware. Humans running
`bun run test:evals` foreground in their own terminal don't need this — Ctrl-C `bun run test:evals` foreground in their own terminal don't need this — Ctrl-C
is intended there. is intended there.
+2 -1
View File
@@ -28,6 +28,7 @@ gstack/
│ ├── skill-check.ts # Health dashboard │ ├── skill-check.ts # Health dashboard
│ ├── test-free-shards.ts # Strict parallel free-suite runner (GSTACK_FREE_JOBS, opt-in flaky retry) │ ├── test-free-shards.ts # Strict parallel free-suite runner (GSTACK_FREE_JOBS, opt-in flaky retry)
│ ├── test-paid-shards.ts # Sharded paid-tier runner (one Bun process per shard) │ ├── test-paid-shards.ts # Sharded paid-tier runner (one Bun process per shard)
│ ├── eval-flake-rank.ts # Flake-telemetry dial: ranks tests by retried passes across eval runs + the free-lane ledger
│ ├── sandbox-doctor.sh # One-command cloud-sandbox fixer: makes the free suite run green │ ├── sandbox-doctor.sh # One-command cloud-sandbox fixer: makes the free suite run green
│ └── dev-skill.ts # Watch mode │ └── dev-skill.ts # Watch mode
├── test/ # Skill validation + eval tests ├── test/ # Skill validation + eval tests
@@ -71,7 +72,7 @@ gstack/
├── patches/ # bun `patchedDependencies` patches (playwright-core windowsHide) ├── patches/ # bun `patchedDependencies` patches (playwright-core windowsHide)
├── docs/designs/ # Design documents ├── docs/designs/ # Design documents
├── setup-deploy/ # /setup-deploy skill (one-time deploy config) ├── setup-deploy/ # /setup-deploy skill (one-time deploy config)
├── .github/ # CI workflows + Docker image ├── .github/ # CI workflows + shared composite actions (.github/actions/) + Docker image (claude CLI pinned)
│ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, and 8 more (windows, periodic evals, release gates, ci-image) │ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, and 8 more (windows, periodic evals, release gates, ci-image)
│ └── docker/ # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium) │ └── docker/ # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium)
├── contrib/ # Contributor-only tools (never installed for users) ├── contrib/ # Contributor-only tools (never installed for users)
+22 -1
View File
@@ -68,6 +68,12 @@ failure), never-started/timed-out taxonomy, and parent-computed diff
selection propagated to children via `EVALS_SELECTION_JSON` (fail-open: a selection propagated to children via `EVALS_SELECTION_JSON` (fail-open: a
child that can't parse it recomputes locally with one warning). Retry parity child that can't parse it recomputes locally with one warning). Retry parity
lives in `RETRY_OVERRIDES` (literals; old matrix rows' earned `retries: 2`). lives in `RETRY_OVERRIDES` (literals; old matrix rows' earned `retries: 2`).
Flake telemetry rides the store: every recorded test carries its 1-based
`attempt` (a pass-on-attempt-2 stays visible forever — bun's own stream hides
it), runs list `flaky_retries`, the report warns on passed-only-on-retry
tests, and `bun run eval:flake-rank` ranks the series (retried passes first,
then failure rate; 60-day recency bound; the free lane's flake ledger is
folded in).
**CI planner/executor/report.** `--emit-plan <path> --slices K` computes **CI planner/executor/report.** `--emit-plan <path> --slices K` computes
selection + the slice plan ONCE (killing per-slice selector divergence); selection + the slice plan ONCE (killing per-slice selector divergence);
@@ -86,12 +92,27 @@ fail-fast verification loop). evals-periodic.yml runs ALL
periodic-tier files weekly (the coverage contract) minus the reasoned periodic-tier files weekly (the coverage contract) minus the reasoned
exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking
required per entry; removal re-activates the file), plus a weekly required per entry; removal re-activates the file), plus a weekly
`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. `EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. The CI
image pins the claude CLI to an exact version (`.github/docker/Dockerfile.ci`,
enforced by `test/ci-image-cli-pin.test.ts` — bumps ride PRs that run the PTY
gate), and every eval-store run records `claude --version`, resolved once in
the runner parent, so a TUI-drift flake hunt is a grep, not archaeology.
**Timeout policy.** Paid tests use the tiers in **Timeout policy.** Paid tests use the tiers in
`test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG); `test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG);
`test/eval-budgets-policy.test.ts` pins that every tier fits the shard wall `test/eval-budgets-policy.test.ts` pins that every tier fits the shard wall
minus overhead and ratchets raw literals. Budget above the wall is fiction. minus overhead and ratchets raw literals. Budget above the wall is fiction.
Session timeouts are two-phase: a silent API dies at the startup grace (90s
local / 300s CI floor, distinct exit reason `timeout_startup`) and the work
budget arms on the first byte — the total wall never grows
(`test/session-runner-startup-grace.test.ts` pins the floor). A timed-out
session kills its whole detached process group (claude, codex, and gemini
runners alike — `test/session-runner-groupkill.test.ts`), so a stray
grandchild can't stretch a 600s budget past 1400s. And sync spawns can't
wedge a shard: every `spawnSync`/`execSync`/`execFileSync`/`Bun.spawnSync`
in the test trees must carry a `timeout`, enforced by
`test/spawnsync-timeout-tripwire.test.ts` with a shrink-only exemption
ratchet.
## Cloud sandboxes (Vercel / Conductor cloud workspaces) ## Cloud sandboxes (Vercel / Conductor cloud workspaces)