From 71b2a0168b00fa6f0dfef0996dea6eab28120492 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Mon, 31 Aug 2026 06:10:04 +0000 Subject: [PATCH] docs: update project documentation for v1.77.0.0 Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 7 ++++--- CONTRIBUTING.md | 7 ++++--- docs/PROJECT_STRUCTURE.md | 3 ++- docs/TESTING_INTERNALS.md | 23 ++++++++++++++++++++++- 4 files changed, 32 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 529199ddd..46f167c28 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,6 +22,7 @@ bun run dev:skill # watch mode: auto-regen + validate on change bun run eval:list # list all eval runs from ~/.gstack/projects//evals/ bun run eval:compare # compare two eval runs (auto-picks most recent) bun run eval:summary # aggregate stats across all eval runs +bun run eval:flake-rank # rank tests by flake signal (retried passes first; --json, --dir, --since-days) bun run slop # full slop-scan report (all files) bun run slop:diff # slop findings in files changed on this branch only ``` @@ -647,9 +648,9 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session floor enforced against the live shard census by test/eval-detach-timeout-floor.test.ts) are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard - process count (default 4); `EVALS_CONCURRENCY` is bun's --max-concurrency - WITHIN a shard (default 4) — they are deliberately separate knobs. `eval:list` / `eval:compare` / - `eval:summary` read the shard dirs too. Or call + process count (default 8); `EVALS_CONCURRENCY` is bun's --max-concurrency + WITHIN a shard (default 2) — they are deliberately separate knobs. `eval:list` / `eval:compare` / + `eval:summary` / `eval:flake-rank` read the shard dirs too. Or call `gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] -- ` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never pass keys in argv). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f81afc0b5..843ee5771 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -236,6 +236,7 @@ When E2E tests run, they produce machine-readable artifacts in `~/.gstack-dev/`: bun run eval:list # list all eval runs (turns, duration, cost per run) bun run eval:compare # compare two runs — shows per-test deltas + Takeaway commentary bun run eval:summary # aggregate stats + per-test efficiency averages across runs +bun run eval:flake-rank # rank tests by flake signal: retried passes first, then failure rate (--json, --dir, --since-days) ``` **Detached runs for agents and long suites.** When an agent (or you, for a run @@ -264,9 +265,9 @@ distinguishes failed vs timed-out vs never-started shards. The runner also selects by diff: shards untouched by your branch are reported as skipped-by-diff, with a selection banner naming the reason (`EVALS_ALL=1` forces everything). `EVALS_JOBS` sets how many shard processes run at once -(default 4); `EVALS_CONCURRENCY` is bun's concurrency WITHIN a shard — they -are deliberately separate knobs. `eval:list`, -`eval:compare`, and `eval:summary` are shard-aware. Humans running +(default 8); `EVALS_CONCURRENCY` is bun's concurrency WITHIN a shard +(default 2) — they are deliberately separate knobs. `eval:list`, +`eval:compare`, `eval:summary`, and `eval:flake-rank` are shard-aware. Humans running `bun run test:evals` foreground in their own terminal don't need this — Ctrl-C is intended there. diff --git a/docs/PROJECT_STRUCTURE.md b/docs/PROJECT_STRUCTURE.md index 49954c165..6d043e7dc 100644 --- a/docs/PROJECT_STRUCTURE.md +++ b/docs/PROJECT_STRUCTURE.md @@ -28,6 +28,7 @@ gstack/ │ ├── skill-check.ts # Health dashboard │ ├── test-free-shards.ts # Strict parallel free-suite runner (GSTACK_FREE_JOBS, opt-in flaky retry) │ ├── test-paid-shards.ts # Sharded paid-tier runner (one Bun process per shard) +│ ├── eval-flake-rank.ts # Flake-telemetry dial: ranks tests by retried passes across eval runs + the free-lane ledger │ ├── sandbox-doctor.sh # One-command cloud-sandbox fixer: makes the free suite run green │ └── dev-skill.ts # Watch mode ├── test/ # Skill validation + eval tests @@ -71,7 +72,7 @@ gstack/ ├── patches/ # bun `patchedDependencies` patches (playwright-core windowsHide) ├── docs/designs/ # Design documents ├── setup-deploy/ # /setup-deploy skill (one-time deploy config) -├── .github/ # CI workflows + Docker image +├── .github/ # CI workflows + shared composite actions (.github/actions/) + Docker image (claude CLI pinned) │ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, and 8 more (windows, periodic evals, release gates, ci-image) │ └── docker/ # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium) ├── contrib/ # Contributor-only tools (never installed for users) diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index be803240b..87ee326c4 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -68,6 +68,12 @@ failure), never-started/timed-out taxonomy, and parent-computed diff selection propagated to children via `EVALS_SELECTION_JSON` (fail-open: a child that can't parse it recomputes locally with one warning). Retry parity lives in `RETRY_OVERRIDES` (literals; old matrix rows' earned `retries: 2`). +Flake telemetry rides the store: every recorded test carries its 1-based +`attempt` (a pass-on-attempt-2 stays visible forever — bun's own stream hides +it), runs list `flaky_retries`, the report warns on passed-only-on-retry +tests, and `bun run eval:flake-rank` ranks the series (retried passes first, +then failure rate; 60-day recency bound; the free lane's flake ledger is +folded in). **CI planner/executor/report.** `--emit-plan --slices K` computes selection + the slice plan ONCE (killing per-slice selector divergence); @@ -86,12 +92,27 @@ fail-fast verification loop). evals-periodic.yml runs ALL periodic-tier files weekly (the coverage contract) minus the reasoned exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking required per entry; removal re-activates the file), plus a weekly -`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. +`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. The CI +image pins the claude CLI to an exact version (`.github/docker/Dockerfile.ci`, +enforced by `test/ci-image-cli-pin.test.ts` — bumps ride PRs that run the PTY +gate), and every eval-store run records `claude --version`, resolved once in +the runner parent, so a TUI-drift flake hunt is a grep, not archaeology. **Timeout policy.** Paid tests use the tiers in `test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG); `test/eval-budgets-policy.test.ts` pins that every tier fits the shard wall minus overhead and ratchets raw literals. Budget above the wall is fiction. +Session timeouts are two-phase: a silent API dies at the startup grace (90s +local / 300s CI floor, distinct exit reason `timeout_startup`) and the work +budget arms on the first byte — the total wall never grows +(`test/session-runner-startup-grace.test.ts` pins the floor). A timed-out +session kills its whole detached process group (claude, codex, and gemini +runners alike — `test/session-runner-groupkill.test.ts`), so a stray +grandchild can't stretch a 600s budget past 1400s. And sync spawns can't +wedge a shard: every `spawnSync`/`execSync`/`execFileSync`/`Bun.spawnSync` +in the test trees must carry a `timeout`, enforced by +`test/spawnsync-timeout-tripwire.test.ts` with a shrink-only exemption +ratchet. ## Cloud sandboxes (Vercel / Conductor cloud workspaces)