From 904ddf91f334f68ce0bdace41b70b7286f9bcd1b Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Mon, 31 Aug 2026 06:25:06 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20cross-model=20doc-review=20fixes=20?= =?UTF-8?q?=E2=80=94=20flake-ledger=20env=20knobs,=20CI=20retry-on=20note,?= =?UTF-8?q?=20stale=20version=20comment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .github/workflows/ci-image.yml | 2 +- ARCHITECTURE.md | 2 +- CONTRIBUTING.md | 4 +++- docs/TESTING_INTERNALS.md | 19 +++++++++++++++---- 4 files changed, 20 insertions(+), 7 deletions(-) diff --git a/.github/workflows/ci-image.yml b/.github/workflows/ci-image.yml index 17d03b5f6..8776fd7a3 100644 --- a/.github/workflows/ci-image.yml +++ b/.github/workflows/ci-image.yml @@ -3,7 +3,7 @@ on: # Weekly self-heal (Monday 4am UTC) — deliberately 2h BEFORE # evals-periodic's 6am cron so the weekly eval run finds the image instead # of racing a half-pushed tag. With the claude CLI pinned in Dockerfile.ci - # (v1.76+), this cron no longer pulls CLI updates: when the content-hash + # (v1.77+), this cron no longer pulls CLI updates: when the content-hash # tag already exists it's a ~30s no-op, and it only rebuilds if the tag # was somehow lost. CLI bumps happen by editing the Dockerfile pin in a PR # that runs the PTY gate against the new TUI. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ddadc72bc..3c52efce8 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -424,7 +424,7 @@ The `EvalCollector` accumulates test results and writes them in two ways: 1. **Incremental:** `savePartial()` writes `_partial-e2e.json` after each test (atomic: write `.tmp`, `fs.renameSync`). Survives kills. 2. **Final:** `finalize()` writes a timestamped eval file (e.g. `e2e-20260314-143022.json`). The partial file is never cleaned up — it persists alongside the final file for observability. -`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack/projects//evals/` (legacy fallback `~/.gstack-dev/evals/`). Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `/shards//` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards//` subdirectories. Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline. +`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack/projects//evals/` (legacy fallback `~/.gstack-dev/evals/`). Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `/shards//` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards//` subdirectories (`eval:flake-rank` reads the same tree recursively, plus the free-suite flake ledger). Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline. ### Test tiers diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 843ee5771..1f398de16 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -168,7 +168,9 @@ silent truncation can never report green. Pass `--verbose` to forward the full child stream; `--wall-timeout ` overrides the per-shard kill deadline. `GSTACK_FREE_JOBS=` overrides the shard count (digits only, loud on garbage), and `GSTACK_FREE_RETRY_FLAKY=1` opts into one serial retry pass for -syscall-supervised sandboxes (off by default — dev boxes should see flakes). +syscall-supervised sandboxes (off by default locally — dev boxes should see +flakes; the required CI free lane turns it on and uploads every flaky pass +in a JSONL ledger artifact that `bun run eval:flake-rank` folds in). Working in a cloud sandbox? Run `scripts/sandbox-doctor.sh` once per boot to make the suite run green (details in [docs/TESTING_INTERNALS.md](docs/TESTING_INTERNALS.md)). diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 87ee326c4..dbab9d8b4 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -72,8 +72,14 @@ Flake telemetry rides the store: every recorded test carries its 1-based `attempt` (a pass-on-attempt-2 stays visible forever — bun's own stream hides it), runs list `flaky_retries`, the report warns on passed-only-on-retry tests, and `bun run eval:flake-rank` ranks the series (retried passes first, -then failure rate; 60-day recency bound; the free lane's flake ledger is -folded in). +then failure rate; 60-day recency bound on eval files; the free lane's flake +ledger is folded in from `flakeLedgerPath()` — override with +`GSTACK_FLAKE_LEDGER`, the same env var the CI free lane sets before +uploading the ledger as the `flake-ledger` artifact). Census integrity is +enforced from the free suite: every `E2E_TOUCHFILES` / `LLM_JUDGE_TOUCHFILES` +key must name a living paid test (`test/touchfiles.test.ts`'s reverse +invariant), and `git show :path` fixtures are banned — vendor the bytes +instead (`test/git-ref-fixture-tripwire.test.ts`). **CI planner/executor/report.** `--emit-plan --slices K` computes selection + the slice plan ONCE (killing per-slice selector divergence); @@ -96,7 +102,9 @@ required per entry; removal re-activates the file), plus a weekly image pins the claude CLI to an exact version (`.github/docker/Dockerfile.ci`, enforced by `test/ci-image-cli-pin.test.ts` — bumps ride PRs that run the PTY gate), and every eval-store run records `claude --version`, resolved once in -the runner parent, so a TUI-drift flake hunt is a grep, not archaeology. +the runner parent and handed to shard children as `GSTACK_CLAUDE_CLI_VERSION` +(never spawned on a test thread), so a TUI-drift flake hunt is a grep, not +archaeology. **Timeout policy.** Paid tests use the tiers in `test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG); @@ -134,4 +142,7 @@ Two runner knobs exist for these environments (both no-ops unless set): serial mega-shard and 6-way sharding both saturate the per-process syscall supervisor), and `GSTACK_FREE_RETRY_FLAKY=1` re-runs attributed failures once serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so -a broken tree can't masquerade as flaky). +a broken tree can't masquerade as flaky). The required CI free lane sets the +retry knob too, appending every flaky pass to the JSONL ledger it uploads +(`GSTACK_FLAKE_LEDGER`) — a flaky pass never reds the lane, but it never +disappears either.