chore: bump version and changelog (v1.66.0.0)

Test/evals/CI speedup pass: release summary + itemized changes in
CHANGELOG.md; TODOS.md marks the free-suite exit-code P1 complete and
files the review-army follow-ups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-15 17:03:57 -07:00
co-authored by Claude Fable 5
parent 3dc2cde346
commit 7726d25396
4 changed files with 216 additions and 33 deletions
+63
View File
@@ -1,5 +1,68 @@
# Changelog
## [1.66.0.0] - 2026-08-15
**The full ~7,000-test suite in about 90 seconds, verified honest.**
**Paid evals now bill by diff, not $38 flat.**
`bun run test` used to take 454 seconds. It now runs as six concurrent shard processes and finishes in about 90 to 100 seconds, under a strict output contract: a shard that exits without bun's own terminal summary line is a failure, a wedged shard is killed at a size-scaled deadline and named in the epilogue, and the console shows only what you need (per-shard status, then `✗ file — test name` for anything red, full stream in a per-run log, `--verbose` for the firehose). Twelve test files that ran under no script and no CI are wired in. A 3,372-line dead eval monolith is deleted, with four never-run tests revived out of it.
Paid evals select by diff. Edit one skill and the runner executes only the shards your change touches, reports the rest as skipped-by-diff, and prints the reason. Selection sees uncommitted and untracked work, fails closed with a named cause on git errors, and an edit to the selection data itself re-runs only the changed keys instead of forcing the full suite.
### The numbers that matter
Measured on this branch. Re-run with `time bun run test` and `bun run eval:select`; eval receipts live in `~/.gstack-dev/evals/`.
| Metric | Before | After | Δ |
|---|---|---|---|
| Free suite wall clock (~7,000 tests) | 454s | ~90-100s, strict-verified | ~4.7x |
| Free-test files with Linux CI coverage | 0 | ~420, as a required PR check | new |
| Paid cost of a one-skill edit | ~$38 (full suite) | 4 of 45 shards, $0.67 | ~57x |
| Slowest CI eval job | 741s, one serial file | three jobs, each under ~250s | ~3x |
| Paid retry amplification | `--retry 2`, +84% measured | `--retry 1` | half |
The $0.67 row is a live rehearsal, not a projection: a scratch edit to `qa/SKILL.md.tmpl` selected 17 of 177 tests, ran 4 of 45 shards, skipped 41 by diff, and the /qa E2E passed.
### What this means for contributors
Runs you used to schedule around now fit inside a thought. `bun run test` before every commit is a real habit again at ~90 seconds, red names the exact test, and green means every file actually ran. Fork PRs get true test signal from the new secretless Linux lane. Ship a change and the eval bill tracks your blast radius.
### Itemized changes
### Added
- Linux free-tests CI lane (`.github/workflows/free-tests.yml`): the whole free suite on every PR and every push to main, required from day one, zero secrets, least-privilege token, failure logs uploaded as an artifact, wiring pinned by `test/free-tests-workflow-wiring.test.ts`.
- Diff-based paid-shard selection: parent-side skipping with a `skipped-by-diff` taxonomy and a selection banner naming the reason (`scripts/test-paid-shards.ts`).
- Map-diff selection for the selection data itself: editing `test/helpers/touchfiles-data.ts` re-runs only added/changed/retiered keys (old version evaluated via `git show` + a bun child; adversarial fixtures in `test/touchfiles-map-diff.test.ts`).
- Selection unions committed, staged/unstaged, and untracked changes; git failures throw naming `EVALS_ALL=1` (fail closed), and non-ASCII filenames select correctly (`core.quotePath=false`).
- `test/helpers/skill-fixture.ts`: E2E fixtures extract the SKILL.md sections a test needs instead of copying 1,800-line files — nine fixture sites cut 58-97%.
- `GSTACK_EVAL_MODEL_JUDGE` env override for the LLM-judge model; eval model resolution centralized in `lib/eval-model.ts` with per-kind `GSTACK_EVAL_MODEL_<KIND>` overrides.
### Changed
- Free suite architecture: N concurrent shard processes (serial within each); tree-mutating tests and tree-measuring ratchet readers run in one serial shard after the parallel phase, so measurements never race regeneration. Shard curation lists are pinned against the live file census, and wall deadlines scale with shard size.
- Agent SDK capture default Opus → Sonnet (D1a). The judge default stays Sonnet: a live A/B on the health rubric scored Haiku 2/2/2 against Sonnet's 4/3/4, so the downgrade was pinned back per D1a's regressor clause (receipts in `test/helpers/llm-judge.ts`).
- Four expensive posture tests demoted gate → periodic (D2a).
- Paid runners: `EVALS_JOBS` (shard process count) split from `EVALS_CONCURRENCY` (within-shard), `--retry 1` on every paid path, one preflight API ping per run instead of ~30, detach timeouts floor-enforced against the live shard census by `test/eval-detach-timeout-floor.test.ts`.
- CI: eval Docker image cache keyed on Dockerfile + bun.lock so version bumps stop rebuilding it; Bun 1.3.13 in the image; `skill-e2e-review` split into three matrix shards; actionlint runs a digest-pinned prebuilt image; five single-core jobs right-sized; lint and skill-docs stop double-running every PR commit; the Windows lane caches bun installs and runs the curated suite instead of a hand list.
- Skill-routing E2E fixture installs skill heads, not ~18 full SKILL.md files.
### Fixed
- Ctrl-C actually cancels a run: the signal forwarders now schedule the parent runner's own exit and both shard pools stop launching new work on `SIGINT`/`SIGTERM` — previously the parent killed the current child and kept spawning API-burning shards.
- The intermittent whole-suite wedge: `browse/src/browser-manager.ts` `close()` captures the Chromium child before the close race and SIGKILLs it when graceful close times out, with unit coverage of the fallback.
- The strict-output classifier keeps stdout and stderr line assembly separate, so interleaved pipe chunks cannot hide a failure line or fake a truncation. Windows shard kills take the whole process tree (`taskkill /T`) instead of orphaning grandchildren.
- Redaction calibration: `${var}` template interpolations and ALL-CAPS `USER:PASSWORD` doc placeholders no longer block pushes, while a bare `$word` password and a literal lowercase `password`/`pass` at the URL-password position still do; the two connection-string validators share one helper so they cannot drift.
- Supabase pooler DSNs percent-encode the password segment, `wait --timeout` rejects non-numeric values instead of polling forever, response-body read failures retry as transport errors, and the CLI entrypoint lets stdout drain before exiting.
- The paid-suite preflight fails fast on a missing `claude` binary, a spawn error, or a timeout — outages surface once in the parent instead of once per shard.
- Same-name branches from different forks can no longer cancel each other's CI runs (concurrency groups key on PR number across the free, eval, and Windows lanes).
- Selection integrity: the `touchfiles.ts` facade, `e2e-helpers.ts`, and `paid-test-set.ts` are global touchfiles (an edit to selection-path code can never select zero tests); duplicate touchfiles keys fail the suite; rehomed E2E files list themselves in their own dependency maps; retro E2E passes require the report on disk.
- The intermittent context-save-list eval test that had never passed in 26 recorded runs now passes.
- `variants-retry-after` HTTP-date flake; watchdog E2E 22.7s → 1.5s; supabase-provision tests 16.5s → 0.45s via an in-process TS port.
- `package.json` version drift against VERSION.
### For contributors
- `test:gate:sharded` / `test:periodic:sharded` run tiers through the sharded paid runner; `eval:bg:*` wrap runs in `gstack-detach` with a per-tier watchdog and the machine-wide `gstack-evals` lock.
- Five pre-existing environment failures quarantined individually with in-file receipts; two dead-architecture security contract tests deleted.
- `test/e2e-tier-alignment.test.ts` enforces tier declarations and fails fatally when a sharded-runner mapper cannot see a gate file.
## [1.65.0.0] - 2026-08-14
**/autoplan, /codex on macOS, and memory ingest work again.**