diff --git a/CHANGELOG.md b/CHANGELOG.md index 1baf18f1e..8d68924f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,68 @@ # Changelog +## [1.91.20.0] - 2026-10-04 + +**A failed eval now says what happened, with the evidence and the next command.** +**Bun is checked before anything installs, and make-pdf and /review work in Codex.** + +This release follows up on the severe fix wave. Its eight census runs had 21 red verdicts, and 27 of the 28 failed trials were labelled `assertion`, including timeouts and provider refusals, so the reports misdescribed their own failures. Now every failed trial records a machine-checked cause beside the unchanged pass/fail class, and each red line in the census report carries the expected and received values (or the judge's failing scores), the artifact holding the transcript, and a command that reruns only that case. For everyone using gstack: setup refuses a Bun too old to keep a project's `.env` away from gstack's tools, auto-update checks that before moving your checkout, and make-pdf and /review work inside Codex. + +### The numbers that matter + +Each row is a regression test on this release that fails on v1.91.19.0, or a measurement from the 2026-10-03/04 census artifacts named in the row. + +| Check | v1.91.19.0 | v1.91.20.0 | +|---|---|---| +| make-pdf blocks on Codex that use `$P` without finding it | 7 of 8 | 0 | +| /review with a read-only `.git` (Codex's sandbox) | prints nothing, "Nothing to review" | reviews against the local base, says `stale at ` | +| `./setup` with Bun 1.3.2 | installs; compiled tools read the project's `.env` | refused before writing anything | +| Census cost line when billing was not captured | `$0.00` | known sum plus "N of M trial(s) cost unknown" | +| plan-mode-no-op recorded wall (census 37151477069) | 346 s (its test times summed) | 176 s (the shard's wall) | +| Gate census slowest slice, replanned with measured walls (census 37198445662) | ~677 s | <=539 s planned | + +The first three rows are the ones users feel. The last three are why the next census reports are believable: costs, times and causes now mean what they say. + +### What changes for you + +- **Failure reports explain themselves.** A red census line keeps its error and adds `cause : ` (refusal, API error, provider stall, session or observer timeout, assertion), Expected/Received or `dimension mean < threshold (n samples) "rationale"`, `evidence: ` and `after a repair: `. Model-written text is sanitized before it reaches an issue or PR comment. +- **Bun floor and safe auto-update.** Setup refuses Bun older than 1.3.3 and warns below 1.4.0, the tested version. Team-mode auto-update and `/gstack-upgrade` read the incoming release's Bun requirement first and hold the update, leaving your install untouched, when your Bun is too old. The first update run by an older auto-updater cannot do this check. +- **make-pdf on Codex** and the other env-var hosts finds its binary in every block, and `MAKE_PDF_BIN` works there too. Install paths with spaces work. +- **/review in Codex's sandbox.** The working-tree fingerprint writes its git objects to a private temp directory, and a base that cannot be fetched is reported as stale instead of ending the review. +- **`browse stop` reaps a surviving headless Chromium** again on Playwright 1.62, only for browsers gstack launched. +- **/ship's documentation gate** records post-audit file hashes with its helper instead of having the model retype them. +- **/plan-ceo-review** shows its mode line (`Mode: ...` or `Auto-decided review mode → ...`) word for word right after the mode is chosen. +- **/qa** names where its report filename parts come from and labels timing as probe budget and guarded command time. **/deslop-shared-libs** names a concrete risk for every recommendation. +- **`GSTACK_DESIGN_IMAGE_MODEL`** switches the design tool's image model (default `gpt-image-2`); a value that is not a gpt-image model name is refused before any request. + +### Behavior changes you may notice + +- **Bun below 1.3.3 stops setup** with `gstack needs Bun 1.3.3 or newer`; nothing is installed. Fix: `bun upgrade`, then `./setup`. +- **Auto-update can say `update held (bun-too-old: ...)`** and `/gstack-upgrade` can stop with `BUN_TOO_OLD`; your current version keeps working. Fix: `bun upgrade`. +- **/review in a read-only checkout reports `Base coverage: stale at `** instead of stopping with "Nothing to review". +- **Browser Quick QA has a 3-minute budget** (was 30 seconds, too short for a homepage and five pages with their checkpoints). +- **/review's plan-completion question is not asked in spawned or non-interactive runs;** they report REQUIREMENTS MISSING and continue without recording anything. +- **`live-model-check.ts` ignores `GSTACK_DESIGN_MODEL`** and always checks the default models. +- For contributors: `bun run eval:pass-rates` rejects unknown flags and case ids (exit 2) and `--gate` also fails when a session ran above 85% of its armed budget; every `claude -p` eval session streams partial messages (only a liveness summary is kept) and writes `session-ledger.jsonl`; overlay comparisons count an overlay-off wrong answer as a measured result (contract v4); the AUQ matrix scores recommendation substance with a 3-sample judge panel. + +### Itemized changes + +#### Added +- `bun run eval:pass-rates --headroom` (slowest armed session per case against its budget), `--reds` (verdict reds by class and cause, with the all-green probability and its interval) and `--run ` (one census's reds, downloading only the slices they name); `--help` works offline. +- `docs/troubleshooting.md` sections for the Bun floor, held updates, stale /review bases, fingerprint temp-dir errors, the image model override, an unrecorded Chromium PID, and census report lines. + +#### Fixed +- `bun run scripts/test-paid-shards.ts --case ` selects cases registered in a loop and whole-file cases such as plan-mode-no-op; it used to select nothing. +- The plan-review floor test approves its own plan-file edit when the diff preview soft-wraps unchanged lines. +- The /plan-ceo-review approach-menu eval no longer asks for "verbatim ... exact prose" output, which tripped the provider's output-extraction refusal. +- The prosons neutral-posture check no longer reads "a coverage call, not a taste call" as the neutral dodge. + +#### For contributors +- Session ledger (`test/helpers/session-ledger.ts`), a pure failure-cause classifier beside `failureClassOf`, and one sanitizer for published text (`scripts/lib/published-text.ts`). Verdicts, `failure_class` and `error` are unchanged under EVAL_POLICY v1 (main and branch report code agree on all 15 stored census lanes). +- Replay corpora for five repeat-offender detectors in the free suite, and an audit of phrase-only detectors (`docs/evals/detector-inventory.md`). The CEO auto-decide detector credits the mode handoff only when the chat shows its line. +- Judge calibration controls (`docs/evals/judge-controls-2026-10.md`). The workflow judges passed bundles with a required step removed, so `test/workflow-required-steps.test.ts` now checks every step and phase heading of the judged workflow skills. +- Paid shard durations refreshed from census 37198445662; a JUnit case's recorded wall is capped by its shard's wall. +- The docsync fixture saves its observation interface to a file its children read instead of having the parent copy it into each child prompt. + ## [1.91.19.0] - 2026-10-03 **A check that did not run now says so, and memory stops losing transcripts.** diff --git a/VERSION b/VERSION index c68c06199..1701e694d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.19.0 +1.91.20.0 diff --git a/package.json b/package.json index 9ab478887..42d8ecc5d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.19", + "version": "1.91.20", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module",