mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-11 08:40:22 +02:00
v1.60.2.0 fix: free-suite drift on dev machines (eval-list cwd, gemini regex, observability floor) (#2470)
* fix(test): eval-list CLI spawns from neutral cwd so slug detection can't dodge the fixture store
getProjectEvalDir() probes cwd-relative .claude/skills/gstack/bin/gstack-slug;
with cwd=ROOT on a dev machine the self-symlink makes it succeed, routing reads
to an empty project-scoped dir instead of the seeded legacy ~/.gstack-dev/evals.
Neutral cwd + absolute script path fails both probes deterministically — same
behavior as CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): case-insensitive remediation-hint match for reworded Gemini NOT-READY message
The message now leads with 'Export GEMINI_API_KEY...' (free-tier OAuth
deprecation); the old pattern only knew lowercase 'export'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): observability check 11 floor 6 -> 5 after shell-free spawn removed promptFile unlink
aa3bd6f0 deleted the prompt temp file (and its /* non-fatal */ marker) when it
dropped shell interpolation. The invariant — every runner I/O path wrapped
non-fatally — still holds at the 5 remaining sites, now named in the comment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(todos): P1 — free-suite exit code masked by in-process force-exits
Five browse test files setTimeout(() => process.exit(0), 500) inside the shared
bun process; the suite can exit 0 before the summary with real failures masked.
Receipts + fix path filed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(release): v1.60.2.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
a3259400a3
commit
960c3a8d6c
@@ -1,5 +1,40 @@
|
||||
# Changelog
|
||||
|
||||
## [1.60.2.0] - 2026-08-07
|
||||
|
||||
## **Three free-suite tests fail-proofed against machine drift.**
|
||||
## **Plus a filed P1: the suite's exit code can lie, and now we know why.**
|
||||
|
||||
A full-suite health check turned up three tests that failed on dev machines while CI stayed green, all test-side drift rather than product bugs. The eval:list CLI test now spawns from a neutral directory, so slug detection cannot route reads away from the fixture store it seeds (the old cwd made it fail on any machine with the dev symlink). The benchmark CLI's remediation-hint check is case-insensitive, matching the reworded Gemini guidance ("Export GEMINI_API_KEY..."). The session-runner observability floor now expects the 5 wrapped I/O sites that actually exist since the shell-free spawn removed the prompt-file unlink.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Source: this branch's investigation logs (~/.gstack-dev/logs/free-suite-*.log) and per-file reruns.
|
||||
|
||||
| Check | Before | After |
|
||||
|-------|--------|-------|
|
||||
| eval-list-cli on dev machines | 1 fail (reads empty project dir) | 2/2 pass, deterministic everywhere |
|
||||
| benchmark-cli remediation hint | 1 fail (case-brittle regex) | 15/15 pass |
|
||||
| observability check 11 floor | expects >= 6 markers, counts 5 | floor matches the 5 real sites |
|
||||
|
||||
One deeper finding got filed instead of rushed: at least five browse test files force-exit the shared bun process with `setTimeout(() => process.exit(0), 500)`, which can exit 0 before the summary prints and mask real failures. That is now a P1 in TODOS.md with receipts, because removing the exits without fixing the handle leaks they paper over would trade silent failure for hangs.
|
||||
|
||||
### What this means for you
|
||||
|
||||
`bun test` gives the same verdict on your laptop as in CI for these three tests, and the exit-code trust problem is documented with a concrete fix path instead of lurking.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Fixed
|
||||
|
||||
- `test/eval-list-cli.test.ts`: spawn from neutral cwd + absolute script path so `getProjectEvalDir()` slug probes fail deterministically and the seeded legacy store is read.
|
||||
- `test/benchmark-cli.test.ts`: remediation-hint pattern made case-insensitive for the updated Gemini NOT-READY message.
|
||||
- `test/helpers/observability.test.ts`: check 11 floor 6 → 5 with the surviving wrapped-I/O sites named.
|
||||
|
||||
#### For contributors
|
||||
|
||||
- TODOS.md: new P1 (free-suite exit code masked by in-process force-exits, with repro + receipts) filed under Test infrastructure.
|
||||
|
||||
## [1.60.1.0] - 2026-07-09
|
||||
|
||||
## **The /autoplan dual-voice eval is back on the board, catching real regressions.**
|
||||
|
||||
Reference in New Issue
Block a user