mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-11 08:40:22 +02:00
v1.60.2.0 fix: free-suite drift on dev machines (eval-list cwd, gemini regex, observability floor) (#2470)
* fix(test): eval-list CLI spawns from neutral cwd so slug detection can't dodge the fixture store
getProjectEvalDir() probes cwd-relative .claude/skills/gstack/bin/gstack-slug;
with cwd=ROOT on a dev machine the self-symlink makes it succeed, routing reads
to an empty project-scoped dir instead of the seeded legacy ~/.gstack-dev/evals.
Neutral cwd + absolute script path fails both probes deterministically — same
behavior as CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): case-insensitive remediation-hint match for reworded Gemini NOT-READY message
The message now leads with 'Export GEMINI_API_KEY...' (free-tier OAuth
deprecation); the old pattern only knew lowercase 'export'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): observability check 11 floor 6 -> 5 after shell-free spawn removed promptFile unlink
aa3bd6f0 deleted the prompt temp file (and its /* non-fatal */ marker) when it
dropped shell interpolation. The invariant — every runner I/O path wrapped
non-fatally — still holds at the 5 remaining sites, now named in the comment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(todos): P1 — free-suite exit code masked by in-process force-exits
Five browse test files setTimeout(() => process.exit(0), 500) inside the shared
bun process; the suite can exit 0 before the summary with real failures masked.
Receipts + fix path filed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(release): v1.60.2.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
a3259400a3
commit
960c3a8d6c
@@ -45,6 +45,37 @@ a silent mistake breaks all 52 skills. High blast radius — needs its own focus
|
||||
|
||||
## Test infrastructure
|
||||
|
||||
### P1: Free suite exit code is untrustworthy — in-process force-exits mask failures
|
||||
|
||||
**Priority:** P1
|
||||
|
||||
**What:** At least five browse test files end with `setTimeout(() => process.exit(0), 500)`
|
||||
(browse/test/commands.test.ts:101, snapshot.test.ts:36, batch.test.ts:47,
|
||||
handoff.test.ts:31, content-security.test.ts:465). The timer fires inside the SHARED
|
||||
`bun test` process, exiting 0 before bun prints its final summary — so `bun test` can
|
||||
report exit 0 while real test failures scrolled by earlier. Remove the force-exits and
|
||||
fix the underlying handle leaks they paper over (lingering Playwright/daemon handles
|
||||
that once made the suite hang), or scope the exit to a spawned child process.
|
||||
|
||||
**Why:** Observed 2026-08-07: three genuinely failing tests (eval-list-cli,
|
||||
benchmark-cli, observability check 11) rode green `bun test` exit codes across
|
||||
multiple runs; the failures only surfaced by grepping logs for "(fail)" lines. A test
|
||||
suite that exits 0 on failure is worse than no suite — it manufactures false
|
||||
confidence at commit time and in any CI job that trusts the exit code.
|
||||
|
||||
**Pros:** Restores the one contract everything (CI, /ship, humans) relies on: exit
|
||||
code == truth. Also un-hides the missing final summary block.
|
||||
**Cons:** The force-exits exist because the suite once hung on leaked handles;
|
||||
removing them without fixing the leaks trades silent failure for hangs. Needs a
|
||||
focused pass: find each leaked handle (daemon children, PTY, Playwright contexts),
|
||||
close them in afterAll, then delete the exits one file at a time.
|
||||
|
||||
**Context / where to start:** `grep -rn "process.exit(0)" browse/test/` — the
|
||||
setTimeout variants are the offenders (server-no-import-side-effects.test.ts:62 is a
|
||||
spawned-child probe, fine). Repro: run the full free suite and note the log ends at
|
||||
the browse files with no "Ran N tests" summary. Receipts:
|
||||
~/.gstack-dev/logs/free-suite-main-check.log (3 masked fails, exit 0).
|
||||
|
||||
### P2: Periodic CI matrix covers 9 of ~66 e2e files — decide the coverage contract
|
||||
|
||||
**Priority:** P2
|
||||
|
||||
Reference in New Issue
Block a user