From e76f65a8da31ec14776a965c608222c1aecad656 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Mon, 31 Aug 2026 08:55:30 -0700 Subject: [PATCH] =?UTF-8?q?v1.77.0.0=20feat:=20test-infrastructure=20overh?= =?UTF-8?q?aul=20wave=201=20=E2=80=94=20matrix=20deletion,=20flake=20telem?= =?UTF-8?q?etry,=20sync-spawn=20wedge=20class=20extinct=20(#2746)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix: pin the claude CLI to an exact version in the CI image + tripwire The image installed @anthropic-ai/claude-code UNPINNED and rebuilt weekly 'to pick up CLI updates' — while bun sat carefully pinned at 1.3.13 two RUN lines above. The PTY harness screen-scrapes this CLI's TUI, and that drift broke it three separate times (welcome-screen wedge on 2.1.233, skillify HOME discovery on 2.1.237, guard/freeze hooks on 2.1.162), each debugged as a flake first. Pin 2.1.251 (current latest), bump deliberately via a PR that runs the PTY gate, and enforce with test/ci-image-cli-pin.test.ts: any global npm install in Dockerfile.ci without an exact @X.Y.Z pin fails the free suite. The weekly ci-image cron stays as a cheap tag self-heal. Co-Authored-By: Claude Fable 5 * feat: stamp the claude CLI version into every eval-store run record Three harness breakages were traced to claude-CLI TUI drift only after long flake hunts, because no run record said which CLI it actually exercised. EvalCollector now stamps claude_cli_version (claude --version, cached once per process, 'unknown' when the binary is absent) into both partial and finalized records — schema-additive optional field, no SCHEMA_VERSION bump. Correlating a flake wave with a CLI release becomes a grep over ~/.gstack/projects//evals/ instead of archaeology. Co-Authored-By: Claude Fable 5 * fix: give the spinning-shard kill test load headroom (30s -> 90s) The test spawns and group-kills three real children (one a busy-loop burning a full core) while five sibling shard processes compete for eight vCPUs. Under full-suite load it blew bun's default 30s per-test ceiling at 30,009ms — while passing in isolation in 1.4s — and red the only required lane. Every assertion in it is event-based (statuses, group-kill proof, heartbeat lines); the sole latency claim is the <30s kill-deadline sanity bound, which stays. Explicit 90s headroom, not a weakened oracle. Co-Authored-By: Claude Fable 5 * feat: green-by-skip census — skip counts in the classifier, all-skipped labeling in the paid runner bun's 'Ran N tests' line COUNTS skipped tests, so a codex/gemini shard whose every test self-skipped (binary absent on the runner — true of every CI runner today) exits 0, dodges the hollow-shard guard, and reads as coverage in the weekly census. The classifier now parses bun's ' N skip' / ' N pass' recap lines; ShardOutcome carries skippedTests; formatSummary and the fail-closed slices report label an all-skipped pass explicitly: 'all N tests SKIPPED — verified nothing'. Status stays 'passed' (external service availability is host state, not a repo regression) but the census can no longer mistake absence for coverage. Co-Authored-By: Claude Fable 5 * refactor: extract composite actions for eval-lane setup; surviving lanes gain the fail-fast registry verification 'Fix bun temp' x3, 'Restore deps' x5, 'Seed claude interactive config' x3, and 'Register gstack skills' x3 were byte-near-identical copies across the legacy matrix, the sliced lane, and the periodic lane — and only the MATRIX copy of register-skills carried the 19-line dangling-symlink + frontmatter fail-fast loop written after a silent 'Unknown command' + 35-min-timeout incident. Extract all four into .github/actions/ composites; the register composite carries the verification loop (generalized over the skill list), so the sliced and periodic lanes — the lanes that SURVIVE the matrix deletion — now inherit the check they had silently dropped. Matrix-job inline copies are left untouched: that job is deleted next. Co-Authored-By: Claude Fable 5 * feat: delete the legacy 17-row eval matrix — the sliced lane is the only paid lane Every PR paid twice: the hand-enumerated matrix (18 test files, 22.6 min, ~$21 API measured on run 33263204465) ran serialized AHEAD of the strictly superior sliced lane via 'needs: evals' — 35.5 min wall and ~2x paid spend for the same diff. 14 of 17 rows carried no tier:, so periodic Opus benchmarks leaked into every PR (the e2e-plan row alone: 12/12 tests, 21.7 min, $7.28 — the wall-clock bound of ALL of CI). Parity receipt (static, pre-deletion): the sliced lane's gate census (49 files, derived from the runner itself) strictly contains all 18 matrix test files, plus 31 files the matrix never ran. Pure deletion — one revert restores it. The PR comment moved into slices-report (same '## E2E Evals' upsert marker, now sourced from slice artifacts + carrying the fail-closed reconciliation verdict). plan-slices loses the needs edge; the dead workflow-level EVALS_TIER env goes with it. test/evals-workflow-matrix.test.ts (and its KNOWN_MATRIX_GAPS / KNOWN_TIER_UNSET burn-down ratchets — retired: the sliced census makes 'every gate file runs' true by construction) is rewritten as test/evals-workflow-wiring.test.ts: matrix stays deleted, planner/executor/ report tier + slice-count agreement, both surviving lanes on the shared register-skills composite with its fail-fast verification loop, PR comment survival. Expected: PR eval wall 35.5 -> ~13 min, per-PR paid spend ~halved. Co-Authored-By: Claude Fable 5 * fix: provider-runner timeouts kill the whole process GROUP; codex/gemini inherit the orphan-drain hardening All three provider runners (claude/codex/gemini) killed only the direct child on timeout: tool subprocesses the CLI spawned survived as orphans holding our pipes open and burning shared API rate (observed: a 600s timeout stretching past 1400s; a stalled run once burned a core for 15 hours). gstack-detach's watchdog had the same shape one level up — killpg SIGTERM, 5s grace, then a direct-child proc.kill() that orphaned grandchildren. Fix: spawn provider children via node:child_process with detached (own process group) and killProcessGroup(SIGKILL) in the timeout handler — runShardChild's proven pattern, EPERM/ESRCH fallbacks included. The codex and gemini copies also gain the reader.cancel() + stderr Promise.race hardening only the claude copy had (they still carried the blocked-drain hang it fixed). gstack-detach's watchdog now group-SIGKILLs after the grace. Regression net: test/session-runner-groupkill.test.ts drives the REAL runSkillTest against a fake claude shim (PATH override) that spawns a grandchild and wedges — the run must classify timeout within budget and leave neither shim nor grandchild alive — plus source pins on all three runners (detached + killProcessGroup, no bare timeout kill, no Bun.spawn reversion). Co-Authored-By: Claude Fable 5 * fix: skill-e2e-opus-47 renders SKILL.md fixtures into a mkdtemp — never the live tree mkEvalRoot ran gen-skill-docs with cwd=ROOT, regenerating every in-repo SKILL.md mid-run while concurrent paid shards copyFileSync those same files in their beforeAll (EVALS_JOBS>=4 locally, 2 per CI slice) — a sibling could capture a half-regenerated or opus-rendered SKILL.md, and a timeout before afterAll stranded the whole tree at the wrong model for every later shard. A cross-shard race that could flake ANY concurrent paid test. Render via the --out-dir flag gen-skill-docs grew for exactly this reason (mirrors the repo layout, which is all the fixture reads), read the skill heads from the render dir, delete it, and drop the afterAll restore-regen entirely. Co-Authored-By: Claude Fable 5 * fix: claude CLI version resolves in the runner parent, never on a test thread Eng-review finding: getClaudeCliVersion's fallback is a SYNCHRONOUS spawnSync on the same thread that polls concurrent PTY/session tests — the judgePtyState blocking class this overhaul kills elsewhere. The paid runner parent now resolves it once (cached) and stamps GSTACK_CLAUDE_CLI_VERSION into every shard's env; eval-store short-circuits on the env var, and the fallback spawn's budget tightens 10s -> 3s (bounded one-time stall, records 'unknown' on a slow CLI). Co-Authored-By: Claude Fable 5 * test: wire skippedTests end-to-end through runPaidShard The census unit tests hand-built outcomes and the classifier tests parsed strings; nothing proved a real child's ' N skip' recap flows into outcome.skippedTests and the formatSummary label. A commandFor fake now prints the recap shape and the test asserts the parsed counts, the all-skipped predicate, and the 'verified nothing' label. Co-Authored-By: Claude Fable 5 * fix: make the setup composites rerun-safe (codex diff-review hardenings) restore-deps: 'cp -r SRC node_modules' with an existing node_modules NESTS the copy and leaves stale deps active — rm first. register-gstack-skills: 'ln -snf' hard-errors under set -eu when a REAL directory occupies the gstack slot — clear a non-symlink leftover first. CI workspaces are fresh today; a reusable composite must survive dirty reruns. Co-Authored-By: Claude Fable 5 * fix: sweep — every sync spawn in the test trees carries a timeout (436 sites, 157 files) spawnSync/execSync/Bun.spawnSync BLOCK the main thread, so bun's in-process per-test timeout can never fire while one waits — a hung child (stdin read, network probe, dead daemon) wedges the whole shard until the runner's external wall-clock SIGKILL. This exact class reached main: free-tests run 33262077256, test/gstack-memory-ingest.test.ts (normally 2.3s) held shard 2 at the 360s wall while its five siblings finished in ~65s. Mechanical sweep in two waves (12 + 4 fan-out agents, every edit verified against its call site): default timeout: 30_000 (matches the free runner's per-test budget), 120_000 for genuinely slow ops (installs, builds, playwright, provider CLIs), helper wrappers fixed ONCE where call sites route through them. Sites that only LOOK like calls (string fixtures, grep needles, comments) were skipped with reasons — the enforcement commit that follows marks them exempt. Co-Authored-By: Claude Fable 5 * test: sync-spawn timeout tripwire — the wedge class stays extinct Free scanner over all test trees (test/, browse/test/, design/test/, make-pdf/test/, ios-qa, browser-skills): every spawnSync/execSync/ Bun.spawnSync call site must carry a timeout within a 30-line options window, or an explicit '// tripwire-exempt: ' marker. Comment lines are skipped; exemptions are counted and ratcheted shrink-only (ceiling 6 = the 6 string-fixture/grep-needle sites where the pattern is CONTENT, not a call — marked in this commit). A scan-sanity test pins that the scanner still sees >100 real call sites so it can never rot to a vacuous green. Companion to the 436-site sweep in the previous commit. Co-Authored-By: Claude Fable 5 * feat: paid-lane flake telemetry — record-level attempts, flaky_retries, report surfacing bun --retry leaves a retried pass INVISIBLE in its output: a fail-then-pass prints the error detail but no (fail) result line and recaps as a clean pass (probed live on 1.3.10). So attempts are recorded where they cannot lie: EvalCollector.addTest stamps a 1-based attempt on same-name re-records (a retried test runs its body again and re-records), finalized runs carry flaky_retries, printSummary warns loudly, and the fail-closed slices report lists every passed-only-on-retry test — recorded and ranked, never blocking and never silent. Cross-model confirmed (codex reached the same don't-parse -the-stream conclusion independently). Co-Authored-By: Claude Fable 5 * feat: free-lane flake ledger — retry ON in CI, flaky-passes recorded and uploaded The runner's attribution-gated flaky-retry pass (cap 5, truncation veto) was OFF in the required lane and its FLAKY-PASS evidence was console-only — so a single timing flake red the merge gate while repeat offenders stayed unenumerable. free-tests.yml now sets GSTACK_FREE_RETRY_FLAKY=1 and points GSTACK_FLAKE_LEDGER at runner.temp; every flaky-pass appends a JSONL entry (SINGLE writer: the parent runner — no concurrent-append hazard by construction; fail-open with a loud warning so a broken ledger can never red the lane) and the artifact uploads UNCONDITIONALLY — a flaky-pass run is green, which is exactly when the evidence matters. Wiring pinned by free-tests-workflow-wiring; ledger behavior unit-tested incl. the fail-open path. Matches 2026 industry practice (retry for data, quarantine out of merge-blocking but never out of logging) with the repo's own receipts. Co-Authored-By: Claude Fable 5 * feat: eval:flake-rank — the flake-telemetry dial Aggregates per-test series across every finalized eval-store run (shard dirs included) plus the free flake ledger: runs, fails, RETRIED PASSES (the flake signature), avg duration — ranked retries-first. This is the readable dial behind two policies: a flaky pass never blocks a merge but is always ranked here, and the WS16 required-check promotion needs weeks of clean flake-rank, not vibes. --json for machines, --dir for downloaded CI artifacts. Co-Authored-By: Claude Fable 5 * feat: two-phase session timeout — silent APIs die at the startup grace, named The single spawn-armed timer charged API queue latency to the work budget: the recurring '0 turns / $0.00 / x3 attempts' failure with four budget-bump receipts (180->300s, 240->360s, 300->420s, 90->300s). Split: startup phase (no NDJSON byte yet) kills EARLY at min(grace, timeout) with the distinct exitReason 'timeout_startup' — an availability verdict, not transcript archaeology — and the work phase arms on the first byte for the REMAINING budget, so total wall never exceeds the timeout (tier envelopes are margin-free: tests pass timeout: CAPTURE_MS and bun-budget the same tier). Local grace 90s (observed queue latency 60-90s), CI floor 300s (TODOS-filed; shared runners queue harder), both pinned by the new grace tests with fake -claude shims covering the late-first-byte and silent-API paths. Co-Authored-By: Claude Fable 5 * fix: census integrity — 17 phantom selection keys deleted, reverse invariant added, gitignored dep patterns replaced, local map forks derived The merge-blocking gate census counted tests that could not run. Deleted (critic-verified against both quoted-occurrence and dep-registration liveness): 7 *-prosons-format keys with no declaring test, ship-plan- completion/-verification, review-plan-completion, design-shotgun-path/ session/full, autoplan-core (dead ~10 months), e2e-harness-audit (its namesake is a FREE-suite file), plus 2 dead LLM-judge keys and 2 free-file keys (budget-regression-pty, global-discover) misplaced in the PAID maps. Census: 191 -> 174 keys, gate 86 -> 78 honest. The new reverse invariant in touchfiles.test.ts makes the class structurally impossible: every key must be quoted in a living paid test file OR registered to an existing paid test file via its dep list (the constructed- name binding the 2026-08 self-registration sweep established) — zero exceptions needed today, with a live-file check on any future exception. Also: '.agents/skills/**' dep patterns replaced with the generator (scripts/gen-skill-docs.ts) — .agents/ is gitignored, so those patterns could NEVER match a git diff and review-template edits silently stopped selecting codex/gemini tests; the codex/gemini local touchfile maps are now DERIVED from the canonical map (loud throw if a key vanishes) instead of hand-forked copies that had already drifted. ios-qa-e2e demoted gate -> periodic: its gate declaration was never executable in CI (hardware exclusion only applies at tier=periodic), so every Linux PR planned a hollow shard. Co-Authored-By: Claude Fable 5 * fix: routing journeys lose their answer key and end at the routing decision The journey tests exist to catch skill-DESCRIPTION regressions (touchfiles: */SKILL.md.tmpl), but the fixture CLAUDE.md shipped an explicit prompt->skill lookup table — with the answer key in context, a badly regressed frontmatter description still routed correctly, so the tests could not fail on the exact class they select for. The fixture now carries only the generic invoke-skills nudge; the frontmatter carries the routing load. Also capped all 10 journeys at maxTurns 2 / tools [Skill, Read]: only the FIRST Skill call is asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — roughly halves each journey's cost. Co-Authored-By: Claude Fable 5 * fix: retire decided A/B experiments; vendor the pre-cut fixture; ban raw-SHA fixtures Three one-shot decision experiments kept re-running weekly as N=1 stochastic comparisons — flaky by construction with near-zero remaining information: skill-e2e-auq-repetition-cut-ab (its own header: gate "passed pre-landing, approved 2026-08-25"), skill-e2e-preamble-script-ab ("demoted post-Phase-3"), and opus-47's fanout arm-vs-arm (parA >= parB across two SINGLE stochastic runs — a coin flip). Deleted, with their selection keys; the SDK overlay-harness stays as the maintained instrument for the next experiment, and opus-47 keeps its routing-precision cases. verboseSkill() now reads the VENDORED test/fixtures/auq-pre-cut-...-SKILL.md instead of `git show ab66193e^:...` — a branch-local ref that dies on branch prune and already failed on shallow clones. New free tripwire (test/git-ref-fixture-tripwire.test.ts) bans the raw-SHA fixture class outright: quoted SHA:path rev-specs and gitRef-style hex defaults in the test trees fail the suite with the vendor-instead instruction. Co-Authored-By: Claude Fable 5 * fix: demote plan-ceo-review-expansion-energy to periodic Opus generator + a subjective 2-axis >=4/5 LLM-judge threshold sat in the MERGE-BLOCKING gate — the exact class its sibling posture tests were demoted for, with a receipt (a +21-line preamble change once flipped the score). CLAUDE.md's own tiering rule: Opus model test -> periodic. The weekly lane keeps the regression signal; merges stop paying a judge- temperament tax. Co-Authored-By: Claude Fable 5 * fix: paid shards get per-shard TMPDIR + CHROMIUM_PROFILE isolation and a kill-path cleanup backstop The free runner treats this isolation as MANDATORY (two concurrent shards on one Chromium profile kill each other's browser; shared tmp cross-contaminates) — the paid lane had none of it. Doubly load-bearing here: a shard that hits its 30-min wall is group-SIGKILLed, so per-test afterAll cleanup never runs; the rmSync backstop is the only thing keeping wedged runs from accumulating full git-repo workspaces in the shared tmpdir forever. This is the DAG prerequisite for raising EVALS_JOBS (next commit) — more concurrency on shared state amplifies exactly the shared-tree race class opus-47 exhibited. Co-Authored-By: Claude Fable 5 * feat: paid-runner defaults 4x4 -> 8x2 — halve the local gate worst case 39 of 75 skill-e2e files hold exactly ONE test, so within-shard concurrency was dead weight for most shards: 4 jobs x 4 concurrency yielded only ~4-6 real in-flight sessions and a 13-wave local gate worst case (~6.5h). 8 jobs x 2 gives ~10-13 in-flight — under the documented-safe ~15 — and ~7 waves (~3.3h worst case). CI lanes keep their explicit EVALS_JOBS env (2 per slice; 4 for gate-census); this changes local defaults. Rollback trigger: sustained 429 storms in the WS1 telemetry across 2 PR cycles. test/eval-detach-timeout-floor.test.ts recomputed green (the raise LOWERS the worst-case floor). Co-Authored-By: Claude Fable 5 * fix: SHA-pin every action in the secrets-bearing eval lanes evals.yml and evals-periodic.yml execute PR-authored code with three provider API keys in env, yet rode mutable action tags (@v7/@v8/@v2/@v4) — while quality-gate.yml, osv-scanner.yml, and dependency-review.yml already model the SHA-pin pattern. All 30 uses sites across both lanes now pin the exact commit (tag noted in a trailing comment); dependabot's github-actions ecosystem keeps them fresh via PRs instead of silent tag moves. Pulled forward from the plan's endgame on the CEO-review + outside- voice agreement: supply-chain pins on secret lanes go first, not last. Co-Authored-By: Claude Fable 5 * fix: sweep wave 3 — the execFileSync family gets timeouts (90 sites, 17 files) The tripwire's regex covered spawnSync/execSync/Bun.spawnSync but not execFileSync — an entire blocking sync-spawn API family that could reintroduce the shard-wedge class undetected (ship review army). Same mechanical recipe as waves 1-2: timeout: 30_000 default, 120_000 for slow ops, shared wrappers fixed once, string-needle sites skipped with reasons. Co-Authored-By: Claude Fable 5 * test: review-army + adversarial test hardening - Tripwire scans execFileSync too (ceiling 8: two more grep-needle string exemptions); merge-introduced timeout-less spawnSync in question-preference-hook fixed — the tripwire caught a site that landed on main AFTER the sweep, on its first day. - gstack-detach gains TWO watchdog kill regression tests: TERM-immune grandchild (the killpg-after-grace escalation) and the leader-dies variant (the pgid-at-spawn fix — the case the first test cannot see). - eval-flake-rank gets its unit suite (final-attempt accounting, artifact exclusion, shard recursion, recency bound). - Groupkill/startup-grace shim markers are per-run unique (pid-suffixed sleep durations): sibling Conductor worktrees run free suites with no machine lock, and fixed markers let one run pgrep/pkill the other's shims — a cross-run flake inside the anti-flake tests. - flake-ledger test pins the project-scoped local default; stale empty section headers in touchfiles-data deleted (they invited entries under deliberately retired categories). Co-Authored-By: Claude Fable 5 * fix: adversarial-review runtime fixes across the telemetry + kill paths - session-runner: exit-labeling keys off 'exit', not 'close' — an orphan holding the pipes could relabel a REAL exit (auth failure) as 'timeout_startup' availability noise; the kill path still always group-kills and cancels the reader (labeling and unblocking are separate concerns). Work phase arms on a flag, not firstResponseMs===0 (a same-ms first byte left the startup timer live all run). The CI startup grace is now a real FLOOR (Math.max), matching its name and pinning test. - gstack-detach: pgid captured AT SPAWN (== child pid under start_new_session) — resolving it after the grace raised ESRCH once the leader died on SIGTERM, orphaning TERM-immune grandchildren forever. - test-free-shards: ledger entries carry branch + git_sha (rev-parse split: '--abbrev-ref HEAD HEAD' printed the branch twice and recorded it as the sha); local ledger default is per-PROJECT, not the machine-global tmpdir. - eval-flake-rank: per-LINE ledger parse (one torn JSONL line vanished the whole series), 60-day recency bound (transcript-bearing files are MBs), shared isFinalizedEvalResultFile predicate (the artifact-taxonomy rule lived in three places); eval-store exports the predicate and finalize stops computing flakyRetries twice; paid-shards cleanup uses async rm (a SIGKILLed shard's git-workspace teardown blocked every sibling's stream classification on the parent event loop). Co-Authored-By: Claude Fable 5 * fix: CI trust-boundary + fail-closed repairs (adversarial findings) - Token/exec separation restored: slices-report (runs PR-authored code: bun install + the reconcile runner) drops to contents:read; the PR comment moves to a NEW slices-comment job holding the write token with ZERO repo code — no checkout, no bun, only downloaded artifacts + jq/gh. $GITHUB_ENV/BASH_ENV persistence is job-scoped, so the split is the boundary. The matrix-era report job had this property; the consolidation had regressed it. Pinned by the wiring test. - Reconcile exit captured via PIPESTATUS[0] in BOTH lanes: GitHub's default run-step shell has no pipefail, so `$?` after `| tee` was tee's exit — the fail-closed gate was silently fail-open. Wiring test pins it. - PR comment: final-attempt accounting restored the dropped COST accumulation (the dial read $0 forever), flaky passes render as the warning they are (never as failures), and a malformed tests[] artifact skips that file instead of aborting the whole comment under bash -e. - Remaining mutable action tags pinned (free-tests upload-artifact, ci-image checkout/docker trio — the image publisher holds packages:write and feeds the secret-bearing lanes). restore-deps fallback installs --frozen-lockfile; register-gstack-skills validates skill names before its rm -rf. Co-Authored-By: Claude Fable 5 * chore: bump version and changelog (v1.77.0.0) Co-Authored-By: Claude Fable 5 * docs: update project documentation for v1.77.0.0 Co-Authored-By: Claude Fable 5 * docs: cross-model doc-review fixes — flake-ledger env knobs, CI retry-on note, stale version comment Co-Authored-By: Claude Fable 5 * docs: correct CHANGELOG receipt numbers to measured values Gate census keys: 78 -> 77 (bun-imported E2E_TIERS count). Sweep receipt: 586 sites/176 files -> 499 sites/146 files, measured by running this branch's spawnsync-timeout-tripwire against origin/main (exit 1, 499 violations across 146 unique files; green on this branch). Co-Authored-By: Claude Fable 5 * fix: slices-comment creates the PR comment via REST — the write-token job has no git context The token/exec split gives slices-comment NO checkout by design, and gh's pr-comment subcommand resolves the repo FROM git — it died with 'not a git repository' on PR #2746's first run (the update-existing PATCH path was already explicit-repo REST and worked). Create now posts through gh api repos/.../issues/N/comments, and the wiring test pins that no git-context-requiring comment call can creep back into the job. Co-Authored-By: Claude Fable 5 * fix: startup-grace probes clear CI for local semantics; new probe pins the floor clamp The two shim probes pass explicit 2s/4s graces, but in CI the runner clamps any explicit grace up to the 300s floor (deliberate adversarial-review fix), so 'silent API killed at the grace' died at the 30s work cap instead of 2s — a deterministic red on every CI run, green locally. The probes now pin LOCAL semantics with CI cleared (same save/restore pattern as their PATH shim), and a fourth probe pins the clamp itself: CI=1 + 2s grace + 6s timeout must kill at the 6s cap, still in the startup phase — proof an explicit low grace cannot bypass the floor. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Fable 5 --- .github/actions/fix-bun-temp/action.yml | 18 + .../actions/register-gstack-skills/action.yml | 108 +++ .github/actions/restore-deps/action.yml | 27 + .github/actions/seed-claude-config/action.yml | 30 + .github/docker/Dockerfile.ci | 11 +- .github/workflows/ci-image.yml | 18 +- .github/workflows/evals-periodic.yml | 126 +--- .github/workflows/evals.yml | 688 +++++------------- .github/workflows/free-tests.yml | 25 +- ARCHITECTURE.md | 2 +- CHANGELOG.md | 56 ++ CLAUDE.md | 7 +- CONTRIBUTING.md | 11 +- TODOS.md | 30 +- VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- bin/gstack-detach | 21 +- browse/test/build.test.ts | 2 +- browse/test/bun-polyfill.test.ts | 28 +- browse/test/config.test.ts | 16 +- browse/test/file-permissions.test.ts | 4 +- browse/test/findport.test.ts | 6 +- browse/test/gstack-config.test.ts | 1 + browse/test/gstack-update-check.test.ts | 1 + browse/test/temp-dirs.test.ts | 2 +- browse/test/windows-spawn-hide.test.ts | 1 + browse/test/xvfb.test.ts | 2 +- docs/PROJECT_STRUCTURE.md | 3 +- docs/TESTING_INTERNALS.md | 47 +- ios-qa/daemon/test/cli-mint.test.ts | 3 +- ios-qa/scripts/gen-accessors.test.ts | 16 +- ios-qa/scripts/gen-accessors.ts | 3 +- make-pdf/test/e2e/combined-gate.test.ts | 1 + make-pdf/test/e2e/diagram-gate.test.ts | 1 + package.json | 3 +- scripts/eval-flake-rank.ts | 144 ++++ scripts/test-free-shards.ts | 76 ++ scripts/test-paid-shards.ts | 115 ++- scripts/test-strict-output.ts | 23 + test/artifacts-init-migration.test.ts | 2 + test/auq-error-fallback-hook.test.ts | 1 + test/bin-context-windows-slug.test.ts | 42 +- test/binding-template-drift.test.ts | 4 +- test/brain-sync.test.ts | 24 +- test/branch-slug-hygiene.test.ts | 10 +- test/builder-profile.test.ts | 2 +- test/ci-image-cli-pin.test.ts | 67 ++ test/code-intelligence.test.ts | 20 +- test/codex-e2e-plan-format.test.ts | 2 +- ...codex-e2e-recommendation-substance.test.ts | 2 +- test/codex-e2e-sol-scope.test.ts | 4 +- test/codex-e2e.test.ts | 19 +- test/codex-generation-model.test.ts | 2 + test/codex-hardening.test.ts | 2 +- test/codex-resume-flag-semantics.test.ts | 2 +- test/codex-web-search-flag.test.ts | 2 +- test/context-bill.test.ts | 2 +- test/design-flag-utils.test.ts | 2 +- test/diagram-render-drift.test.ts | 2 +- test/distill-apply.test.ts | 2 +- test/distill-free-text.test.ts | 8 +- test/e2e-tier-alignment.test.ts | 8 +- test/egress-receipt.test.ts | 8 +- test/empty-find-fallthrough.test.ts | 5 +- test/eval-budgets-policy.test.ts | 2 +- test/eval-flake-rank.test.ts | 70 ++ test/eval-list-cli.test.ts | 1 + test/evals-workflow-matrix.test.ts | 197 ----- test/evals-workflow-wiring.test.ts | 169 +++++ test/flake-ledger.test.ts | 64 ++ test/founder-resources-optout.test.ts | 2 + test/free-tests-workflow-wiring.test.ts | 11 + test/fs-utils.test.ts | 7 +- test/gbrain-detect-install.test.ts | 1 + test/gbrain-exec-invariant.test.ts | 3 + test/gbrain-init-rollback.test.ts | 1 + test/gbrain-init-voyage-code-3.test.ts | 4 +- test/gbrain-lib-verify.test.ts | 2 + test/gbrain-local-status.test.ts | 4 + test/gbrain-repo-policy-client.test.ts | 2 +- test/gbrain-repo-policy.test.ts | 4 +- test/gbrain-source-worktree-advance.test.ts | 3 +- test/gbrain-sync-skip.test.ts | 5 +- ...ain-sync-voyage-code-3-integration.test.ts | 12 +- test/gemini-e2e.test.ts | 18 +- test/gen-skill-docs-import-purity.test.ts | 2 +- test/gen-skill-docs-out-dir.test.ts | 2 +- test/gen-skill-docs.test.ts | 19 +- test/git-ref-fixture-tripwire.test.ts | 64 ++ test/global-discover.test.ts | 3 +- test/gstack-artifacts-init.test.ts | 17 +- test/gstack-artifacts-url.test.ts | 2 +- test/gstack-codex-session-import.test.ts | 4 +- test/gstack-config-defaults.test.ts | 1 + test/gstack-config-key-locale.test.ts | 1 + test/gstack-config-redact-keys.test.ts | 1 + test/gstack-decision-bins.test.ts | 6 +- test/gstack-detach.test.ts | 59 +- test/gstack-developer-profile.test.ts | 2 + test/gstack-egress-cli.test.ts | 3 + test/gstack-gbrain-detect-mcp-mode.test.ts | 1 + test/gstack-gbrain-mcp-verify.test.ts | 3 + test/gstack-gbrain-source-wireup.test.ts | 21 +- test/gstack-gbrain-sync.test.ts | 52 +- test/gstack-home-module-scope.test.ts | 2 +- test/gstack-learnings-search.test.ts | 1 + test/gstack-memory-ingest.test.ts | 16 +- test/gstack-next-version.test.ts | 32 +- test/gstack-paths.test.ts | 3 + test/gstack-question-log.test.ts | 1 + test/gstack-question-preference.test.ts | 2 + test/gstack-redact-cli.test.ts | 3 +- test/gstack-repo-mode.test.ts | 1 + test/gstack-session-kind.test.ts | 1 + .../gstack-settings-hook-schema-aware.test.ts | 1 + test/gstack-skill-start.test.ts | 5 +- test/gstack-slug-cwd-walk-up.test.ts | 3 +- test/gstack-slug-sanitize.test.ts | 3 + test/gstack-state-root-override.test.ts | 1 + test/gstack-team-init-hook-schema.test.ts | 1 + ...gstack-upgrade-migration-v1_17_0_0.test.ts | 1 + ...gstack-upgrade-migration-v1_40_0_0.test.ts | 1 + test/gstack-version-bump.test.ts | 126 ++-- test/helpers/auq-sdk-capture.ts | 16 +- test/helpers/capture-parity-baseline.ts | 4 +- test/helpers/codex-session-runner.ts | 45 +- test/helpers/eval-store.test.ts | 30 + test/helpers/eval-store.ts | 84 ++- test/helpers/gemini-session-runner.ts | 44 +- test/helpers/session-runner.ts | 131 +++- test/helpers/touchfiles-data.ts | 73 +- test/heredoc-pipe-deadlock.test.ts | 6 +- test/host-config.test.ts | 4 +- test/ios-qa-regen.test.ts | 10 +- test/learnings.test.ts | 2 +- test/memory-cache-injection.test.ts | 1 + test/memory-ingest-include-gitignored.test.ts | 3 +- test/migrations-v1.27.0.0.test.ts | 1 + test/migrations-v1.65.0.0.test.ts | 1 + test/mktemp-portability.test.ts | 4 +- test/no-stale-gstack-brain-refs.test.ts | 2 +- test/paid-orphan-tripwire.test.ts | 9 +- test/paid-shards.test.ts | 66 +- test/plan-tune-gates.test.ts | 2 +- test/plan-tune.test.ts | 14 +- test/pr-title-rewrite.test.ts | 4 +- test/preamble-first-task-scaffold.test.ts | 2 +- test/question-log-hook.test.ts | 4 +- test/question-preference-hook.test.ts | 4 +- test/readme-throughput.test.ts | 1 + test/redact-audit-log.test.ts | 2 +- test/redact-prepush-hook.test.ts | 17 +- test/redact-prepush-rebase-force-push.test.ts | 5 +- test/redact-prepush-scan-range.test.ts | 5 +- test/regression-issue2091-bsd-mktemp.test.ts | 1 + test/regression-pr1169-build-app-sed.test.ts | 3 +- test/resolvers-gbrain-put-rewrite.test.ts | 1 + test/review-log.test.ts | 4 +- test/routing-probe.test.ts | 4 +- test/sandbox-doctor-shell.test.ts | 2 +- test/session-runner-groupkill.test.ts | 110 +++ test/session-runner-startup-grace.test.ts | 156 ++++ test/session-update-autostash.test.ts | 4 +- test/setup-bun-cmd-and-pipe-bugs.test.ts | 2 +- test/setup-conductor-worktree.test.ts | 10 +- ...tup-plan-tune-hooks-noninteractive.test.ts | 19 +- test/ship-test-detection-markers.test.ts | 4 +- test/ship-version-sync.test.ts | 6 +- test/skill-e2e-auq-repetition-cut-ab.test.ts | 124 ---- test/skill-e2e-auto-decide-preserved.test.ts | 3 + test/skill-e2e-design.test.ts | 2 +- test/skill-e2e-first-task-scaffold.test.ts | 6 +- test/skill-e2e-ios-device.test.ts | 2 +- test/skill-e2e-ios-swift-build.test.ts | 6 +- test/skill-e2e-opus-47.test.ts | 140 +--- test/skill-e2e-plan-tune-cathedral.test.ts | 16 +- test/skill-e2e-preamble-script-ab.test.ts | 108 --- test/skill-e2e-qa-workflow.test.ts | 4 +- test/skill-e2e-review-attribution.test.ts | 2 +- test/skill-e2e-session-intelligence.test.ts | 6 +- test/skill-e2e-ship-idempotency.test.ts | 4 +- test/skill-e2e-triage.test.ts | 1 + test/skill-e2e-workflow.test.ts | 2 +- test/skill-routing-e2e.test.ts | 105 ++- test/skill-size-budget.test.ts | 3 +- test/skill-validation.test.ts | 24 +- test/spawnsync-timeout-tripwire.test.ts | 132 ++++ test/strict-output.test.ts | 32 + test/tasks-section-jq.test.ts | 2 +- test/taste-engine.test.ts | 2 +- test/team-mode.test.ts | 16 +- test/telemetry-repo-strip.test.ts | 3 +- test/test-free-shards.test.ts | 2 + test/timeline.test.ts | 2 +- test/touchfiles.test.ts | 72 +- test/tracker-guard-wiring.test.ts | 2 +- test/uninstall.test.ts | 15 +- test/upgrade-migration-v1.test.ts | 1 + test/user-render-out-dir-install.test.ts | 2 +- test/worktree.test.ts | 20 +- 200 files changed, 3003 insertions(+), 1753 deletions(-) create mode 100644 .github/actions/fix-bun-temp/action.yml create mode 100644 .github/actions/register-gstack-skills/action.yml create mode 100644 .github/actions/restore-deps/action.yml create mode 100644 .github/actions/seed-claude-config/action.yml create mode 100644 scripts/eval-flake-rank.ts create mode 100644 test/ci-image-cli-pin.test.ts create mode 100644 test/eval-flake-rank.test.ts delete mode 100644 test/evals-workflow-matrix.test.ts create mode 100644 test/evals-workflow-wiring.test.ts create mode 100644 test/flake-ledger.test.ts create mode 100644 test/git-ref-fixture-tripwire.test.ts create mode 100644 test/session-runner-groupkill.test.ts create mode 100644 test/session-runner-startup-grace.test.ts delete mode 100644 test/skill-e2e-auq-repetition-cut-ab.test.ts delete mode 100644 test/skill-e2e-preamble-script-ab.test.ts create mode 100644 test/spawnsync-timeout-tripwire.test.ts diff --git a/.github/actions/fix-bun-temp/action.yml b/.github/actions/fix-bun-temp/action.yml new file mode 100644 index 000000000..02a4b0444 --- /dev/null +++ b/.github/actions/fix-bun-temp/action.yml @@ -0,0 +1,18 @@ +name: Fix bun temp +description: > + Redirect bun's cache/temp to a runner-writable dir. Bun creates root-owned + temp dirs during the Docker image build; GH Actions runs container jobs as + the `runner` user with HOME=/github/home, so without this redirect bun hits + EACCES on its default temp paths. Shared by every eval lane — extracted from + three byte-identical copies (legacy matrix, sliced lane, periodic). +runs: + using: composite + steps: + - shell: bash + run: | + mkdir -p /home/runner/.cache/bun + { + echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun" + echo "BUN_TMPDIR=/home/runner/.cache/bun" + echo "TMPDIR=/home/runner/.cache" + } >> "$GITHUB_ENV" diff --git a/.github/actions/register-gstack-skills/action.yml b/.github/actions/register-gstack-skills/action.yml new file mode 100644 index 000000000..1a96953d1 --- /dev/null +++ b/.github/actions/register-gstack-skills/action.yml @@ -0,0 +1,108 @@ +name: Register gstack skills for PTY tests +description: > + Register the skills PTY smokes invoke, in every place claude looks: + user-scoped ($HOME/.claude/skills — a gstack root symlink for the + preamble's absolute runtime paths, plus per-skill REAL-FILE copies because + claude's interactive-TUI skill scanner does not follow the + /github/home -> /__w cross-mount symlink), and project-scoped + (/.claude/skills, which the TUI reads for /slash commands and which + is gitignored so absent on a fresh CI checkout). Also pre-seeds every + one-time preamble marker so no PTY child takes a first-run branch mid-test + (feature discovery under ~/.claude/skills/gstack trips Claude Code's + sensitive-file permission prompt — the documented intermittent + scope-gate-question-NOT-observed failure). + + Ends with the fail-fast verification loop the legacy matrix copy grew + after a silent "Unknown command" + 35-min-timeout incident: a dangling + symlink or renamed committed target fails HERE, in seconds, with a named + path — never as a wedged PTY session at the shard wall. Every consuming + lane inherits the loop by construction (it previously existed only in the + matrix copy; the sliced + periodic copies had silently dropped it). +inputs: + skills: + description: Space-separated skill dirs (each must have SKILL.md + sections/). + required: false + default: office-hours plan-ceo-review plan-eng-review plan-design-review +runs: + using: composite + steps: + - shell: bash + env: + SKILLS: ${{ inputs.skills }} + run: | + set -eu + SKILLS_DIR="$HOME/.claude/skills" + REPO="$GITHUB_WORKSPACE" + mkdir -p "$SKILLS_DIR" + # The gstack root stays a symlink — the preamble's runtime bash resolves + # ~/.claude/skills/gstack/bin/* and ~/.claude/skills/gstack//sections/* + # through it, and bash follows cross-mount symlinks fine. + # `ln -snf` replaces a stale SYMLINK but hard-errors on a REAL + # directory under set -eu — clear a real-dir leftover first so the + # composite is rerun-safe (codex diff review). CI HOMEs are fresh; + # this only fires on a dirty rerun. + if [ -d "$SKILLS_DIR/gstack" ] && [ ! -L "$SKILLS_DIR/gstack" ]; then + rm -rf "${SKILLS_DIR:?}/gstack" + fi + ln -snf "$REPO" "$SKILLS_DIR/gstack" + for s in $SKILLS; do + # Input validation before rm -rf: a future caller passing '..' or a + # path segment would delete $HOME/.claude (claude adversarial). + case "$s" in */*|.|..|'') echo "ERROR: invalid skill name: '$s'" >&2; exit 1;; esac + rm -rf "${SKILLS_DIR:?}/$s" + mkdir -p "$SKILLS_DIR/$s" + cp "$REPO/$s/SKILL.md" "$SKILLS_DIR/$s/SKILL.md" + cp -R "$REPO/$s/sections" "$SKILLS_DIR/$s/sections" + done + PROJ_SKILLS="$REPO/.claude/skills" + mkdir -p "$PROJ_SKILLS" + for s in $SKILLS; do + rm -rf "${PROJ_SKILLS:?}/$s" + mkdir -p "$PROJ_SKILLS/$s" + cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md" + cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections" + done + # Pre-seed every ONE-TIME preamble marker so no PTY child ever takes a + # first-run branch mid-test. + mkdir -p "$HOME/.gstack" + touch "$HOME/.gstack/.activated" \ + "$HOME/.gstack/.first-loop-tip-shown" \ + "$HOME/.gstack/.telemetry-prompted" \ + "$HOME/.gstack/.proactive-prompted" \ + "$HOME/.gstack/.completeness-intro-seen" \ + "$HOME/.gstack/.plan-tune-nudge-shown" + # These two resolve through the gstack root symlink into $REPO — + # untracked scratch in the CI checkout, exactly where the preamble looks. + touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \ + "$SKILLS_DIR/gstack/.feature-prompted-model-overlay" + echo "--- registry under $SKILLS_DIR ---" + ls -la "$SKILLS_DIR/gstack" + # ── Fail-fast verification ────────────────────────────────────────── + # A dangling symlink or moved/renamed committed target must fail here + # with a named path, not resurface as a silent "Unknown command" and a + # PTY session wedged to its wall timeout. + for f in \ + "$SKILLS_DIR/gstack/bin/gstack-update-check" \ + "$SKILLS_DIR/gstack/scripts/gen-skill-docs.ts"; do + if [ ! -e "$f" ]; then + echo "ERROR: gstack root symlink dangles or target moved: $f" >&2 + exit 1 + fi + done + for s in $SKILLS; do + if [ ! -e "$SKILLS_DIR/$s/SKILL.md" ]; then + echo "ERROR: skill-registry target missing: $SKILLS_DIR/$s/SKILL.md" >&2 + exit 1 + fi + grep -m1 "^name: $s\$" "$SKILLS_DIR/$s/SKILL.md" >/dev/null \ + || { echo "ERROR: $s SKILL.md missing 'name: $s' frontmatter" >&2; exit 1; } + # Sections must exist BOTH as the copied real files (TUI reads) and + # through the gstack root symlink (the preamble's runtime paths). + for d in "$SKILLS_DIR/$s/sections" "$SKILLS_DIR/gstack/$s/sections"; do + if [ ! -d "$d" ] || [ -z "$(ls -A "$d")" ]; then + echo "ERROR: skill sections missing or empty: $d" >&2 + exit 1 + fi + done + done + echo "skill registry OK" diff --git a/.github/actions/restore-deps/action.yml b/.github/actions/restore-deps/action.yml new file mode 100644 index 000000000..f3bd7bc47 --- /dev/null +++ b/.github/actions/restore-deps/action.yml @@ -0,0 +1,27 @@ +name: Restore deps +description: > + Restore the CI image's pre-installed node_modules via recursive copy, or + fall back to bun install when the lockfile changed. Symlinking breaks bun's + realpath-based module resolution (realpath escapes the workspace and + sibling deps stop resolving); hardlink copy fails across overlay-fs layers + ("Invalid cross-device link"). Recursive copy costs ~5s for ~200 packages — + still far cheaper than a network install. Extracted from five byte-similar + copies across the eval lanes. +runs: + using: composite + steps: + - shell: bash + run: | + if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then + # rm first: `cp -r SRC node_modules` with an existing node_modules + # NESTS the copy (node_modules/node_modules_cache) and leaves stale + # deps active. CI workspaces are fresh today, but a reusable + # composite must survive a rerun/dirty workspace (codex diff review). + rm -rf node_modules + cp -r /opt/node_modules_cache node_modules + else + # Frozen: this composite is canonical for lanes that run PR code + # with provider keys in env — a drifted lockfile must fail loudly, + # never silently re-resolve versions (claude adversarial). + bun install --frozen-lockfile + fi diff --git a/.github/actions/seed-claude-config/action.yml b/.github/actions/seed-claude-config/action.yml new file mode 100644 index 000000000..0ac4d32b9 --- /dev/null +++ b/.github/actions/seed-claude-config/action.yml @@ -0,0 +1,30 @@ +name: Seed claude interactive config +description: > + Seed ~/.claude.json with onboarding-complete plus the API-key approval so + PTY sessions never wedge on the fresh-container onboarding / "use detected + ANTHROPIC_API_KEY?" dialog. Mirrors what the hermetic E2E child env seeds. + Idempotent — safe to run unconditionally in sliced lanes where any slice + can host a PTY test. Only the key's last 20 chars are persisted (the + approval-hash form claude itself writes), never the whole key. +inputs: + anthropic-api-key: + description: API key whose suffix gets pre-approved. Pass from secrets at the call site. + required: false + default: '' +runs: + using: composite + steps: + - shell: bash + env: + ANTHROPIC_API_KEY: ${{ inputs.anthropic-api-key }} + run: | + node -e ' + const fs = require("fs"), os = require("os"), path = require("path"); + const p = path.join(os.homedir(), ".claude.json"); + const seed = fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, "utf8")) : {}; + seed.hasCompletedOnboarding = true; + const key = process.env.ANTHROPIC_API_KEY || ""; + if (key) seed.customApiKeyResponses = { approved: [key.slice(-20)], rejected: [] }; + fs.writeFileSync(p, JSON.stringify(seed, null, 2)); + console.log("seeded", p); + ' diff --git a/.github/docker/Dockerfile.ci b/.github/docker/Dockerfile.ci index 550c90d3a..39247f4a1 100644 --- a/.github/docker/Dockerfile.ci +++ b/.github/docker/Dockerfile.ci @@ -74,8 +74,15 @@ ENV BUN_INSTALL="/usr/local" RUN curl --retry 5 --retry-delay 5 --retry-connrefused -fsSL https://bun.sh/install \ | bash -s "bun-v1.3.13" -# Claude CLI -RUN npm i -g @anthropic-ai/claude-code +# Claude CLI — pinned to an EXACT version, same discipline as the bun pin +# above. The PTY harness (test/helpers/claude-pty-runner.ts) screen-scrapes +# this CLI's TUI (trust dialog, input prompt, spinner glyphs); an unpinned +# install rebuilt weekly rode the TUI wherever it drifted, and that drift +# broke the harness three separate times (welcome-screen wedge on 2.1.233, +# skillify HOME discovery on 2.1.237, guard/freeze hooks on 2.1.162). +# Bump deliberately, via a PR that runs the PTY gate against the new TUI. +# test/ci-image-cli-pin.test.ts fails the free suite if this pin is removed. +RUN npm i -g @anthropic-ai/claude-code@2.1.251 # Playwright system deps (Chromium) — needed for browse E2E tests RUN npx playwright install-deps chromium diff --git a/.github/workflows/ci-image.yml b/.github/workflows/ci-image.yml index 19eceb94b..8776fd7a3 100644 --- a/.github/workflows/ci-image.yml +++ b/.github/workflows/ci-image.yml @@ -1,8 +1,12 @@ name: Build CI Image on: - # Rebuild weekly (Monday 4am UTC) to pick up CLI updates — deliberately 2h - # BEFORE evals-periodic's 6am cron so the weekly eval run finds a fresh - # image instead of racing a half-pushed tag or duplicating the build. + # Weekly self-heal (Monday 4am UTC) — deliberately 2h BEFORE + # evals-periodic's 6am cron so the weekly eval run finds the image instead + # of racing a half-pushed tag. With the claude CLI pinned in Dockerfile.ci + # (v1.77+), this cron no longer pulls CLI updates: when the content-hash + # tag already exists it's a ~30s no-op, and it only rebuilds if the tag + # was somehow lost. CLI bumps happen by editing the Dockerfile pin in a PR + # that runs the PTY gate against the new TUI. schedule: - cron: '0 4 * * 1' # Rebuild on Dockerfile or lockfile changes. package.json is deliberately @@ -32,7 +36,7 @@ jobs: contents: read packages: write steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 # Copy lockfile + package.json into Docker build context - run: cp package.json bun.lock .github/docker/ && cp -R patches .github/docker/patches @@ -44,7 +48,7 @@ jobs: - id: meta run: echo "tag=ghcr.io/${{ github.repository }}/ci:${{ hashFiles('.github/docker/Dockerfile.ci', 'bun.lock', 'patches/**') }}" >> "$GITHUB_OUTPUT" - - uses: docker/login-action@v4 + - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 with: registry: ghcr.io username: ${{ github.actor }} @@ -65,10 +69,10 @@ jobs: # Registry cache export needs a docker-container builder — the default # `docker` driver hard-errors on cache-to. - if: steps.check.outputs.exists == 'false' - uses: docker/setup-buildx-action@v4 + uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 - if: steps.check.outputs.exists == 'false' - uses: docker/build-push-action@v7 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7 with: context: .github/docker file: .github/docker/Dockerfile.ci diff --git a/.github/workflows/evals-periodic.yml b/.github/workflows/evals-periodic.yml index c7b4f1ef2..67c92ef1a 100644 --- a/.github/workflows/evals-periodic.yml +++ b/.github/workflows/evals-periodic.yml @@ -32,7 +32,7 @@ jobs: outputs: image-tag: ${{ steps.meta.outputs.tag }} steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - id: meta # Keep in sync with evals.yml — key on Dockerfile + lockfile only @@ -40,7 +40,7 @@ jobs: # Byte-identity pinned by test/ci-image-tag-binding.test.ts. run: echo "tag=${{ env.IMAGE }}:${{ hashFiles('.github/docker/Dockerfile.ci', 'bun.lock', 'patches/**') }}" >> "$GITHUB_OUTPUT" - - uses: docker/login-action@v4 + - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 with: registry: ghcr.io username: ${{ github.actor }} @@ -61,10 +61,10 @@ jobs: # Registry cache export needs a docker-container builder — the default # `docker` driver hard-errors on cache-to. - if: steps.check.outputs.exists == 'false' - uses: docker/setup-buildx-action@v4 + uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 - if: steps.check.outputs.exists == 'false' - uses: docker/build-push-action@v7 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7 with: context: .github/docker file: .github/docker/Dockerfile.ci @@ -90,24 +90,19 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} options: --user runner steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false - name: Restore deps - run: | - if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then - cp -r /opt/node_modules_cache node_modules - else - bun install - fi + uses: ./.github/actions/restore-deps - name: Emit run manifest (ALL periodic tests minus reasoned excludes) env: EVALS_ALL: "1" run: EVALS_TIER=periodic bun run scripts/test-paid-shards.ts --tier periodic --emit-plan /tmp/paid-plan/manifest.json --slices 6 - - uses: actions/upload-artifact@v7 + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: paid-plan path: /tmp/paid-plan/manifest.json @@ -133,7 +128,7 @@ jobs: matrix: slice: [1, 2, 3, 4, 5, 6] steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: # Full history: files with SELF-derived selection (the LLM-judge # map, routing) walk git at module load, and selection is @@ -144,73 +139,27 @@ jobs: persist-credentials: false - name: Fix bun temp - run: | - mkdir -p /home/runner/.cache/bun - { - echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun" - echo "BUN_TMPDIR=/home/runner/.cache/bun" - echo "TMPDIR=/home/runner/.cache" - } >> "$GITHUB_ENV" + uses: ./.github/actions/fix-bun-temp - name: Restore deps - run: | - if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then - cp -r /opt/node_modules_cache node_modules - else - bun install - fi + uses: ./.github/actions/restore-deps - run: bun run build # Any slice can host a PTY test — seed + registration run - # unconditionally (idempotent; mirrors evals.yml's sliced lane). + # unconditionally (idempotent; mirrors evals.yml's sliced lane). The + # register composite carries the fail-fast dangling-symlink/frontmatter + # verification loop — this lane previously LACKED it, so a moved skill + # target surfaced as a silent "Unknown command" + wedged PTY session. - name: Seed claude interactive config - env: - ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} - run: | - node -e ' - const fs = require("fs"), os = require("os"), path = require("path"); - const p = path.join(os.homedir(), ".claude.json"); - const seed = fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, "utf8")) : {}; - seed.hasCompletedOnboarding = true; - const key = process.env.ANTHROPIC_API_KEY || ""; - if (key) seed.customApiKeyResponses = { approved: [key.slice(-20)], rejected: [] }; - fs.writeFileSync(p, JSON.stringify(seed, null, 2)); - console.log("seeded", p); - ' + uses: ./.github/actions/seed-claude-config + with: + anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }} - name: Register gstack skills for PTY tests - run: | - set -eu - SKILLS_DIR="$HOME/.claude/skills" - REPO="$GITHUB_WORKSPACE" - mkdir -p "$SKILLS_DIR" - ln -snf "$REPO" "$SKILLS_DIR/gstack" - for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do - rm -rf "${SKILLS_DIR:?}/$s" - mkdir -p "$SKILLS_DIR/$s" - cp "$REPO/$s/SKILL.md" "$SKILLS_DIR/$s/SKILL.md" - cp -R "$REPO/$s/sections" "$SKILLS_DIR/$s/sections" - done - PROJ_SKILLS="$REPO/.claude/skills" - mkdir -p "$PROJ_SKILLS" - for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do - rm -rf "${PROJ_SKILLS:?}/$s" - mkdir -p "$PROJ_SKILLS/$s" - cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md" - cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections" - done - mkdir -p "$HOME/.gstack" - touch "$HOME/.gstack/.activated" \ - "$HOME/.gstack/.first-loop-tip-shown" \ - "$HOME/.gstack/.telemetry-prompted" \ - "$HOME/.gstack/.proactive-prompted" \ - "$HOME/.gstack/.completeness-intro-seen" \ - "$HOME/.gstack/.plan-tune-nudge-shown" - touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \ - "$SKILLS_DIR/gstack/.feature-prompted-model-overlay" + uses: ./.github/actions/register-gstack-skills - - uses: actions/download-artifact@v8 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: paid-plan path: /tmp/paid-plan @@ -228,7 +177,7 @@ jobs: - name: Upload slice results if: always() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: paid-slice-${{ matrix.slice }} path: /tmp/paid-slice-results @@ -236,7 +185,7 @@ jobs: - name: Upload shard logs on failure if: failure() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: paid-slice-${{ matrix.slice }}-logs # The Fix-bun-temp step points TMPDIR at /home/runner/.cache, so the @@ -266,7 +215,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} options: --user runner steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: # Full history: files with SELF-derived selection (the LLM-judge # map, routing) walk git at module load, and selection is @@ -277,21 +226,10 @@ jobs: persist-credentials: false - name: Fix bun temp - run: | - mkdir -p /home/runner/.cache/bun - { - echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun" - echo "BUN_TMPDIR=/home/runner/.cache/bun" - echo "TMPDIR=/home/runner/.cache" - } >> "$GITHUB_ENV" + uses: ./.github/actions/fix-bun-temp - name: Restore deps - run: | - if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then - cp -r /opt/node_modules_cache node_modules - else - bun install - fi + uses: ./.github/actions/restore-deps - run: bun run build @@ -309,7 +247,7 @@ jobs: - name: Upload census results if: always() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: gate-census path: /tmp/gate-census-results @@ -328,22 +266,22 @@ jobs: # `gh api /issues` — gated by the issues permission. issues: write steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false - - uses: oven-sh/setup-bun@v2 + - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: bun-version: 1.3.13 - run: bun install --frozen-lockfile - - uses: actions/download-artifact@v8 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: paid-plan path: /tmp/paid-report - - uses: actions/download-artifact@v8 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: pattern: paid-slice-[0-9]* path: /tmp/paid-report @@ -354,7 +292,11 @@ jobs: run: | set +e EVALS_TIER=periodic bun run scripts/test-paid-shards.ts --tier periodic --report /tmp/paid-report | tee /tmp/report.txt - echo "exit=$?" >> "$GITHUB_OUTPUT" + # PIPESTATUS[0], NOT $?: GitHub's default run-step shell is + # `bash -e {0}` with NO pipefail, so $? after the pipe is tee's + # exit (always 0) — the fail-closed gate was silently fail-open + # (caught by the ship review army; the wiring test now pins this). + echo "exit=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT" # A red weekly lane nobody must action is waste — upsert ONE tracking # issue (never a new issue per week) with the reconciliation output, so diff --git a/.github/workflows/evals.yml b/.github/workflows/evals.yml index 2d7314d2f..71231a87a 100644 --- a/.github/workflows/evals.yml +++ b/.github/workflows/evals.yml @@ -15,7 +15,6 @@ concurrency: env: IMAGE: ghcr.io/${{ github.repository }}/ci - EVALS_TIER: gate jobs: # Build Docker image with pre-baked toolchain (cached — only rebuilds on Dockerfile/lockfile change) @@ -34,7 +33,7 @@ jobs: outputs: image-tag: ${{ steps.meta.outputs.tag }} steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - id: meta # Key on Dockerfile + lockfile only. package.json is deliberately NOT @@ -44,7 +43,7 @@ jobs: # overwrites /workspace and node_modules comes from the lockfile. run: echo "tag=${{ env.IMAGE }}:${{ hashFiles('.github/docker/Dockerfile.ci', 'bun.lock', 'patches/**') }}" >> "$GITHUB_OUTPUT" - - uses: docker/login-action@v4 + - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 with: registry: ghcr.io username: ${{ github.actor }} @@ -69,10 +68,10 @@ jobs: # Registry cache export needs a docker-container builder — the default # `docker` driver hard-errors on cache-to (first live run of the trio). - if: steps.check.outputs.exists == 'false' - uses: docker/setup-buildx-action@v4 + uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 - if: steps.check.outputs.exists == 'false' - uses: docker/build-push-action@v7 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7 with: context: .github/docker file: .github/docker/Dockerfile.ci @@ -85,430 +84,27 @@ jobs: ${{ steps.meta.outputs.tag }} ${{ env.IMAGE }}:latest - # Fork PRs never receive repository secrets (ANTHROPIC_API_KEY et al), so every - # API-calling eval fails at SDK auth before a model runs. Skip deterministically - # rather than leaving the outcome to Docker-cache luck: a warm cache let these - # run and fail, a cold one made build-image fail its push and the shards skip. - # Same-repo PRs, pushes, and workflow_dispatch keep full coverage. Fork work - # gets real coverage via a trusted base-repo branch. - evals: - runs-on: ${{ matrix.suite.runner || 'ubicloud-standard-8' }} - needs: build-image - if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository - # Least privilege for the job that executes PR-authored code with three - # provider API keys in env: read-only contents, packages:read for the - # container-image pull below. Without this block the job ran on the - # repo-default token grant. - permissions: - contents: read - packages: read - container: - image: ${{ needs.build-image.outputs.image-tag }} - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - options: --user runner - timeout-minutes: ${{ matrix.suite.timeout || 25 }} - strategy: - fail-fast: false - matrix: - suite: - - name: llm-judge - file: test/skill-llm-eval.test.ts - - name: e2e-browse - file: test/skill-e2e-bws.test.ts - runner: ubicloud-standard-8 - - name: e2e-plan - file: test/skill-e2e-plan.test.ts - - name: e2e-deploy - file: test/skill-e2e-deploy.test.ts - - name: e2e-design - file: test/skill-e2e-design.test.ts - - name: e2e-qa-bugs - file: test/skill-e2e-qa-bugs.test.ts - - name: e2e-qa-workflow - file: test/skill-e2e-qa-workflow.test.ts - - name: e2e-review - file: test/skill-e2e-review.test.ts - - name: e2e-retro - file: test/skill-e2e-retro.test.ts - - name: e2e-review-attribution - file: test/skill-e2e-review-attribution.test.ts - - name: e2e-workflow - file: test/skill-e2e-workflow.test.ts - # Earned its extra attempt with receipts: document-release is a - # long multi-step E2E that timed out on attempt 2 under in-shard - # concurrency (PR #2593 round 4) while passing other rounds. - retries: 2 - # Rehomed from the deleted pre-split monolith (its filename never - # matched the skill-e2e-* glob, so these gate tests silently never - # ran). Both files hold gate-tier tests: review/plan-eng coverage - # audits and the /ship failure-ownership triage. - - name: e2e-coverage-audit - file: test/skill-e2e-coverage-audit.test.ts - - name: e2e-triage - file: test/skill-e2e-triage.test.ts - # ship-docsync is whole-file tier-gated (describeE2ETier('gate') keeps - # it out of the periodic shard census), so its row MUST set tier: gate - # — without it the self-gate skips every test and the job reports a - # hollow green (the same silent-skip class as the rehomed monolith - # above, one layer deeper). The Run step exports EVALS_TIER from this - # property; rows without it keep EVALS_TIER empty (= unset: every - # reader is `=== ''` or truthiness). Enforced by - # test/evals-workflow-matrix.test.ts. - - name: e2e-ship-docsync - file: test/skill-e2e-ship-docsync.test.ts - tier: gate - # #2733 behavioral proof: the document-release JSON contract survives - # a firing AUQ gate inside a spawned-marked subagent. Whole-file - # tier-gated like ship-docsync, so the row carries tier: gate. - - name: e2e-docsync-spawned - file: test/skill-e2e-docsync-spawned.test.ts - tier: gate - # Consent-gate guardrail for the Third-Party Web Actions contract - # (Aside recommended-driver rewrite): hermetic shims, deterministic - # grep assertions, gate tier in E2E_TIERS. - - name: e2e-third-party-actions - file: test/skill-e2e-third-party-actions.test.ts - tier: gate - - name: e2e-routing - file: test/skill-routing-e2e.test.ts - # (e2e-codex / e2e-gemini rows deleted: both files are whole-file - # periodic-tier, so with no row tier: they ran ZERO tests and - # reported green on every PR — ~2 min of runner per PR of pure - # false confidence. The periodic lane owns these suites.) - # Real-PTY plan-mode smokes. Only the deterministically-reliable ones - # are CI-gated: office-hours (asks its mode question first, caught by - # the collapsed/bullet prose-AUQ detector) and plan-mode-no-op (no - # ask-first dependency). The plan-eng/plan-design plan-mode + floor - # smokes are periodic (stochastic ask-first — see touchfiles E2E_TIERS). - # Needs the interactive-config seed step below; PTY sessions otherwise - # wedge on the fresh-container onboarding/API-key dialog. - - name: e2e-pty-plan-smoke - file: test/skill-e2e-office-hours-auto-mode.test.ts test/skill-e2e-plan-mode-no-op.test.ts - # Both files are whole-file describeE2ETier('gate') — without this - # row tier: the job burned ~7 min of setup then skipped every - # describe (hollow-green since the files adopted the self-gate). - tier: gate - timeout: 35 - # The documented contention-heavy PTY family: ROTATING members - # failed attempt 2 in consecutive PR #2593 rounds - # (plan-design-review, then plan-eng-review) while the family - # passes on branches still running three attempts. Every other - # row keeps --retry 1. - retries: 2 - steps: - - uses: actions/checkout@v7 - with: - fetch-depth: 0 - # Don't write the token into .git/config — this job runs - # PR-authored code; nothing in it pushes. - persist-credentials: false - - # Bun creates root-owned temp dirs during Docker build. GH Actions runs as - # runner user with HOME=/github/home. Redirect bun's cache to a writable dir. - - name: Fix bun temp - run: | - mkdir -p /home/runner/.cache/bun - { - echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun" - echo "BUN_TMPDIR=/home/runner/.cache/bun" - echo "TMPDIR=/home/runner/.cache" - } >> "$GITHUB_ENV" - - # Restore pre-installed node_modules from Docker image via recursive - # copy. Symlink (`ln -s`) breaks bun's module resolution because bun - # resolves a file's realpath when walking up to find node_modules/; - # from a symlinked path, realpath escapes the workspace and sibling - # deps no longer resolve. Hardlink copy (`cp -al`) fails because /opt - # and /workspace are on different overlay-fs layers ("Invalid - # cross-device link"). Recursive copy works on every layout. Cost: - # ~5s for ~200 packages of small JS files vs ~0s for symlink — still - # vastly cheaper than rerunning `bun install` (network + resolution). - - name: Restore deps - run: | - if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then - cp -r /opt/node_modules_cache node_modules - else - bun install - fi - - - run: bun run build - - # Verify Playwright can launch Chromium (fails fast if sandbox/deps are broken) - - name: Verify Chromium - if: matrix.suite.name == 'e2e-browse' - run: | - echo "whoami=$(whoami) HOME=$HOME TMPDIR=${TMPDIR:-unset}" - touch /tmp/.bun-test && rm /tmp/.bun-test && echo "/tmp writable" - bun -e "import {chromium} from 'playwright';const b=await chromium.launch({args:['--no-sandbox']});console.log('Chromium OK');await b.close()" - - # PTY smokes spawn the interactive `claude` TUI. A fresh container has no - # ~/.claude.json, so claude wedges on the onboarding + "use detected - # ANTHROPIC_API_KEY?" dialog and the spawned session never reaches the - # skill. Seed onboarding-complete + the key approval (mirrors what the - # hermetic E2E child env seeds). Scoped to this suite; needs its OWN key - # env (the secrets block below is on the Run step only). - - name: Seed claude interactive config - if: matrix.suite.name == 'e2e-pty-plan-smoke' - env: - ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} - run: | - node -e ' - const fs = require("fs"), os = require("os"), path = require("path"); - const p = path.join(os.homedir(), ".claude.json"); - const seed = fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, "utf8")) : {}; - seed.hasCompletedOnboarding = true; - const key = process.env.ANTHROPIC_API_KEY || ""; - if (key) seed.customApiKeyResponses = { approved: [key.slice(-20)], rejected: [] }; - fs.writeFileSync(p, JSON.stringify(seed, null, 2)); - console.log("seeded", p); - ' - - # PTY smokes drive the interactive `claude` TUI and send /office-hours, - # /plan-ceo-review, /plan-eng-review, and /plan-design-review. Claude Code - # discovers user-scoped skills from $HOME/.claude/skills//SKILL.md, - # but .claude/skills is gitignored, so a fresh CI checkout has NO registry - # — claude prints "Unknown command: /plan-ceo-review". Mirror setup's - # --no-prefix registry minimally: a gstack root symlink (resolves the - # preamble's absolute ~/.claude/skills/gstack/bin/* and - # ~/.claude/skills/gstack//sections/* paths) plus a per-skill - # top-level dir holding SKILL.md (+ sections) symlinks for the four skills - # these tests invoke. No ./setup (it builds binaries, launches Chromium, - # installs fonts, reads a /dev/tty prompt) and no binary build (SKILL.md + - # bin/ + sections/ are committed). $HOME is /github/home here; the spawned - # claude inherits it (this runner adds no HOME/CLAUDE_CONFIG_DIR override, - # no hermetic mode) and the Seed step already proved claude reads $HOME. - # - # KEEP THIS STEP even though seedSkills/hermeticSkillsConfigDir() now - # registers skills for hermetic PTY children: that registry is SYMLINKS - # into the repo checkout, and this container's cross-mount symlinks - # defeat the TUI skill scanner (see the note inside the step below) — - # the real-file copies here are what the TUI actually reads. HOME is - # also not hermeticized, so the absolute ~/.claude/skills/gstack/... - # preamble paths resolve through the gstack root symlink this step makes. - - name: Register gstack skills for PTY smoke - if: matrix.suite.name == 'e2e-pty-plan-smoke' - run: | - set -eu - SKILLS_DIR="$HOME/.claude/skills" - REPO="$GITHUB_WORKSPACE" # /__w/gstack/gstack - mkdir -p "$SKILLS_DIR" - # The gstack root stays a symlink — the preamble's runtime bash resolves - # ~/.claude/skills/gstack/bin/* and ~/.claude/skills/gstack//sections/* - # through it, and bash follows cross-mount symlinks fine. - ln -snf "$REPO" "$SKILLS_DIR/gstack" - # But the per-skill SKILL.md the TUI DISCOVERS must be a REAL file on the - # same mount as $HOME. claude 2.1.187's interactive-TUI skill scanner does - # not follow the /github/home -> /__w cross-mount symlink (proven: `claude - # -p` discovered the skill — READY — while the TUI rejected /office-hours - # as "Unknown command"; a local macOS repro with the identical symlinked - # registry recognized it, isolating the failure to the container's - # cross-mount symlink). Copy SKILL.md + sections as real files so the TUI - # reads them directly. - for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do - rm -rf "${SKILLS_DIR:?}/$s" - mkdir -p "$SKILLS_DIR/$s" - cp "$REPO/$s/SKILL.md" "$SKILLS_DIR/$s/SKILL.md" - cp -R "$REPO/$s/sections" "$SKILLS_DIR/$s/sections" - done - # Also register PROJECT-scoped (cwd) skills. claude's interactive TUI - # surfaces /slash commands from /.claude/skills, and the smokes run - # with cwd=$REPO whose .claude/skills is gitignored (absent on a fresh CI - # checkout) — the user-dir registration above feeds `claude -p` but the - # TUI looks here. No gstack symlink in the project dir: it would point at - # its own parent ($REPO). Runtime preamble paths use the user-dir - # ~/.claude/skills/gstack symlink above. - PROJ_SKILLS="$REPO/.claude/skills" - mkdir -p "$PROJ_SKILLS" - for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do - rm -rf "${PROJ_SKILLS:?}/$s" - mkdir -p "$PROJ_SKILLS/$s" - cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md" - cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections" - done - # Pre-seed every ONE-TIME preamble marker so no PTY child ever takes a - # first-run branch mid-test. On a fresh runner these are all missing, so - # each smoke's preamble fires feature discovery / telemetry / lake-intro - # prompts before the behavior under test — and touching the - # feature-discovery marker under ~/.claude/skills/gstack/ trips Claude - # Code's sensitive-file permission prompt, stalling the run before the - # scope gate renders (the documented intermittent - # scope-gate-question-NOT-observed failure: outcome=asked was the - # permission dialog, not the gate). Dev machines never hit this because - # the operator's markers already exist; CI must seed them explicitly. - mkdir -p "$HOME/.gstack" - touch "$HOME/.gstack/.activated" \ - "$HOME/.gstack/.first-loop-tip-shown" \ - "$HOME/.gstack/.telemetry-prompted" \ - "$HOME/.gstack/.proactive-prompted" \ - "$HOME/.gstack/.completeness-intro-seen" \ - "$HOME/.gstack/.plan-tune-nudge-shown" - # These two resolve through the gstack root symlink into $REPO — - # untracked scratch in the CI checkout, exactly where the preamble looks. - touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \ - "$SKILLS_DIR/gstack/.feature-prompted-model-overlay" - echo "--- registry under $SKILLS_DIR ---" - ls -la "$SKILLS_DIR/gstack" "$SKILLS_DIR/office-hours" "$SKILLS_DIR/plan-ceo-review" - # Fail fast if any committed target moved/renamed — a dangling symlink - # would otherwise resurface as a silent "Unknown command" + 35-min timeout. - for f in \ - "$SKILLS_DIR/office-hours/SKILL.md" \ - "$SKILLS_DIR/plan-ceo-review/SKILL.md" \ - "$SKILLS_DIR/plan-eng-review/SKILL.md" \ - "$SKILLS_DIR/plan-design-review/SKILL.md" \ - "$SKILLS_DIR/gstack/bin/gstack-update-check" \ - "$SKILLS_DIR/gstack/office-hours/sections/design-and-handoff.md" \ - "$SKILLS_DIR/gstack/plan-ceo-review/sections/review-sections.md" \ - "$SKILLS_DIR/gstack/plan-eng-review/sections/review-sections.md" \ - "$SKILLS_DIR/gstack/plan-design-review/sections/review-sections.md"; do - if [ ! -e "$f" ]; then - echo "ERROR: skill-registry target missing (symlink dangles): $f" >&2 - exit 1 - fi - done - for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do - grep -m1 "^name: $s\$" "$SKILLS_DIR/$s/SKILL.md" >/dev/null \ - || { echo "ERROR: $s SKILL.md missing 'name: $s' frontmatter" >&2; exit 1; } - done - echo "skill registry OK" - - - name: Run ${{ matrix.suite.name }} - env: - ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} - OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} - GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} - EVALS_CONCURRENCY: "40" - PLAYWRIGHT_BROWSERS_PATH: /opt/playwright-browsers - # Per-row tier activation for whole-file-gated suites. Empty when the - # row declares no tier — every EVALS_TIER reader treats empty as unset - # (`=== ''` comparisons and the truthiness check in - # test/helpers/e2e-helpers.ts:70), so untiered rows are byte-for-byte - # unaffected. - EVALS_TIER: ${{ matrix.suite.tier || '' }} - run: EVALS=1 bun test --retry ${{ matrix.suite.retries || 1 }} --concurrent --max-concurrency 40 ${{ matrix.suite.file }} - - - name: Upload eval results - if: always() - uses: actions/upload-artifact@v7 - with: - name: eval-${{ matrix.suite.name }} - path: ~/.gstack-dev/evals/*.json - retention-days: 90 - - report: - runs-on: ubicloud-standard-2 - needs: evals - if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository - timeout-minutes: 5 - permissions: - contents: read - pull-requests: write - # The comment upsert below calls the REST `/issues/{n}/comments` endpoints - # (gh api ... issues/comments). With GITHUB_TOKEN those are gated by the - # `issues` permission, not `pull-requests` — without it the GET returns 401 - # on every PR that produces eval artifacts (PRs with no artifacts exit - # early and never hit it, which is why this stayed hidden). See #1802 CI fix. - issues: write - steps: - - uses: actions/checkout@v7 - with: - fetch-depth: 1 - - - name: Download all eval artifacts - uses: actions/download-artifact@v8 - with: - pattern: eval-* - path: /tmp/eval-results - merge-multiple: true - - - name: Post PR comment - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - # shellcheck disable=SC2086,SC2059 - RESULTS=$(find /tmp/eval-results -name '*.json' 2>/dev/null | sort) - if [ -z "$RESULTS" ]; then - echo "No eval results found" - exit 0 - fi - - TOTAL=0; PASSED=0; FAILED=0; COST="0" - SUITE_LINES="" - for f in $RESULTS; do - if ! jq -e '.total_tests' "$f" >/dev/null 2>&1; then - echo "Skipping malformed JSON: $f" - continue - fi - T=$(jq -r '.total_tests // 0' "$f") - P=$(jq -r '.passed // 0' "$f") - F=$(jq -r '.failed // 0' "$f") - C=$(jq -r '.total_cost_usd // 0' "$f") - TIER=$(jq -r '.tier // "unknown"' "$f") - [ "$T" -eq 0 ] && continue - TOTAL=$((TOTAL + T)) - PASSED=$((PASSED + P)) - FAILED=$((FAILED + F)) - COST=$(echo "$COST + $C" | bc) - STATUS_ICON="✅" - [ "$F" -gt 0 ] && STATUS_ICON="❌" - SUITE_LINES="${SUITE_LINES}| ${TIER} | ${P}/${T} | ${STATUS_ICON} | \$${C} |\n" - done - - STATUS="✅ PASS" - [ "$FAILED" -gt 0 ] && STATUS="❌ FAIL" - - BODY="## E2E Evals: ${STATUS} - - **${PASSED}/${TOTAL}** tests passed | **\$${COST}** total cost - - | Suite | Result | Status | Cost | - |-------|--------|--------|------| - $(echo -e "$SUITE_LINES") - - --- - *ubicloud-standard-8 runners (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite*" - - if [ "$FAILED" -gt 0 ]; then - FAILURES="" - for f in $RESULTS; do - if ! jq -e '.failed' "$f" >/dev/null 2>&1; then continue; fi - F=$(jq -r '.failed // 0' "$f") - [ "$F" -eq 0 ] && continue - FAILS=$(jq -r '.tests[] | select(.passed == false) | "- ❌ \(.name): \(.exit_reason // "unknown")"' "$f" 2>/dev/null || echo "- ⚠️ $(basename "$f"): parse error") - FAILURES="${FAILURES}${FAILS}\n" - done - BODY="${BODY} - - ### Failures - $(echo -e "$FAILURES")" - fi - - # Update existing comment or create new one - COMMENT_ID=$(gh api repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments \ - --jq '.[] | select(.body | startswith("## E2E Evals")) | .id' | tail -1) - - if [ -n "$COMMENT_ID" ]; then - gh api "repos/${{ github.repository }}/issues/comments/${COMMENT_ID}" \ - -X PATCH -f body="$BODY" - else - gh pr comment "${{ github.event.pull_request.number }}" --body "$BODY" - fi - - # ── Sliced lane (paid-CI re-platform, parity phase) ───────────────────────── + # ── Sliced lane (the ONLY paid lane; legacy 17-row matrix deleted) ────────── # One PLANNER computes diff selection + the slice plan ONCE (killing # per-slice selector divergence); K executors consume the manifest; the # report reconciles results against it FAIL-CLOSED (a slice whose artifact # never landed is a failure, a planned shard nobody reported is a failure — - # hollow lanes cannot aggregate green). Runs AFTER the matrix (`needs: - # evals`) so provider concurrency never doubles while both lanes coexist; - # once parity is demonstrated the matrix + its ratchets are deleted and this - # lane loses the needs edge. Engine: scripts/test-paid-shards.ts — the same - # runner local eval:bg:gate uses, so CI and local share one selection engine. + # hollow lanes cannot aggregate green). Engine: scripts/test-paid-shards.ts — + # the same runner local eval:bg:gate uses, so CI and local share one + # selection engine, and every gate-tier file is in the census by + # construction (no hand-enumerated rows to drift). The legacy matrix ran + # 18 enumerated files for 22.6 min/$21 per PR serialized AHEAD of this + # lane's 49-file diff-selected census; parity was demonstrated (sliced + # census ⊇ matrix files) and the matrix deleted — one revert restores it. + # + # Fork PRs never receive repository secrets (ANTHROPIC_API_KEY et al), so + # every API-calling eval fails at SDK auth before a model runs. Skip + # deterministically; fork work gets real coverage via a trusted base-repo + # branch (see CLAUDE.md's garrytan-agents workflow). plan-slices: runs-on: ubicloud-standard-8 - needs: [build-image, evals] - if: always() && needs.build-image.result == 'success' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) + needs: build-image + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository timeout-minutes: 10 permissions: contents: read @@ -520,7 +116,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} options: --user runner steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: # The planner is the ONE place that needs history: diff selection # resolves a merge-base. Executors run from the manifest and stay @@ -530,19 +126,14 @@ jobs: persist-credentials: false - name: Restore deps - run: | - if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then - cp -r /opt/node_modules_cache node_modules - else - bun install - fi + uses: ./.github/actions/restore-deps - name: Emit run manifest env: EVALS_ALL: ${{ (github.event_name == 'workflow_dispatch' && inputs.evals_all) && '1' || '' }} run: EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --emit-plan /tmp/paid-plan/manifest.json --slices 6 - - uses: actions/upload-artifact@v7 + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: paid-plan path: /tmp/paid-plan/manifest.json @@ -572,7 +163,7 @@ jobs: matrix: slice: [1, 2, 3, 4, 5, 6] steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: # Full history: files with SELF-derived selection (the LLM-judge # map, routing) walk git at module load, and selection is @@ -583,74 +174,28 @@ jobs: persist-credentials: false - name: Fix bun temp - run: | - mkdir -p /home/runner/.cache/bun - { - echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun" - echo "BUN_TMPDIR=/home/runner/.cache/bun" - echo "TMPDIR=/home/runner/.cache" - } >> "$GITHUB_ENV" + uses: ./.github/actions/fix-bun-temp - name: Restore deps - run: | - if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then - cp -r /opt/node_modules_cache node_modules - else - bun install - fi + uses: ./.github/actions/restore-deps - run: bun run build # Any slice can host a PTY smoke, so the seed/registration steps run # UNCONDITIONALLY (both are idempotent) — the old matrix keyed them on - # matrix.suite.name, which a sliced lane cannot do. + # matrix.suite.name, which a sliced lane cannot do. The register + # composite carries the fail-fast dangling-symlink/frontmatter + # verification loop, so a moved skill target fails HERE in seconds, + # not as a wedged PTY session at the shard wall. - name: Seed claude interactive config - env: - ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} - run: | - node -e ' - const fs = require("fs"), os = require("os"), path = require("path"); - const p = path.join(os.homedir(), ".claude.json"); - const seed = fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, "utf8")) : {}; - seed.hasCompletedOnboarding = true; - const key = process.env.ANTHROPIC_API_KEY || ""; - if (key) seed.customApiKeyResponses = { approved: [key.slice(-20)], rejected: [] }; - fs.writeFileSync(p, JSON.stringify(seed, null, 2)); - console.log("seeded", p); - ' + uses: ./.github/actions/seed-claude-config + with: + anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }} - name: Register gstack skills for PTY smokes - run: | - set -eu - SKILLS_DIR="$HOME/.claude/skills" - REPO="$GITHUB_WORKSPACE" - mkdir -p "$SKILLS_DIR" - ln -snf "$REPO" "$SKILLS_DIR/gstack" - for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do - rm -rf "${SKILLS_DIR:?}/$s" - mkdir -p "$SKILLS_DIR/$s" - cp "$REPO/$s/SKILL.md" "$SKILLS_DIR/$s/SKILL.md" - cp -R "$REPO/$s/sections" "$SKILLS_DIR/$s/sections" - done - PROJ_SKILLS="$REPO/.claude/skills" - mkdir -p "$PROJ_SKILLS" - for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do - rm -rf "${PROJ_SKILLS:?}/$s" - mkdir -p "$PROJ_SKILLS/$s" - cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md" - cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections" - done - mkdir -p "$HOME/.gstack" - touch "$HOME/.gstack/.activated" \ - "$HOME/.gstack/.first-loop-tip-shown" \ - "$HOME/.gstack/.telemetry-prompted" \ - "$HOME/.gstack/.proactive-prompted" \ - "$HOME/.gstack/.completeness-intro-seen" \ - "$HOME/.gstack/.plan-tune-nudge-shown" - touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \ - "$SKILLS_DIR/gstack/.feature-prompted-model-overlay" + uses: ./.github/actions/register-gstack-skills - - uses: actions/download-artifact@v8 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: paid-plan path: /tmp/paid-plan @@ -668,7 +213,7 @@ jobs: - name: Upload slice results if: always() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: paid-slice-${{ matrix.slice }} path: /tmp/paid-slice-results @@ -678,7 +223,7 @@ jobs: # later needs more than a summary line. - name: Upload shard logs on failure if: failure() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: paid-slice-${{ matrix.slice }}-logs # The Fix-bun-temp step points TMPDIR at /home/runner/.cache, so the @@ -697,29 +242,180 @@ jobs: # missing slice artifact reading as green is the class this lane kills. if: always() && needs.plan-slices.result == 'success' timeout-minutes: 5 + # contents:read ONLY — this job executes PR-authored code (bun install + # lifecycle scripts + the reconcile runner from the PR checkout), so it + # must never hold a write-scoped token. The PR comment lives in the + # separate slices-comment job below, which runs NO repo code: a + # $GITHUB_ENV/BASH_ENV persistence trick is job-scoped, so the split is + # the trust boundary (codex adversarial finding, 2026-08-31 — the old + # matrix-era report job had this separation and the consolidation had + # regressed it). permissions: contents: read + outputs: + reconcile-exit: ${{ steps.reconcile.outputs.exit }} steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false - - uses: oven-sh/setup-bun@v2 + - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: bun-version: 1.3.13 - run: bun install --frozen-lockfile - - uses: actions/download-artifact@v8 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: paid-plan path: /tmp/paid-report - - uses: actions/download-artifact@v8 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: pattern: paid-slice-[0-9]* path: /tmp/paid-report merge-multiple: true - name: Reconcile slices against the manifest (fail-closed) - run: EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --report /tmp/paid-report + id: reconcile + run: | + set +e + EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --report /tmp/paid-report | tee /tmp/report.txt + # PIPESTATUS[0], NOT $?: GitHub's default run-step shell is + # `bash -e {0}` with NO pipefail, so $? after the pipe is tee's + # exit (always 0) — the fail-closed gate was silently fail-open + # (caught by the ship review army; the wiring test now pins this). + echo "exit=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT" + + - name: Upload reconciliation output for the comment job + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: report-verdict + path: /tmp/report.txt + if-no-files-found: ignore + retention-days: 30 + + - name: Fail the workflow when reconciliation failed + if: steps.reconcile.outputs.exit != '0' + run: exit 1 + + # PR comment in its OWN job with the write token and ZERO repo code: no + # checkout, no bun install — only downloaded artifacts, jq, and gh. See the + # trust-boundary note on slices-report. + slices-comment: + runs-on: ubicloud-standard-2 + needs: slices-report + if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && needs.slices-report.result != 'skipped' + timeout-minutes: 5 + permissions: + pull-requests: write + # The comment upsert calls the REST `/issues/{n}/comments` endpoints + # (gh api ... issues/comments). With GITHUB_TOKEN those are gated by the + # `issues` permission, not `pull-requests` (#1802 CI fix). + issues: write + steps: + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + pattern: paid-slice-[0-9]* + path: /tmp/paid-report + merge-multiple: true + + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: report-verdict + path: /tmp/verdict + continue-on-error: true + + # Sourced from the slice artifacts' eval-store JSONs. Keeps the + # "## E2E Evals" marker so the upsert keeps updating the same comment. + # Runs even when reconciliation failed — a red lane on the PR is the point. + - name: Post PR comment + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + RECONCILE_EXIT: ${{ needs.slices-report.outputs.reconcile-exit }} + run: | + # shellcheck disable=SC2086,SC2059 + RESULTS=$(find /tmp/paid-report -name '*.json' ! -name 'manifest.json' ! -name 'slice-*.json' ! -name '_partial*' 2>/dev/null | sort) + TOTAL=0; PASSED=0; FAILED=0; FLAKY=0; COST="0" + SUITE_LINES="" + for f in $RESULTS; do + if ! jq -e '.total_tests' "$f" >/dev/null 2>&1; then + echo "Skipping malformed JSON: $f" + continue + fi + # FINAL-attempt accounting: eval-store keeps EVERY retry attempt + # as its own record (that's the flake telemetry), so counting raw + # records marks a pass-on-retry as a failure and inflates totals. + # Group by test name, judge the LAST record — flaky passes render + # as the ⚠ line, never as ❌ (WS1 policy: recorded, not blocking). + # Guarded: a file with total_tests but a null/non-array `tests` + # passes the -e probe, the group_by then fails, and an empty $T + # would abort the whole step under bash -e ([ "" -eq 0 ] is an + # error) — killing the comment on exactly the corrupted-artifact + # runs where the red evidence matters (claude adversarial). + STATS=$(jq -r '[.tests | group_by(.name)[] | last] as $final | "\($final | length) \([$final[] | select(.passed)] | length) \([$final[] | select(.passed | not)] | length) \(.flaky_retries // [] | length)"' "$f" 2>/dev/null) || { echo "Skipping malformed tests[] in: $f"; continue; } + read -r T P F FL <<< "$STATS" + [ -z "$T" ] && { echo "Skipping malformed tests[] in: $f"; continue; } + C=$(jq -r '.total_cost_usd // 0' "$f") + TIER=$(jq -r '.tier // "unknown"' "$f") + SHARD=$(jq -r '.shard // "-"' "$f") + [ "$T" -eq 0 ] && continue + TOTAL=$((TOTAL + T)) + PASSED=$((PASSED + P)) + FAILED=$((FAILED + F)) + FLAKY=$((FLAKY + FL)) + COST=$(echo "$COST + $C" | bc) + STATUS_ICON="✅" + [ "$F" -gt 0 ] && STATUS_ICON="❌" + [ "$F" -eq 0 ] && [ "$FL" -gt 0 ] && STATUS_ICON="✅⚠" + SUITE_LINES="${SUITE_LINES}| ${TIER}/${SHARD} | ${P}/${T} | ${STATUS_ICON} | \$${C} |\n" + done + + STATUS="✅ PASS" + if [ "${RECONCILE_EXIT:-1}" != "0" ] || [ "$FAILED" -gt 0 ]; then STATUS="❌ FAIL"; fi + + BODY="## E2E Evals: ${STATUS} + + **${PASSED}/${TOTAL}** tests passed | **\$${COST}** total cost | reconcile exit: ${RECONCILE_EXIT:-missing}$([ "$FLAKY" -gt 0 ] && printf ' | ⚠ %s flaky pass(es) — recorded, not blocking' "$FLAKY") + + | Shard | Result | Status | Cost | + |-------|--------|--------|------| + $(echo -e "$SUITE_LINES") + +
Fail-closed reconciliation + + \`\`\` + $(tail -c 4000 /tmp/verdict/report.txt 2>/dev/null || echo '(no reconciliation output)') + \`\`\` +
+ + --- + *Sliced lane: diff-selected gate census via scripts/test-paid-shards.ts (planner → 6 executors → fail-closed report)*" + + if [ "$FAILED" -gt 0 ]; then + FAILURES="" + for f in $RESULTS; do + if ! jq -e '.failed' "$f" >/dev/null 2>&1; then continue; fi + FAILS=$(jq -r '[.tests | group_by(.name)[] | last | select(.passed == false)][] | "- ❌ \(.name): \(.exit_reason // "unknown")"' "$f" 2>/dev/null || echo "- ⚠️ parse error") + FAILURES="${FAILURES}${FAILS}\n" + done + BODY="${BODY} + + ### Failures + $(echo -e "$FAILURES")" + fi + + COMMENT_ID=$(gh api repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments \ + --jq '.[] | select(.body | startswith("## E2E Evals")) | .id' | tail -1) + + if [ -n "$COMMENT_ID" ]; then + gh api "repos/${{ github.repository }}/issues/comments/${COMMENT_ID}" \ + -X PATCH -f body="$BODY" + else + # REST, not gh's pr-comment subcommand: this job runs with NO + # checkout (the token/exec split), and that subcommand resolves + # the repo FROM git — it dies with "not a git repository" here. + gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \ + -X POST -f body="$BODY" + fi diff --git a/.github/workflows/free-tests.yml b/.github/workflows/free-tests.yml index 76b579d27..3dcfc1ad5 100644 --- a/.github/workflows/free-tests.yml +++ b/.github/workflows/free-tests.yml @@ -48,7 +48,7 @@ jobs: runs-on: ubicloud-standard-8 timeout-minutes: 20 steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false @@ -129,6 +129,27 @@ jobs: run: xvfb-run -a bun run test:free env: GSTACK_EXPECT_BINARIES: "1" + # WS1 flake telemetry: a single timing flake must not red the only + # required lane — the runner's attribution-gated retry pass (cap 5, + # truncation veto) re-runs failing files once, serially, and a + # clean retry downgrades to a LOUD flaky-pass. Every flaky-pass is + # appended to the ledger (single writer: the parent runner) and + # uploaded below, so repeat offenders are an enumerable series — + # recorded and ranked, never masked. Pinned by + # test/free-tests-workflow-wiring.test.ts. + GSTACK_FREE_RETRY_FLAKY: "1" + GSTACK_FLAKE_LEDGER: ${{ runner.temp }}/flake-ledger.jsonl + + # Uploaded unconditionally (not just on failure): a flaky-pass run is + # GREEN — that's the point — so its evidence must survive green runs. + - name: Upload flake ledger + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: flake-ledger + path: ${{ runner.temp }}/flake-ledger.jsonl + if-no-files-found: ignore + retention-days: 90 # The runner streams the full child output to per-run logs under the OS # tmpdir and prints only the quiet contract to the console. Without this @@ -137,7 +158,7 @@ jobs: # need a local re-run, which fork contributors can't do on this image. - name: Upload shard logs on failure if: failure() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: free-test-shard-logs path: /tmp/gstack-free-test-*.log diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ddadc72bc..3c52efce8 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -424,7 +424,7 @@ The `EvalCollector` accumulates test results and writes them in two ways: 1. **Incremental:** `savePartial()` writes `_partial-e2e.json` after each test (atomic: write `.tmp`, `fs.renameSync`). Survives kills. 2. **Final:** `finalize()` writes a timestamped eval file (e.g. `e2e-20260314-143022.json`). The partial file is never cleaned up — it persists alongside the final file for observability. -`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack/projects//evals/` (legacy fallback `~/.gstack-dev/evals/`). Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `/shards//` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards//` subdirectories. Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline. +`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack/projects//evals/` (legacy fallback `~/.gstack-dev/evals/`). Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `/shards//` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards//` subdirectories (`eval:flake-rank` reads the same tree recursively, plus the free-suite flake ledger). Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline. ### Test tiers diff --git a/CHANGELOG.md b/CHANGELOG.md index 7380116dc..082ee62dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,61 @@ # Changelog +## [1.77.0.0] - 2026-08-31 + +**Every PR stops paying for evals twice.** +**Flakes are now measured, killed at the root, and fenced.** + +The test infrastructure got its overhaul, wave 1. The legacy 17-row eval matrix that ran serialized AHEAD of the sliced lane on every PR is deleted: one paid lane, its gate census derived from the runner itself, so a new gate test is in the census the moment its file lands. No hand-enumerated rows to drift, and the drift already tried, a new matrix row landed on main mid-branch and the merge resolved to the derived census that covers it by construction. + +The flake war moved from anecdotes to instruments. Every retried pass is now recorded where it cannot hide (bun's own output shows a retry as a clean pass, we probed it), the free lane retries a failing file once, loudly, and appends every flaky pass to a per-project ledger uploaded from CI on green runs. `bun run eval:flake-rank` ranks the series. And the wedge class that hit main, a hung child under a blocking spawnSync that no in-process timeout can interrupt, is extinct: 499 timeout-less sync-spawn sites across 146 files (the branch tripwire's own count against main) swept to zero, with a ratcheted tripwire that failed its first real offender the day a timeout-less spawn arrived from a merge. + +### The numbers that matter + +Source: CI runs 33263204465 / 33262077256 (measured 2026-08-29), the repo census (`test/helpers/touchfiles-data.ts`), and the sweep tripwire (`test/spawnsync-timeout-tripwire.test.ts`). + +| Metric | Before | After | Δ | +|---|---|---|---| +| Paid lanes per PR | 2 (serialized) | 1 | eval wall 35.5 → ~13 min (target) | +| Measured duplicate spend per PR | ~$20.94 | $0 | matrix deleted | +| Gate census keys | 86 (6 phantoms) | 77, all provably alive | reverse invariant enforces | +| Sync spawns that can wedge a shard | 499 unbounded | 0 (8 reasoned exemptions) | tripwire-ratcheted | +| Local gate worst case | ~6.5 h (4×4 jobs) | ~3.3 h (8×2) | isolation landed first | +| Retried-pass visibility | invisible | recorded + ranked | `eval:flake-rank` | + +The phantom-census number is the quiet one that matters: six merge-blocking "tests" existed only as map keys. They are deleted, and a key without a living test now fails the free suite. + +What this means for anyone shipping here: PRs get one honest paid verdict faster and cheaper, a flaky pass never blocks your merge but never disappears either, and a test that hangs takes down thirty seconds, not a shard. Run `bun run eval:flake-rank` when you want the flake ledger's verdict. + +### Itemized changes + +#### Added +- **Flake telemetry, end to end.** Eval-store records every retry attempt (`attempt`, `flaky_retries`), the paid report lists passed-only-on-retry tests as warnings, the free lane's flaky-retry pass is ON in CI with a single-writer JSONL ledger (branch + sha attributed, per-project by default) uploaded as an artifact on every run, and `bun run eval:flake-rank` aggregates the series with final-attempt accounting and a 60-day recency bound. +- **Two-phase session timeouts.** A silent API dies at a startup grace (90s local, 300s CI floor, a real `Math.max` floor) with the distinct reason `timeout_startup` instead of burning a 600s work budget into an opaque `0 turns / $0.00` failure; the work budget arms on the first byte and the total wall never grows. +- **Green-by-skip census.** "Ran N tests" counts skips, so a codex/gemini file whose every test self-skipped used to read as coverage. The classifier now parses bun's skip/pass recap and the paid report labels an all-skipped pass "verified nothing." +- **Sync-spawn timeout tripwire** (spawnSync / execSync / execFileSync / Bun.spawnSync, comment-aware, 30-line window, shrink-only exemption ratchet) plus a raw-SHA fixture ban (`git show :path` fixtures must be vendored; the one live offender now reads a committed fixture). +- **Behavioral kill-semantics tests**: a fake-claude shim proves a timed-out session leaves neither the CLI nor its grandchild alive; two gstack-detach watchdog tests cover TERM-immune grandchildren and the leader-dies-first case. +- **CLI version stamping**: every eval run records `claude --version` (resolved once in the runner parent), so the next TUI-drift flake hunt is a grep, not archaeology. + +#### Changed +- **One paid lane.** The legacy evals.yml matrix is deleted (pure deletion, one revert restores it) after a static parity receipt: the sliced lane's 49-file derived census strictly contained the matrix's 18 files. The PR comment moved into the sliced lane with final-attempt accounting and a fail-closed reconciliation verdict. +- **Paid runner defaults 4×4 → 8×2**: ~10-13 real in-flight sessions (under the documented-safe 15) instead of ~4-6; per-shard TMPDIR/Chromium-profile isolation and a kill-path cleanup backstop landed first, deliberately. +- **CI setup deduplicated into four composite actions**; the register-skills composite carries the fail-fast dangling-symlink verification loop that only the deleted matrix copy had, so the surviving lanes inherit it. Rerun-safe, frozen-lockfile fallback, input-validated. +- **Supply chain pinned**: the claude CLI in the CI image is an exact version (bumps ride PRs that run the PTY gate, ending the weekly-latest drift that broke the harness three times), and every action in the secrets-bearing and image-publishing workflows is SHA-pinned. +- **Routing journeys lost their answer key**: the fixture no longer ships a prompt→skill lookup table, so a regressed skill description can actually fail the test again, at roughly half the previous per-journey cost (2 turns, [Skill, Read] only). +- Decided A/B experiments retired (auq-repetition-cut, preamble-script, the opus-47 single-run fanout comparison): one-shot questions answered months ago no longer re-run weekly as coin flips. `plan-ceo-review-expansion-energy` and `ios-qa-e2e` moved to the periodic tier with reasons. + +#### Fixed +- **A fail-open reconcile gate**: GitHub's default run-step shell has no pipefail, so the fail-closed report's exit was read from `tee` (always 0) in both paid lanes. Now `PIPESTATUS[0]`, pinned by a wiring test. +- **A write-token trust boundary**: the job that executes PR-authored code no longer holds the PR-comment write token; commenting moved to a job that runs zero repo code. +- **Provider-runner orphans**: timeouts kill the whole process group (claude, codex, gemini, and gstack-detach's watchdog with the group id captured at spawn), and the codex/gemini runners inherited the orphan-drain hardening only the claude copy had. The observed 600s-timeout-stretching-past-1400s class is gone, with a regression net. +- **Selection integrity**: 17 phantom selection keys deleted (a reverse invariant now requires every key to name a living test), gitignored `.agents/**` dep patterns that could never match a git diff replaced with their generators, and the codex/gemini local touchfile forks now derive from the canonical map. +- **A cross-shard SKILL.md race**: the opus-47 eval regenerated the live tree's skill files mid-run; it now renders into a scratch dir via `--out-dir`. + +#### For contributors +- `bun run eval:flake-rank` (with `--json`, `--dir`, `--since-days`) is the promotion-clock dial; the WS16 required-check decision reads it. +- The overhaul plan (16 workstreams, reviewed by CEO + eng passes with two cross-model outside voices) continues: budget-aware shard walls, PTY readiness events, free-suite splits, judge determinism, and required-check promotion are the next waves. + + ## [1.76.0.0] - 2026-08-31 **Ship's doc-sync now survives Conductor.** diff --git a/CLAUDE.md b/CLAUDE.md index 529199ddd..46f167c28 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,6 +22,7 @@ bun run dev:skill # watch mode: auto-regen + validate on change bun run eval:list # list all eval runs from ~/.gstack/projects//evals/ bun run eval:compare # compare two eval runs (auto-picks most recent) bun run eval:summary # aggregate stats across all eval runs +bun run eval:flake-rank # rank tests by flake signal (retried passes first; --json, --dir, --since-days) bun run slop # full slop-scan report (all files) bun run slop:diff # slop findings in files changed on this branch only ``` @@ -647,9 +648,9 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session floor enforced against the live shard census by test/eval-detach-timeout-floor.test.ts) are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard - process count (default 4); `EVALS_CONCURRENCY` is bun's --max-concurrency - WITHIN a shard (default 4) — they are deliberately separate knobs. `eval:list` / `eval:compare` / - `eval:summary` read the shard dirs too. Or call + process count (default 8); `EVALS_CONCURRENCY` is bun's --max-concurrency + WITHIN a shard (default 2) — they are deliberately separate knobs. `eval:list` / `eval:compare` / + `eval:summary` / `eval:flake-rank` read the shard dirs too. Or call `gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] -- ` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never pass keys in argv). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f81afc0b5..1f398de16 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -168,7 +168,9 @@ silent truncation can never report green. Pass `--verbose` to forward the full child stream; `--wall-timeout ` overrides the per-shard kill deadline. `GSTACK_FREE_JOBS=` overrides the shard count (digits only, loud on garbage), and `GSTACK_FREE_RETRY_FLAKY=1` opts into one serial retry pass for -syscall-supervised sandboxes (off by default — dev boxes should see flakes). +syscall-supervised sandboxes (off by default locally — dev boxes should see +flakes; the required CI free lane turns it on and uploads every flaky pass +in a JSONL ledger artifact that `bun run eval:flake-rank` folds in). Working in a cloud sandbox? Run `scripts/sandbox-doctor.sh` once per boot to make the suite run green (details in [docs/TESTING_INTERNALS.md](docs/TESTING_INTERNALS.md)). @@ -236,6 +238,7 @@ When E2E tests run, they produce machine-readable artifacts in `~/.gstack-dev/`: bun run eval:list # list all eval runs (turns, duration, cost per run) bun run eval:compare # compare two runs — shows per-test deltas + Takeaway commentary bun run eval:summary # aggregate stats + per-test efficiency averages across runs +bun run eval:flake-rank # rank tests by flake signal: retried passes first, then failure rate (--json, --dir, --since-days) ``` **Detached runs for agents and long suites.** When an agent (or you, for a run @@ -264,9 +267,9 @@ distinguishes failed vs timed-out vs never-started shards. The runner also selects by diff: shards untouched by your branch are reported as skipped-by-diff, with a selection banner naming the reason (`EVALS_ALL=1` forces everything). `EVALS_JOBS` sets how many shard processes run at once -(default 4); `EVALS_CONCURRENCY` is bun's concurrency WITHIN a shard — they -are deliberately separate knobs. `eval:list`, -`eval:compare`, and `eval:summary` are shard-aware. Humans running +(default 8); `EVALS_CONCURRENCY` is bun's concurrency WITHIN a shard +(default 2) — they are deliberately separate knobs. `eval:list`, +`eval:compare`, `eval:summary`, and `eval:flake-rank` are shard-aware. Humans running `bun run test:evals` foreground in their own terminal don't need this — Ctrl-C is intended there. diff --git a/TODOS.md b/TODOS.md index f82c82119..3accde79a 100644 --- a/TODOS.md +++ b/TODOS.md @@ -567,14 +567,15 @@ duration-packed free shards, the sharded paid runner as the CI engine coverage contract + gate census, eval-budget timeout tiers, and the coverage fill. Remaining, in rough priority order: -- **P1 — Delete the legacy evals.yml matrix after parity.** The sliced lane - runs alongside the 18-row matrix (`needs: evals`, so provider concurrency - never doubles). After 1-2 PR cycles of parity (compare executed-test sets: - intersection strict + the 8 KNOWN_MATRIX_GAPS files as expected additions; - stochastic outcomes informational), delete the matrix as a PURE-DELETION - commit (one revert restores it), drop the `needs: evals` edge, rewrite - test/evals-workflow-matrix.test.ts into a runner-wiring pin, and retire - KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET wholesale. Effort S. +- **DONE (v1.77.0.0 test-infra wave 1) — Delete the legacy evals.yml matrix after + parity.** Deleted as a pure-deletion commit (one revert restores it) after + a static parity receipt: sliced gate census (49 files) ⊇ matrix files (18), + 31 files of extra coverage. `needs: evals` edge dropped, PR comment moved + into slices-report, KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET retired, + test/evals-workflow-matrix.test.ts rewritten as + test/evals-workflow-wiring.test.ts. The register-skills fail-fast + verification loop was ported to the surviving lanes FIRST via the shared + .github/actions/register-gstack-skills composite. - **P1 — Maintainer decision: make `slices-report` a required check** once post-migration flake data exists (the Codex outside-voice's "green means green is not delivered while paid stays advisory" point — correct, and @@ -812,7 +813,18 @@ SKILL.md untouched). `bun test` is green again. ## Scope-gate follow-ups (filed via /plan-eng-review on the plan-mode auto-select-B change) -### P2: SDK eval budgets charge API-queue latency to the work budget — pick a structural fix +### DONE (v1.77.0.0) — SDK eval budgets charge API-queue latency to the work budget + +**Shipped shape:** the two-phase timer landed WITHOUT the codemod this entry +feared: the total wall stays <= timeout (work phase = remainder after first +byte), so every outer/inner bun-timeout relationship is untouched; a silent +API now dies EARLY at the startup grace (90s local / 300s CI floor, enforced +Math.max) with the distinct reason 'timeout_startup'. Option (b)'s 300s CI +floor is in (test/session-runner-startup-grace.test.ts pins it). The +budget-EXTENSION variant (work budget = full timeout from first byte, which +DOES need the tier/wall reshape) remains wave-2 scope in the overhaul plan. + +Original entry follows for context: **What:** `runSkillTest`'s single `setTimeout(timeout)` arms at spawn, so session startup AND the model's first-completion queue time are charged against the diff --git a/VERSION b/VERSION index 84e596762..6734beb17 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.76.0.0 +1.77.0.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index ccdd0618c..b73164ced 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.76.0.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.77.0.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-detach b/bin/gstack-detach index 101e86eee..1f27f9246 100755 --- a/bin/gstack-detach +++ b/bin/gstack-detach @@ -96,20 +96,35 @@ def child_run(args, log): proc = subprocess.Popen( cmd, stdout=f, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, start_new_session=True ) + # Capture the PGID AT SPAWN (== proc.pid: start_new_session makes the + # child a session/group leader). Resolving it later via + # os.getpgid(proc.pid) raises ESRCH once the leader exits — a leader + # that died on the SIGTERM while a TERM-immune grandchild survived + # left that grandchild alive forever (codex adversarial finding). + pgid = proc.pid if args.timeout and args.timeout > 0: try: code = proc.wait(timeout=args.timeout) except subprocess.TimeoutExpired: log_line(log, f"### gstack-detach WATCHDOG fired after {args.timeout}s — killing ### {_now()}") try: - os.killpg(os.getpgid(proc.pid), signal.SIGTERM) + os.killpg(pgid, signal.SIGTERM) except Exception: pass time.sleep(5) + # Group SIGKILL after the grace, not a direct-child kill: + # eval runs spawn claude/codex grandchildren that survive a + # proc.kill() and burn cores + API for hours (the observed + # 15-hour-orphan class). ESRCH here just means the group + # honored the SIGTERM. Uses the SAVED pgid so a dead leader + # cannot orphan its group. try: - proc.kill() + os.killpg(pgid, signal.SIGKILL) except Exception: - pass + try: + proc.kill() + except Exception: + pass code = "timeout" else: code = proc.wait() diff --git a/browse/test/build.test.ts b/browse/test/build.test.ts index 050f35764..38e3b1891 100644 --- a/browse/test/build.test.ts +++ b/browse/test/build.test.ts @@ -13,7 +13,7 @@ describe('build: server-node.mjs', () => { // Skip rather than fail so plain `bun test` without a prior build passes. return; } - expect(() => execSync(`node --check ${SERVER_NODE}`, { stdio: 'pipe' })).not.toThrow(); + expect(() => execSync(`node --check ${SERVER_NODE}`, { stdio: 'pipe', timeout: 30_000 })).not.toThrow(); }); test('does not inline @ngrok/ngrok (must be external)', () => { diff --git a/browse/test/bun-polyfill.test.ts b/browse/test/bun-polyfill.test.ts index 21ab985cb..7df28b0d4 100644 --- a/browse/test/bun-polyfill.test.ts +++ b/browse/test/bun-polyfill.test.ts @@ -27,7 +27,7 @@ describe('bun-polyfill', () => { const elapsed = Date.now() - start; console.log(elapsed >= 40 ? 'OK' : 'TOO_FAST'); })(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.stdout.toString().trim()).toBe('OK'); expect(result.exitCode).toBe(0); }); @@ -38,7 +38,7 @@ describe('bun-polyfill', () => { const r = Bun.spawnSync(['echo', 'hello'], { stdout: 'pipe' }); console.log(r.stdout.toString().trim()); console.log('exit:' + r.exitCode); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const lines = result.stdout.toString().trim().split('\n'); expect(lines[0]).toBe('hello'); expect(lines[1]).toBe('exit:0'); @@ -51,7 +51,7 @@ describe('bun-polyfill', () => { console.log(typeof p.pid === 'number' ? 'HAS_PID' : 'NO_PID'); console.log(typeof p.kill === 'function' ? 'HAS_KILL' : 'NO_KILL'); console.log(typeof p.unref === 'function' ? 'HAS_UNREF' : 'NO_UNREF'); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const lines = result.stdout.toString().trim().split('\n'); expect(lines[0]).toBe('HAS_PID'); expect(lines[1]).toBe('HAS_KILL'); @@ -70,7 +70,7 @@ describe('bun-polyfill', () => { console.log(typeof p.exited === 'object' && typeof p.exited.then === 'function' ? 'IS_PROMISE' : 'NOT_PROMISE'); console.log('exit:' + await p.exited); })(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const lines = result.stdout.toString().trim().split('\n'); expect(lines[0]).toBe('IS_PROMISE'); expect(lines[1]).toBe('exit:0'); @@ -83,7 +83,7 @@ describe('bun-polyfill', () => { const p = Bun.spawn(['node', '-e', 'process.exit(3)'], { stdio: ['ignore', 'ignore', 'ignore'] }); console.log('exit:' + await p.exited); })(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.stdout.toString().trim()).toBe('exit:3'); }); @@ -100,7 +100,7 @@ describe('bun-polyfill', () => { const out = await new Response(p.stdout).text(); console.log(out + ':' + code); })(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.stdout.toString().trim()).toBe('ready:0'); }); @@ -120,7 +120,7 @@ describe('bun-polyfill', () => { ]).catch(() => 'TIMEOUT'); console.log('exit:' + code); })(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); // Anything other than 'TIMEOUT' (and ideally a non-zero number) means the // lifecycle promise resolved on the spawn error. const out = result.stdout.toString().trim(); @@ -139,7 +139,7 @@ describe('bun-polyfill', () => { setTimeout(() => p.kill('SIGTERM'), 150); console.log('exit:' + await p.exited); })(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); // SIGTERM = 15 → 128 + 15 = 143. expect(result.stdout.toString().trim()).toBe('exit:143'); }); @@ -166,7 +166,7 @@ describe('bun-polyfill', () => { const out = await new Response(p.stdout).text(); console.log(out.length + ':' + code); })(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.stdout.toString().trim()).toBe('1024:0'); }); @@ -197,7 +197,7 @@ describe('bun-polyfill', () => { const out = await new Response(p.stdout).text(); console.log(out.length + ':' + code); })().catch((e) => { console.log('THREW:' + e.message); }); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.stdout.toString().trim()).toBe('1048576:0'); }, 15000); @@ -217,7 +217,7 @@ describe('bun-polyfill', () => { console.log(typeof server.stop === 'function' ? 'HAS_STOP' : 'NO_STOP'); console.log(typeof server.port === 'number' ? 'HAS_PORT' : 'NO_PORT'); server.stop(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const lines = result.stdout.toString().trim().split('\n'); expect(lines[0]).toBe('HAS_STOP'); expect(lines[1]).toBe('HAS_PORT'); @@ -237,7 +237,7 @@ describe('bun-polyfill', () => { require(${JSON.stringify(polyfillPath)}); Bun.spawn(['node', '-e', ''], { stdio: ['ignore', 'ignore', 'ignore'] }); console.log('windowsHide:' + seen.windowsHide); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.stdout.toString().trim()).toBe('windowsHide:true'); }); @@ -250,7 +250,7 @@ describe('bun-polyfill', () => { require(${JSON.stringify(polyfillPath)}); Bun.spawnSync(['node', '-e', '']); console.log('windowsHide:' + seen.windowsHide); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.stdout.toString().trim()).toBe('windowsHide:true'); }); @@ -263,7 +263,7 @@ describe('bun-polyfill', () => { require(${JSON.stringify(polyfillPath)}); Bun.spawn(['node', '-e', ''], { stdio: ['ignore', 'ignore', 'ignore'], windowsHide: false }); console.log('windowsHide:' + seen.windowsHide); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.stdout.toString().trim()).toBe('windowsHide:false'); }); }); diff --git a/browse/test/config.test.ts b/browse/test/config.test.ts index 09aa40d15..c20bad8f2 100644 --- a/browse/test/config.test.ts +++ b/browse/test/config.test.ts @@ -85,7 +85,7 @@ describe('config', () => { // it (the exact bug the fix removed) would skip the guard here. const tmpDir = path.join(os.tmpdir(), `browse-gitignored-repo-test-${Date.now()}`); fs.mkdirSync(tmpDir, { recursive: true }); - Bun.spawnSync(['git', 'init'], { cwd: tmpDir, stdout: 'ignore', stderr: 'ignore' }); + Bun.spawnSync(['git', 'init'], { cwd: tmpDir, stdout: 'ignore', stderr: 'ignore', timeout: 30_000 }); fs.writeFileSync(path.join(tmpDir, '.gitignore'), '.gstack/\n'); const config = resolveConfig({ BROWSE_STATE_FILE: path.join(tmpDir, '.gstack', 'browse.json') }); ensureStateDir(config); @@ -166,22 +166,22 @@ describe('config', () => { fs.mkdirSync(tmpDir, { recursive: true }); // Set up a real git repo - spawnSync('git', ['init', '-q'], { cwd: tmpDir }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir }); + spawnSync('git', ['init', '-q'], { cwd: tmpDir, timeout: 30_000 }); + spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, timeout: 30_000 }); + spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, timeout: 30_000 }); // Write a global excludes file that ignores .gstack/ const excludesFile = path.join(tmpDir, 'global-gitignore'); fs.writeFileSync(excludesFile, '.gstack/\n'); - spawnSync('git', ['config', 'core.excludesFile', excludesFile], { cwd: tmpDir }); + spawnSync('git', ['config', 'core.excludesFile', excludesFile], { cwd: tmpDir, timeout: 30_000 }); // .gitignore exists but does NOT contain .gstack/ fs.writeFileSync(path.join(tmpDir, '.gitignore'), 'node_modules/\n'); - spawnSync('git', ['add', '.gitignore'], { cwd: tmpDir }); - spawnSync('git', ['commit', '-qm', 'init'], { cwd: tmpDir }); + spawnSync('git', ['add', '.gitignore'], { cwd: tmpDir, timeout: 30_000 }); + spawnSync('git', ['commit', '-qm', 'init'], { cwd: tmpDir, timeout: 30_000 }); // Verify git knows .gstack/ is ignored - const check = spawnSync('git', ['check-ignore', '-q', '.gstack/'], { cwd: tmpDir }); + const check = spawnSync('git', ['check-ignore', '-q', '.gstack/'], { cwd: tmpDir, timeout: 30_000 }); expect(check.status).toBe(0); const config = resolveConfig({ BROWSE_STATE_FILE: path.join(tmpDir, '.gstack', 'browse.json') }); diff --git a/browse/test/file-permissions.test.ts b/browse/test/file-permissions.test.ts index 057d164fd..6a86e7f4d 100644 --- a/browse/test/file-permissions.test.ts +++ b/browse/test/file-permissions.test.ts @@ -87,7 +87,7 @@ describe('restrictDirectoryPermissions', () => { fs.mkdirSync(d); // System chmod, not fs.chmodSync: Bun masks the sticky bit off chmod/ // mkdir modes, so 0o1777 through the fs API lands as 0o777. - Bun.spawnSync(['chmod', '1777', d]); + Bun.spawnSync(['chmod', '1777', d], { timeout: 30_000 }); expect(fs.statSync(d).mode & 0o7777).toBe(0o1777); // fixture took restrictDirectoryPermissions(d); expect(fs.statSync(d).mode & 0o7777).toBe(0o1777); @@ -244,7 +244,7 @@ describe('mkdirSecure', () => { fs.mkdirSync(d); // System chmod: Bun's fs API masks the sticky bit off modes (see the // restrictDirectoryPermissions sticky-dir test). - Bun.spawnSync(['chmod', '1777', d]); + Bun.spawnSync(['chmod', '1777', d], { timeout: 30_000 }); expect(fs.statSync(d).mode & 0o7777).toBe(0o1777); // fixture took mkdirSecure(d); expect(fs.statSync(d).mode & 0o7777).toBe(0o1777); diff --git a/browse/test/findport.test.ts b/browse/test/findport.test.ts index 3255d240c..47c0182d8 100644 --- a/browse/test/findport.test.ts +++ b/browse/test/findport.test.ts @@ -160,7 +160,7 @@ describe('findPort / isPortAvailable', () => { } test(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const output = result.stdout.toString().trim(); // Confirms the polyfill's stop() is fire-and-forget — callers @@ -171,7 +171,7 @@ describe('findPort / isPortAvailable', () => { test('net.createServer approach does not have the race condition', async () => { // Prove the fix: net.createServer with proper async bind/close // releases the port cleanly - const result = Bun.spawnSync(['node', '-e', ` + const result = Bun.spawnSync(['node', '-e', ` // timeout in trailing options const net = require('net'); async function testFix() { @@ -205,7 +205,7 @@ describe('findPort / isPortAvailable', () => { } testFix(); - `], { stdout: 'pipe', stderr: 'pipe' }); + `], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const output = result.stdout.toString().trim(); expect(output).toBe('FIX_WORKS'); diff --git a/browse/test/gstack-config.test.ts b/browse/test/gstack-config.test.ts index 097e25f75..323eefad3 100644 --- a/browse/test/gstack-config.test.ts +++ b/browse/test/gstack-config.test.ts @@ -31,6 +31,7 @@ function run(args: string[] = [], extraEnv: Record = {}) { env, stdout: 'pipe', stderr: 'pipe', + timeout: 30_000, }); return { exitCode: result.exitCode, diff --git a/browse/test/gstack-update-check.test.ts b/browse/test/gstack-update-check.test.ts index 16bf7b851..6b78263f7 100644 --- a/browse/test/gstack-update-check.test.ts +++ b/browse/test/gstack-update-check.test.ts @@ -34,6 +34,7 @@ function run(extraEnv: Record = {}, args: string[] = []) { env, stdout: 'pipe', stderr: 'pipe', + timeout: 30_000, }); return { exitCode: result.exitCode, diff --git a/browse/test/temp-dirs.test.ts b/browse/test/temp-dirs.test.ts index c3ee14ffe..f534a2cb6 100644 --- a/browse/test/temp-dirs.test.ts +++ b/browse/test/temp-dirs.test.ts @@ -107,7 +107,7 @@ describe('untrustable TMPDIR values never widen the allowlist', () => { const r = Bun.spawnSync([ process.execPath, '-e', "import { TEMP_DIRS } from './browse/src/platform'; console.log(JSON.stringify(TEMP_DIRS));", - ], { env: { ...process.env, TMPDIR: tmpdir }, cwd: path.resolve(import.meta.dir, '..', '..') }); + ], { env: { ...process.env, TMPDIR: tmpdir }, cwd: path.resolve(import.meta.dir, '..', '..'), timeout: 30_000 }); return JSON.parse(r.stdout.toString().trim().split('\n').pop()!); }; diff --git a/browse/test/windows-spawn-hide.test.ts b/browse/test/windows-spawn-hide.test.ts index 512279446..10d538aae 100644 --- a/browse/test/windows-spawn-hide.test.ts +++ b/browse/test/windows-spawn-hide.test.ts @@ -70,6 +70,7 @@ describe('windowsHide on Windows-reachable spawns (#1835)', () => { const EXEMPT: Array<{ file: string; needle: string; reason: string }> = [ { file: 'domain-skill-commands.ts', + // tripwire-exempt: grep NEEDLE string for this census, not a call needle: 'spawnSync(editor', reason: "interactive $EDITOR with stdio:'inherit' — windowsHide would detach a console editor into an invisible console", }, diff --git a/browse/test/xvfb.test.ts b/browse/test/xvfb.test.ts index dab242a8d..840aa43c5 100644 --- a/browse/test/xvfb.test.ts +++ b/browse/test/xvfb.test.ts @@ -11,7 +11,7 @@ import { const HAS_XVFB = (() => { if (process.platform !== 'linux') return false; - const result = Bun.spawnSync(['which', 'Xvfb'], { stdout: 'pipe', stderr: 'pipe' }); + const result = Bun.spawnSync(['which', 'Xvfb'], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); return result.exitCode === 0; })(); diff --git a/docs/PROJECT_STRUCTURE.md b/docs/PROJECT_STRUCTURE.md index 49954c165..6d043e7dc 100644 --- a/docs/PROJECT_STRUCTURE.md +++ b/docs/PROJECT_STRUCTURE.md @@ -28,6 +28,7 @@ gstack/ │ ├── skill-check.ts # Health dashboard │ ├── test-free-shards.ts # Strict parallel free-suite runner (GSTACK_FREE_JOBS, opt-in flaky retry) │ ├── test-paid-shards.ts # Sharded paid-tier runner (one Bun process per shard) +│ ├── eval-flake-rank.ts # Flake-telemetry dial: ranks tests by retried passes across eval runs + the free-lane ledger │ ├── sandbox-doctor.sh # One-command cloud-sandbox fixer: makes the free suite run green │ └── dev-skill.ts # Watch mode ├── test/ # Skill validation + eval tests @@ -71,7 +72,7 @@ gstack/ ├── patches/ # bun `patchedDependencies` patches (playwright-core windowsHide) ├── docs/designs/ # Design documents ├── setup-deploy/ # /setup-deploy skill (one-time deploy config) -├── .github/ # CI workflows + Docker image +├── .github/ # CI workflows + shared composite actions (.github/actions/) + Docker image (claude CLI pinned) │ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, and 8 more (windows, periodic evals, release gates, ci-image) │ └── docker/ # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium) ├── contrib/ # Contributor-only tools (never installed for users) diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 97d756dff..dbab9d8b4 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -68,6 +68,18 @@ failure), never-started/timed-out taxonomy, and parent-computed diff selection propagated to children via `EVALS_SELECTION_JSON` (fail-open: a child that can't parse it recomputes locally with one warning). Retry parity lives in `RETRY_OVERRIDES` (literals; old matrix rows' earned `retries: 2`). +Flake telemetry rides the store: every recorded test carries its 1-based +`attempt` (a pass-on-attempt-2 stays visible forever — bun's own stream hides +it), runs list `flaky_retries`, the report warns on passed-only-on-retry +tests, and `bun run eval:flake-rank` ranks the series (retried passes first, +then failure rate; 60-day recency bound on eval files; the free lane's flake +ledger is folded in from `flakeLedgerPath()` — override with +`GSTACK_FLAKE_LEDGER`, the same env var the CI free lane sets before +uploading the ledger as the `flake-ledger` artifact). Census integrity is +enforced from the free suite: every `E2E_TOUCHFILES` / `LLM_JUDGE_TOUCHFILES` +key must name a living paid test (`test/touchfiles.test.ts`'s reverse +invariant), and `git show :path` fixtures are banned — vendor the bytes +instead (`test/git-ref-fixture-tripwire.test.ts`). **CI planner/executor/report.** `--emit-plan --slices K` computes selection + the slice plan ONCE (killing per-slice selector divergence); @@ -76,19 +88,39 @@ slice-result artifacts; `--report ` reconciles them FAIL-CLOSED (a slice whose artifact never landed, or a planned shard nobody reported, is a failure). Under `EVALS_ALL` the hollow-shard guard marks exit-0 shards with ZERO executed tests `passed-empty` (a failure) — census-health, not just -test runs. evals.yml runs the sliced gate lane per PR (parity phase: -alongside the legacy matrix, `needs:`-sequenced so provider concurrency -never doubles; the matrix and its `KNOWN_MATRIX_GAPS`/`KNOWN_TIER_UNSET` -ratchets are deleted after demonstrated parity). evals-periodic.yml runs ALL +test runs. evals.yml runs the sliced gate lane per PR — the ONLY paid lane +since the legacy 17-row matrix (22.6 min/$21 per PR serialized ahead of the +slices) was deleted after demonstrated parity; its +`KNOWN_MATRIX_GAPS`/`KNOWN_TIER_UNSET` ratchets retired with it and +`test/evals-workflow-wiring.test.ts` pins the surviving wiring (slice-count +agreement, tier consistency, the shared register-skills composite with its +fail-fast verification loop). evals-periodic.yml runs ALL periodic-tier files weekly (the coverage contract) minus the reasoned exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking required per entry; removal re-activates the file), plus a weekly -`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. +`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. The CI +image pins the claude CLI to an exact version (`.github/docker/Dockerfile.ci`, +enforced by `test/ci-image-cli-pin.test.ts` — bumps ride PRs that run the PTY +gate), and every eval-store run records `claude --version`, resolved once in +the runner parent and handed to shard children as `GSTACK_CLAUDE_CLI_VERSION` +(never spawned on a test thread), so a TUI-drift flake hunt is a grep, not +archaeology. **Timeout policy.** Paid tests use the tiers in `test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG); `test/eval-budgets-policy.test.ts` pins that every tier fits the shard wall minus overhead and ratchets raw literals. Budget above the wall is fiction. +Session timeouts are two-phase: a silent API dies at the startup grace (90s +local / 300s CI floor, distinct exit reason `timeout_startup`) and the work +budget arms on the first byte — the total wall never grows +(`test/session-runner-startup-grace.test.ts` pins the floor). A timed-out +session kills its whole detached process group (claude, codex, and gemini +runners alike — `test/session-runner-groupkill.test.ts`), so a stray +grandchild can't stretch a 600s budget past 1400s. And sync spawns can't +wedge a shard: every `spawnSync`/`execSync`/`execFileSync`/`Bun.spawnSync` +in the test trees must carry a `timeout`, enforced by +`test/spawnsync-timeout-tripwire.test.ts` with a shrink-only exemption +ratchet. ## Cloud sandboxes (Vercel / Conductor cloud workspaces) @@ -110,4 +142,7 @@ Two runner knobs exist for these environments (both no-ops unless set): serial mega-shard and 6-way sharding both saturate the per-process syscall supervisor), and `GSTACK_FREE_RETRY_FLAKY=1` re-runs attributed failures once serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so -a broken tree can't masquerade as flaky). +a broken tree can't masquerade as flaky). The required CI free lane sets the +retry knob too, appending every flaky pass to the JSONL ledger it uploads +(`GSTACK_FLAKE_LEDGER`) — a flaky pass never reds the lane, but it never +disappears either. diff --git a/ios-qa/daemon/test/cli-mint.test.ts b/ios-qa/daemon/test/cli-mint.test.ts index f5416bf88..53c07d012 100644 --- a/ios-qa/daemon/test/cli-mint.test.ts +++ b/ios-qa/daemon/test/cli-mint.test.ts @@ -14,7 +14,7 @@ const MINT_BIN = join(ROOT, 'bin', 'gstack-ios-qa-mint'); const DAEMON_BIN = join(ROOT, 'bin', 'gstack-ios-qa-daemon'); function runMint(args: string[]) { - return spawnSync(MINT_BIN, args, { stdio: 'pipe', encoding: 'utf-8' }); + return spawnSync(MINT_BIN, args, { stdio: 'pipe', encoding: 'utf-8', timeout: 30_000 }); } describe('bin/gstack-ios-qa-mint launcher', () => { @@ -112,6 +112,7 @@ describe('bin/gstack-ios-qa-daemon launcher', () => { stdio: 'pipe', encoding: 'utf-8', env: { PATH: '/usr/bin:/bin' }, + timeout: 30_000, }); expect(r.status).not.toBe(0); expect(r.stderr).toContain('bun'); diff --git a/ios-qa/scripts/gen-accessors.test.ts b/ios-qa/scripts/gen-accessors.test.ts index 0c5df37d5..1ed68e653 100644 --- a/ios-qa/scripts/gen-accessors.test.ts +++ b/ios-qa/scripts/gen-accessors.test.ts @@ -612,6 +612,7 @@ class AppState { '--output', outputDir, ], { encoding: 'utf8', + timeout: 30_000, env: { ...process.env, GSTACK_IOS_CACHE_ROOT: join(workDir, 'cache') }, }); expect(result.status).toBe(4); @@ -717,7 +718,7 @@ describe('render', () => { }); test('typechecks beside an internal @Observable app state using a comment marker', () => { - if (spawnSync('swiftc', ['--version'], { encoding: 'utf8' }).status !== 0) return; + if (spawnSync('swiftc', ['--version'], { encoding: 'utf8', timeout: 30_000 }).status !== 0) return; const coreSource = join(workDir, 'DebugBridgeCore.swift'); const coreModule = join(workDir, 'DebugBridgeCore.swiftmodule'); @@ -752,7 +753,7 @@ public final class StateServer { '-module-name', 'DebugBridgeCore', coreSource, '-emit-module-path', coreModule, - ], { encoding: 'utf8' }); + ], { encoding: 'utf8', timeout: 120_000 }); if (emitModule.status !== 0) { throw new Error(`failed to build DebugBridgeCore test stub:\n${emitModule.stderr}`); } @@ -775,7 +776,7 @@ ${render([{ '-D', 'DEBUG', '-I', workDir, appSource, - ], { encoding: 'utf8' }); + ], { encoding: 'utf8', timeout: 120_000 }); if (typecheck.status !== 0) { throw new Error(`generated accessor failed Swift type checking:\n${typecheck.stderr}`); } @@ -783,7 +784,7 @@ ${render([{ test('strict JSON typing and cross-model validate-before-apply restore run correctly', () => { if (process.platform !== 'darwin') return; - if (spawnSync('swiftc', ['--version'], { encoding: 'utf8' }).status !== 0) return; + if (spawnSync('swiftc', ['--version'], { encoding: 'utf8', timeout: 30_000 }).status !== 0) return; const coreSource = join(workDir, 'DebugBridgeCore.swift'); const coreModule = join(workDir, 'DebugBridgeCore.swiftmodule'); @@ -830,7 +831,7 @@ public final class StateServer { '-module-name', 'DebugBridgeCore', coreSource, '-emit-module-path', coreModule, '-o', coreLibrary, - ], { encoding: 'utf8' }); + ], { encoding: 'utf8', timeout: 120_000 }); if (emitCore.status !== 0) throw new Error(`failed to build runtime stub:\n${emitCore.stderr}`); const appSource = join(workDir, 'OptionalRoundTrip.swift'); @@ -915,10 +916,11 @@ struct Runner { const compile = spawnSync('swiftc', [ '-D', 'DEBUG', '-I', workDir, '-L', workDir, '-lDebugBridgeCore', '-parse-as-library', appSource, '-o', executable, - ], { encoding: 'utf8' }); + ], { encoding: 'utf8', timeout: 120_000 }); if (compile.status !== 0) throw new Error(`generated Optional accessor failed compilation:\n${compile.stderr}`); const run = spawnSync(executable, [], { encoding: 'utf8', + timeout: 30_000, env: { ...process.env, DYLD_LIBRARY_PATH: workDir }, }); if (run.status !== 0) throw new Error(`generated Optional accessor failed at runtime:\n${run.stderr}`); @@ -928,7 +930,7 @@ struct Runner { describe('SwiftSyntax generator parity', () => { test('isolates canonical markers and rejects inaccessible fields', () => { if (process.platform !== 'darwin') return; - if (spawnSync('swift', ['--version'], { encoding: 'utf8' }).status !== 0) return; + if (spawnSync('swift', ['--version'], { encoding: 'utf8', timeout: 30_000 }).status !== 0) return; const packageDir = join(import.meta.dir, 'gen-accessors-tool'); const inputDir = join(workDir, 'swift-syntax-input'); diff --git a/ios-qa/scripts/gen-accessors.ts b/ios-qa/scripts/gen-accessors.ts index 738e08364..315f2bde6 100644 --- a/ios-qa/scripts/gen-accessors.ts +++ b/ios-qa/scripts/gen-accessors.ts @@ -739,7 +739,7 @@ export function render(specs: AccessorSpec[], buildId: string, accessorHash: str function detectSwiftVersion(): string { if (process.env.SWIFT_VERSION) return process.env.SWIFT_VERSION; try { - const out = execSync('swift --version', { stdio: ['ignore', 'pipe', 'ignore'] }).toString(); + const out = execSync('swift --version', { stdio: ['ignore', 'pipe', 'ignore'], timeout: 30_000 }).toString(); const m = out.match(/Apple Swift version (\d+\.\d+\.\d+)/); if (m) return m[1]!; } catch { @@ -754,6 +754,7 @@ function detectToolGitRev(): string { return execSync('git rev-parse --short HEAD', { cwd: dirname(new URL(import.meta.url).pathname), stdio: ['ignore', 'pipe', 'ignore'], + timeout: 30_000, }).toString().trim(); } catch { return 'dev'; diff --git a/make-pdf/test/e2e/combined-gate.test.ts b/make-pdf/test/e2e/combined-gate.test.ts index 562d46e49..fcaff4265 100644 --- a/make-pdf/test/e2e/combined-gate.test.ts +++ b/make-pdf/test/e2e/combined-gate.test.ts @@ -50,6 +50,7 @@ describe("combined-features copy-paste gate", () => { try { execFileSync(PDF_BIN, ["generate", FIXTURE, outputPdf, "--quiet"], { encoding: "utf8", + timeout: 30_000, env: { ...process.env, BROWSE_BIN }, stdio: ["ignore", "pipe", "pipe"], }); diff --git a/make-pdf/test/e2e/diagram-gate.test.ts b/make-pdf/test/e2e/diagram-gate.test.ts index a3473592a..37d7ffdf0 100644 --- a/make-pdf/test/e2e/diagram-gate.test.ts +++ b/make-pdf/test/e2e/diagram-gate.test.ts @@ -83,6 +83,7 @@ describe("diagram render gate", () => { env: { ...process.env, BROWSE_BIN }, stdout: "pipe", stderr: "pipe", + timeout: 120_000, }); const stderr = new TextDecoder().decode(run.stderr); if (run.exitCode !== 0) { diff --git a/package.json b/package.json index e8b1d28cd..863c5d659 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.76.0", + "version": "1.77.0", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", @@ -44,6 +44,7 @@ "eval:list": "bun run scripts/eval-list.ts", "eval:compare": "bun run scripts/eval-compare.ts", "eval:summary": "bun run scripts/eval-summary.ts", + "eval:flake-rank": "bun run scripts/eval-flake-rank.ts", "eval:watch": "bun run scripts/eval-watch.ts", "eval:select": "bun run scripts/eval-select.ts", "analytics": "bun run scripts/analytics.ts", diff --git a/scripts/eval-flake-rank.ts b/scripts/eval-flake-rank.ts new file mode 100644 index 000000000..b6698ae48 --- /dev/null +++ b/scripts/eval-flake-rank.ts @@ -0,0 +1,144 @@ +#!/usr/bin/env bun +/** + * eval-flake-rank — the flake-telemetry dial (WS1). + * + * Aggregates per-test series across every FINALIZED eval-store run on this + * machine (default: ~/.gstack/projects//evals/, shard dirs included) + * plus the free suite's flake ledger, and ranks tests by flake signal: + * retried passes first (a test that needs attempt 2 to go green is the + * definition of a flake), then failure rate. + * + * This is the readable dial behind two policies: + * - a flaky pass never blocks a merge, but it is recorded and RANKED here; + * - the required-check promotion (WS16) needs weeks of clean flake-rank, + * not vibes. + * + * Usage: + * bun run eval:flake-rank # project eval dir + * bun run eval:flake-rank --dir # e.g. downloaded CI artifacts + * bun run eval:flake-rank --json # machine-readable + */ + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { getProjectEvalDir, isPartialEval, isFinalizedEvalResultFile, type EvalResult } from '../test/helpers/eval-store'; +import { flakeLedgerPath, type FlakeLedgerEntry } from './test-free-shards'; + +interface TestSeries { + name: string; + runs: number; + passes: number; + fails: number; + retriedPasses: number; + totalAttempts: number; + totalCostUsd: number; + totalDurationMs: number; + lastSeen: string; +} + +export function aggregate(evalFiles: string[]): Map { + const series = new Map(); + for (const file of evalFiles) { + let run: EvalResult; + try { + run = JSON.parse(fs.readFileSync(file, 'utf-8')); + } catch { continue; } + if (isPartialEval(run, file)) continue; // in-progress accumulators are not runs + if (!Array.isArray(run.tests)) continue; + // Group this run's entries by name so N attempts = 1 run of that test. + const byName = new Map(); + for (const t of run.tests) { + const list = byName.get(t.name) ?? []; + list.push(t); + byName.set(t.name, list); + } + for (const [name, entries] of byName) { + const s = series.get(name) ?? { + name, runs: 0, passes: 0, fails: 0, retriedPasses: 0, + totalAttempts: 0, totalCostUsd: 0, totalDurationMs: 0, lastSeen: '', + }; + const final = entries[entries.length - 1]; + s.runs += 1; + s.totalAttempts += entries.length; + if (final.passed) s.passes += 1; else s.fails += 1; + if (final.passed && entries.length > 1) s.retriedPasses += 1; + for (const e of entries) { + s.totalCostUsd += e.cost_usd || 0; + s.totalDurationMs += e.duration_ms || 0; + } + if (run.timestamp > s.lastSeen) s.lastSeen = run.timestamp; + series.set(name, s); + } + } + return series; +} + +export function collectEvalFiles(dir: string, sinceDays = 60): string[] { + if (!fs.existsSync(dir)) return []; + const cutoff = Date.now() - sinceDays * 86_400_000; + const out: string[] = []; + for (const name of fs.readdirSync(dir, { recursive: true }) as string[]) { + if (!isFinalizedEvalResultFile(name)) continue; + const full = path.join(dir, name); + try { + // Recency bound (review finding): E2E results embed full transcripts + // (MBs each) and the scan is otherwise unbounded over all-time history. + if (fs.statSync(full).mtimeMs < cutoff) continue; + } catch { continue; } + out.push(full); + } + return out; +} + +function readFreeLedger(): FlakeLedgerEntry[] { + // Per-LINE parse: one malformed JSONL line (torn write, manual edit) must + // drop that line, never vanish the whole series (codex adversarial finding). + let raw: string; + try { + raw = fs.readFileSync(flakeLedgerPath(), 'utf-8'); + } catch { return []; } + const out: FlakeLedgerEntry[] = []; + for (const line of raw.split('\n')) { + if (!line.trim()) continue; + try { out.push(JSON.parse(line)); } catch { /* torn line — skip */ } + } + return out; +} + +if (import.meta.main) { + const argv = process.argv.slice(2); + const dirFlag = argv.indexOf('--dir'); + const dir = dirFlag !== -1 ? argv[dirFlag + 1] : getProjectEvalDir(); + const asJson = argv.includes('--json'); + const sinceFlag = argv.indexOf('--since-days'); + const sinceDays = sinceFlag !== -1 ? Number(argv[sinceFlag + 1]) || 60 : 60; + + const files = collectEvalFiles(dir, sinceDays); + const series = [...aggregate(files).values()] + .sort((a, b) => b.retriedPasses - a.retriedPasses || (b.fails / b.runs) - (a.fails / a.runs)); + const ledger = readFreeLedger(); + + if (asJson) { + console.log(JSON.stringify({ dir, runsScanned: files.length, tests: series, freeLedger: ledger }, null, 2)); + } else { + console.log(`flake-rank: ${files.length} finalized run file(s) under ${dir}`); + const flaky = series.filter((s) => s.retriedPasses > 0 || s.fails > 0); + if (flaky.length === 0) { + console.log(' no retried passes and no failures recorded — clean series'); + } else { + console.log(' retries fails/runs avg-dur test'); + for (const s of flaky.slice(0, 30)) { + console.log(` ${String(s.retriedPasses).padStart(7)} ${String(s.fails).padStart(5)}/${String(s.runs).padEnd(4)} ` + + `${Math.round(s.totalDurationMs / s.totalAttempts / 1000).toString().padStart(5)}s ${s.name}`); + } + } + if (ledger.length > 0) { + const byFile = new Map(); + for (const e of ledger) byFile.set(e.file, (byFile.get(e.file) ?? 0) + 1); + console.log(`free-suite flaky-passes (${flakeLedgerPath()}):`); + for (const [file, n] of [...byFile.entries()].sort((a, b) => b[1] - a[1])) { + console.log(` ${String(n).padStart(3)}x ${file}`); + } + } + } +} diff --git a/scripts/test-free-shards.ts b/scripts/test-free-shards.ts index 7fd8e6814..06aa2c215 100755 --- a/scripts/test-free-shards.ts +++ b/scripts/test-free-shards.ts @@ -1022,6 +1022,62 @@ export function buildRunEpilogue( export type FreeShardStatus = 'passed' | 'failed' | 'timed-out'; +// ─── Flake ledger (WS1 telemetry) ─────────────────────────────────────────── +// Single-writer JSONL: ONLY this parent runner appends (never shards, never +// tests — no concurrent-append hazard by construction). CI points +// GSTACK_FLAKE_LEDGER at $RUNNER_TEMP and uploads it as an artifact every +// run, so repeat offenders become an enumerable series instead of console +// scrollback. Fail-open with a loud stderr warning: a broken ledger must +// never red the only required lane. + +export interface FlakeLedgerEntry { + ts: string; + runner: 'free'; + kind: 'flaky-pass'; + file: string; + /** Shard the original failure surfaced in, when attributable. */ + shard?: number; + /** Code-state attribution (review finding): without branch/sha the series + * can't tie an entry to the state that produced it, and the WS16 + * promotion evidence needs exactly that. */ + branch?: string; + git_sha?: string; +} + +export function flakeLedgerPath(env: NodeJS.ProcessEnv = process.env): string { + if (env.GSTACK_FLAKE_LEDGER) return env.GSTACK_FLAKE_LEDGER; + // Local default: per-PROJECT, not the machine-global tmpdir — sibling + // Conductor worktrees of DIFFERENT repos must not interleave into one + // series (review finding). CI always sets GSTACK_FLAKE_LEDGER explicitly. + try { + const slug = spawnSync('bash', ['-c', '~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null'], { stdio: 'pipe', timeout: 3000 }) + .stdout?.toString().match(/^SLUG=(.+)$/m)?.[1]; + if (slug) { + const dir = path.join(os.homedir(), '.gstack', 'projects', slug); + fs.mkdirSync(dir, { recursive: true }); + return path.join(dir, 'flake-ledger.jsonl'); + } + } catch { /* fall through */ } + return path.join(os.tmpdir(), 'gstack-flake-ledger.jsonl'); +} + +export function appendFlakeLedger( + entries: FlakeLedgerEntry[], + ledgerPath: string, + warn: (line: string) => void = (line) => console.error(line), +): boolean { + if (entries.length === 0) return true; + try { + fs.mkdirSync(path.dirname(ledgerPath), { recursive: true }); + fs.appendFileSync(ledgerPath, entries.map((e) => JSON.stringify(e)).join('\n') + '\n'); + return true; + } catch (error) { + warn(`[test:free] WARNING: could not append flake ledger at ${ledgerPath} ` + + `(${error instanceof Error ? error.message : String(error)}) — flaky-pass telemetry lost for this run, verdict unaffected`); + return false; + } +} + export interface FreeShardOutcome { shard: number; files: string[]; @@ -1511,6 +1567,26 @@ async function main(): Promise { if (retryOutcome.status === 'passed') { console.log(`[test:free] FLAKY-PASS — ${flakyFiles.length} file(s) failed once and passed on serial retry: ${flakyFiles.join(', ')}`); console.log('[test:free] treat repeat offenders as real flakes worth fixing, not noise.'); + // Durable record (WS1): console lines vanish with the scrollback; the + // ledger makes repeat offenders rankable across runs (eval:flake-rank). + const ts = new Date().toISOString(); + // Two separate calls: `rev-parse --abbrev-ref HEAD HEAD` abbreviates + // BOTH revs, printing the branch twice — git_sha recorded the branch + // name (codex adversarial finding). + const ledgerBranch = (spawnSync('git', ['rev-parse', '--abbrev-ref', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5000 }).stdout ?? '').trim(); + const ledgerSha = (spawnSync('git', ['rev-parse', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5000 }).stdout ?? '').trim(); + appendFlakeLedger( + flakyFiles.map((file) => ({ + ts, + runner: 'free' as const, + kind: 'flaky-pass' as const, + file, + shard: outcomes.find((o) => o.failingFiles.includes(file))?.shard, + ...(ledgerBranch ? { branch: ledgerBranch } : {}), + ...(ledgerSha ? { git_sha: ledgerSha.slice(0, 12) } : {}), + })), + flakeLedgerPath(), + ); worst = 0; } else { console.error('[test:free] flaky-retry FAILED — the failures reproduce serially; not flaky.'); diff --git a/scripts/test-paid-shards.ts b/scripts/test-paid-shards.ts index b27fa22d8..cd274743f 100644 --- a/scripts/test-paid-shards.ts +++ b/scripts/test-paid-shards.ts @@ -64,7 +64,7 @@ import { } from './test-strict-output'; import { PAID_TEST_GLOBS, isPaidTestFile } from '../test/helpers/paid-test-set'; import { PERIODIC_CI_EXCLUDE } from '../test/helpers/periodic-exclude-data'; -import { getProjectEvalDir } from '../test/helpers/eval-store'; +import { getProjectEvalDir, getClaudeCliVersion, isFinalizedEvalResultFile } from '../test/helpers/eval-store'; import { preflightAnthropicApi } from '../test/helpers/anthropic-preflight'; import { detectBaseBranch, @@ -85,11 +85,16 @@ export type PaidTier = 'gate' | 'periodic'; export const DEFAULT_TIER: PaidTier = 'gate'; export const DEFAULT_SHARD_TIMEOUT_MS = 30 * 60_000; export const DEFAULT_MAX_FILES_PER_SHARD = 1; -export const DEFAULT_JOBS = 4; -// Within one shard's bun process. 4 jobs × 4 ≈ the legacy single-process -// default of 15, keeping total in-flight `claude` sessions inside known-safe -// API rate headroom. -export const DEFAULT_WITHIN_SHARD_CONCURRENCY = 4; +// 8 jobs × 2 within-shard ≈ 10-13 real in-flight sessions (39 of 75 +// skill-e2e files hold exactly ONE test, so within-shard concurrency is +// dead weight for most shards) — under the documented-safe ~15 the legacy +// 40-way runner established. The old 4×4 yielded only ~4-6 in-flight and a +// 13-wave local gate worst case (~6.5h); 8×2 halves it. Watch the WS1 +// flake telemetry for sustained 429 storms across 2 PR cycles — that is +// the rollback trigger. Prerequisite (landed): per-shard TMPDIR/ +// CHROMIUM_PROFILE isolation in runPaidShard. +export const DEFAULT_JOBS = 8; +export const DEFAULT_WITHIN_SHARD_CONCURRENCY = 2; export function collectPaidTestFiles(rootDir = ROOT): string[] { const testDir = path.join(rootDir, 'test'); @@ -393,6 +398,24 @@ export interface ShardOutcome { groupPid: number | null; /** Tests bun reported executing ("Ran N tests ..."), null when unknown. */ executedTests: number | null; + /** Tests bun reported skipping (" N skip" count line), null when unknown. + * "Ran N tests" COUNTS skips, so executedTests alone cannot distinguish a + * shard that verified work from one whose every test self-skipped — + * codex/gemini files green-by-skip on every CI runner (no binary) and the + * weekly census read them as covered. */ + skippedTests: number | null; +} + +/** + * True when a shard "passed" without verifying anything: every test bun ran + * was a skip. Legitimate for external-service files on hosts without the + * binary, but it must surface as a census warning, never read as coverage. + */ +export function isAllSkippedPass(outcome: Pick): boolean { + return outcome.status === 'passed' + && outcome.executedTests !== null + && outcome.executedTests > 0 + && outcome.skippedTests === outcome.executedTests; } export interface ShardCommand { @@ -475,6 +498,28 @@ export async function runPaidShard( if (options.evalDirBase) { env.GSTACK_EVAL_DIR = path.join(options.evalDirBase, 'shards', shardSlug(files)); } + // Resolve `claude --version` ONCE in the parent (cached across shards) and + // hand it to every child: eval-store's fallback is a synchronous spawn on + // the same thread that polls PTY sessions, so children must never pay it. + if (!env.GSTACK_CLAUDE_CLI_VERSION) { + env.GSTACK_CLAUDE_CLI_VERSION = getClaudeCliVersion(); + } + // Per-shard temp + Chromium-profile isolation — the free runner treats + // this as mandatory (test-free-shards.ts: two concurrent shards on one + // profile dir kill each other's browser; shared tmp cross-contaminates), + // and the paid lane had NONE of it. Doubly load-bearing here: when a + // shard hits its 30-min wall the group-SIGKILL means per-test afterAll + // cleanup never runs — the rmSync backstop below is the only thing + // stopping wedged runs from accumulating full git-repo workspaces in the + // shared tmpdir forever. Prerequisite for raising EVALS_JOBS (more + // concurrency on shared state amplifies exactly the opus-47 race class). + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-paid-shard-')); + const childTmp = path.join(stateDir, 'tmp'); + fs.mkdirSync(childTmp); + env.TMPDIR = childTmp; + env.TEMP = childTmp; + env.TMP = childTmp; + env.CHROMIUM_PROFILE = path.join(stateDir, 'chromium-profile'); const startedAt = Date.now(); log(`${label} START ${files.join(' ')} (timeout ${Math.round(timeoutMs / 1000)}s)`); @@ -529,6 +574,16 @@ export async function runPaidShard( } finally { // Close the spool even when the spawn itself failed. await new Promise((resolve) => logStream.end(() => resolve())); + try { + // async rm: a SIGKILLed shard can leave a full git workspace + Chromium + // profile here; a synchronous recursive delete on the parent's event + // loop would stall every sibling shard's stream classification and + // wall timers for seconds (review finding). + await fs.promises.rm(stateDir, { recursive: true, force: true }); + } catch { + // Best-effort: a locked file must not turn a real verdict into an + // exception (same posture as the free runner's cleanup). + } } const summary = classifier.end(); @@ -562,7 +617,8 @@ export async function runPaidShard( const executedTests = summary.terminalTestCounts.length > 0 ? summary.terminalTestCounts.reduce((a, b) => a + b, 0) : null; - return { shard: shardNumber, files, status, exitCode, elapsedMs, groupPid, executedTests }; + const skippedTests = summary.terminalTestCounts.length > 0 ? summary.skippedTests : null; + return { shard: shardNumber, files, status, exitCode, elapsedMs, groupPid, executedTests, skippedTests }; } export interface RunSummary { @@ -636,6 +692,7 @@ export async function runPaidShards( elapsedMs: 0, groupPid: null, executedTests: null, + skippedTests: null, })); let next = 0; @@ -659,6 +716,7 @@ export async function runPaidShards( elapsedMs: 0, groupPid: null, executedTests: null, + skippedTests: null, }; console.error(`[test:paid] shard ${index + 1} could not run: ${error instanceof Error ? error.message : String(error)}`); } @@ -678,9 +736,14 @@ export function formatSummary(summary: RunSummary): string[] { + `${summary.skippedByDiff} skipped by diff`, ]; for (const outcome of summary.outcomes) { + // A pass whose every test skipped is labeled distinctly: it exited 0 but + // verified NOTHING (codex/gemini files on hosts without the binary). + // Status stays 'passed' — availability of an external service is not a + // repo regression — but the census must never read it as coverage. + const allSkipped = isAllSkippedPass(outcome) ? ` ⚠ all ${outcome.executedTests} tests SKIPPED — verified nothing` : ''; lines.push( ` ${outcome.status.padEnd(15)} ${String(Math.round(outcome.elapsedMs / 1000)).padStart(5)}s ` - + outcome.files.join(' '), + + outcome.files.join(' ') + allSkipped, ); } return lines; @@ -786,7 +849,7 @@ export interface SliceResult { tier: PaidTier; sliceIndex: number; sliceCount: number; - outcomes: Array>; + outcomes: Array>; } /** @@ -970,6 +1033,35 @@ async function main(): Promise { for (const problem of verdict.problems) console.error(` ✗ ${problem}`); return 1; } + // Flake honesty (WS1): surface every test that needed a retry to pass. + // WARNS, never fails — a flaky pass must not block merges; it must also + // never be invisible (bun's own output hides retried passes entirely). + // Source: the finalized eval-store JSONs inside the slice artifacts. + const flaky: Array<{ name: string; attempts: number; file: string }> = []; + for (const name of fs.readdirSync(options.reportDir, { recursive: true }) as string[]) { + if (!isFinalizedEvalResultFile(name)) continue; + try { + const parsed = JSON.parse(fs.readFileSync(path.join(options.reportDir, name), 'utf-8')) as { flaky_retries?: Array<{ name: string; attempts: number }> }; + for (const f of parsed.flaky_retries ?? []) flaky.push({ ...f, file: name }); + } catch { /* non-eval JSON — not this report's business */ } + } + if (flaky.length > 0) { + console.log(`[test:paid] report: ⚠ ${flaky.length} test(s) passed only on retry this run (recorded, not blocking):`); + for (const f of flaky) console.log(` ⚠ ${f.name} (x${f.attempts}) — ${f.file}`); + } + + // Census honesty: a 'passed' shard whose every test skipped verified + // nothing (external-service binary absent on the runner). Not a failure — + // service availability is host state, not a repo regression — but the + // report must say so, or the weekly lane reads codex/gemini as covered + // on runners that never install them. + const allSkipped = results.flatMap((r) => r.outcomes.filter(isAllSkippedPass)); + if (allSkipped.length > 0) { + console.log(`[test:paid] report: ⚠ ${allSkipped.length} shard(s) passed with EVERY test skipped — they verified nothing:`); + for (const outcome of allSkipped) { + console.log(` ⚠ ${outcome.files.join(' ')} (${outcome.executedTests} skipped — external service missing or tier mismatch)`); + } + } console.log('[test:paid] report: every planned shard accounted and passed'); return 0; } @@ -1023,8 +1115,8 @@ async function main(): Promise { tier: manifest.tier, sliceIndex: options.sliceIndex, sliceCount: manifest.sliceCount, - outcomes: guarded.map(({ files, status, exitCode, elapsedMs, executedTests }) => - ({ files, status, exitCode, elapsedMs, executedTests })), + outcomes: guarded.map(({ files, status, exitCode, elapsedMs, executedTests, skippedTests }) => + ({ files, status, exitCode, elapsedMs, executedTests, skippedTests })), }; fs.mkdirSync(evalDirBase, { recursive: true }); const sliceResultPath = path.join(evalDirBase, `slice-${options.sliceIndex}.json`); @@ -1102,6 +1194,7 @@ async function main(): Promise { elapsedMs: 0, groupPid: null, executedTests: null, + skippedTests: null, })); const guardedOutcomes = applyHollowShardGuard(runSummary.outcomes, { evalsAll: process.env.EVALS_ALL === '1', diff --git a/scripts/test-strict-output.ts b/scripts/test-strict-output.ts index 2e91bd096..cfa41e63f 100644 --- a/scripts/test-strict-output.ts +++ b/scripts/test-strict-output.ts @@ -20,6 +20,15 @@ const ANSI_ESCAPE = /\u001B\[[0-?]*[ -/]*[@-~]/g; const BUN_FAIL_RESULT = /^\(fail\) .+ \[(?:\d+(?:\.\d+)?)(?:ns|us|µs|ms|s)\]$/; const BUN_BETWEEN_TESTS_ERROR = '# Unhandled error between tests'; const BUN_TERMINAL_SUMMARY = /^Ran (\d+) tests? across (\d+) files?\. \[(?:\d+(?:\.\d+)?)(?:ns|us|µs|ms|s)\]$/; +// The counts block bun prints just before the terminal summary (" 1 pass", +// " 2 skip", " 0 fail"). "Ran N tests" COUNTS skipped tests, so N alone +// cannot distinguish a shard that verified work from one whose every test +// self-skipped (external-service binary missing, tier mismatch) — the +// green-by-skip class. Anchored to whole-line matches; nested bun-test +// children can still contribute counts (same known limit as the terminal +// summary — see the last-summary-anchoring TODO in the audit). +const BUN_SKIP_COUNT = /^\s*(\d+) skip$/; +const BUN_PASS_COUNT = /^\s*(\d+) pass$/; export type BunTestOutputFinding = 'failed-test' | 'unhandled-between-tests'; @@ -29,6 +38,11 @@ export interface BunTestOutputSummary { terminalFileCounts: number[]; /** Test counts from the same terminal lines — feeds the hollow-shard guard. */ terminalTestCounts: number[]; + /** Sum of bun's " N skip" count lines. "Ran N tests" includes skips, so + * this is what separates verified work from green-by-skip. */ + skippedTests: number; + /** Sum of bun's " N pass" count lines. */ + passedTests: number; } export type ForwardedTerminationSignal = 'SIGINT' | 'SIGTERM'; @@ -229,6 +243,8 @@ export class BunTestOutputClassifier { private unhandledBetweenTests = 0; private terminalFileCounts: number[] = []; private terminalTestCounts: number[] = []; + private skippedTests = 0; + private passedTests = 0; write(chunk: Uint8Array | string, origin: ClassifierOrigin = 'stdout'): void { this.pending[origin] += typeof chunk === 'string' @@ -252,6 +268,8 @@ export class BunTestOutputClassifier { unhandledBetweenTests: this.unhandledBetweenTests, terminalFileCounts: [...this.terminalFileCounts], terminalTestCounts: [...this.terminalTestCounts], + skippedTests: this.skippedTests, + passedTests: this.passedTests, }; } @@ -268,6 +286,11 @@ export class BunTestOutputClassifier { const finding = classifyBunTestOutputLine(line); if (finding === 'failed-test') this.failedTests += 1; if (finding === 'unhandled-between-tests') this.unhandledBetweenTests += 1; + const stripped = stripAnsiLine(line); + const skip = BUN_SKIP_COUNT.exec(stripped); + if (skip !== null) this.skippedTests += Number.parseInt(skip[1], 10); + const pass = BUN_PASS_COUNT.exec(stripped); + if (pass !== null) this.passedTests += Number.parseInt(pass[1], 10); const terminal = parseBunTerminalSummary(line); if (terminal !== null) { this.terminalFileCounts.push(terminal.files); diff --git a/test/artifacts-init-migration.test.ts b/test/artifacts-init-migration.test.ts index c09affffd..206c7653f 100644 --- a/test/artifacts-init-migration.test.ts +++ b/test/artifacts-init-migration.test.ts @@ -22,6 +22,7 @@ function runMigration(fakeHome: string): { code: number; stdout: string; stderr: env: { ...process.env, HOME: fakeHome }, stdout: 'pipe', stderr: 'pipe', + timeout: 30_000, }); return { code: proc.exitCode ?? -1, @@ -217,6 +218,7 @@ function runMigrationV140(fakeHome: string): { code: number; stdout: string; std env: { ...process.env, HOME: fakeHome }, stdout: 'pipe', stderr: 'pipe', + timeout: 30_000, }); return { code: proc.exitCode ?? -1, diff --git a/test/auq-error-fallback-hook.test.ts b/test/auq-error-fallback-hook.test.ts index 0885c95bf..224018ecb 100644 --- a/test/auq-error-fallback-hook.test.ts +++ b/test/auq-error-fallback-hook.test.ts @@ -113,6 +113,7 @@ function runHook(stdin: object, env: Record): { additionalContex input: JSON.stringify(stdin), encoding: 'utf-8', env: { PATH: process.env.PATH ?? '/usr/bin:/bin', ...env }, + timeout: 30_000, }); const parsed = JSON.parse(res.stdout || '{}'); return parsed.hookSpecificOutput ?? {}; diff --git a/test/bin-context-windows-slug.test.ts b/test/bin-context-windows-slug.test.ts index 4764cbcb1..a845229a5 100644 --- a/test/bin-context-windows-slug.test.ts +++ b/test/bin-context-windows-slug.test.ts @@ -69,8 +69,8 @@ describe("native slug fallback mirrors bin/gstack-slug", () => { ["https://gitlab.com/acme/Widget", "acme-Widget"], ] as const) { const cwd = fs.mkdtempSync(path.join(tmp, "repo-")); - spawnSync("git", ["init", "-q"], { cwd }); - spawnSync("git", ["remote", "add", "origin", url], { cwd }); + spawnSync("git", ["init", "-q"], { cwd, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", url], { cwd, timeout: 30_000 }); expect(slugFromEnvironment(path.join(tmp, "home2"), cwd)).toBe(want); } }); @@ -231,8 +231,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => { const projectRoot = path.join(tmp, "realgit"); const subdir = path.join(projectRoot, "src", "deep"); fs.mkdirSync(subdir, { recursive: true }); - spawnSync("git", ["init", "-q", projectRoot]); - spawnSync("git", ["-C", projectRoot, "remote", "add", "origin", "https://github.com/foo/bar.git"]); + spawnSync("git", ["init", "-q", projectRoot], { timeout: 30_000 }); + spawnSync("git", ["-C", projectRoot, "remote", "add", "origin", "https://github.com/foo/bar.git"], { timeout: 30_000 }); expectBoth(subdir, "foo-bar"); }); @@ -269,10 +269,10 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => { const outer = path.join(tmp, "outer-project"); const inner = path.join(outer, "vendor", "inner-lib"); fs.mkdirSync(inner, { recursive: true }); - spawnSync("git", ["init", "-q", outer]); - spawnSync("git", ["-C", outer, "remote", "add", "origin", "git@github.com:acme/outer.git"]); - spawnSync("git", ["init", "-q", inner]); - spawnSync("git", ["-C", inner, "remote", "add", "origin", "git@github.com:vendor/inner.git"]); + spawnSync("git", ["init", "-q", outer], { timeout: 30_000 }); + spawnSync("git", ["-C", outer, "remote", "add", "origin", "git@github.com:acme/outer.git"], { timeout: 30_000 }); + spawnSync("git", ["init", "-q", inner], { timeout: 30_000 }); + spawnSync("git", ["-C", inner, "remote", "add", "origin", "git@github.com:vendor/inner.git"], { timeout: 30_000 }); expectBoth(inner, "acme-outer"); }); @@ -286,8 +286,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => { fs.mkdirSync(path.join(strayHome, ".git"), { recursive: true }); // empty — invalid repo const repo = path.join(strayHome, "work", "repo"); fs.mkdirSync(repo, { recursive: true }); - spawnSync("git", ["init", "-q", repo]); - spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/garrytan/gstack"]); + spawnSync("git", ["init", "-q", repo], { timeout: 30_000 }); + spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/garrytan/gstack"], { timeout: 30_000 }); expectBoth(repo, "garrytan-gstack"); expect(slugFromEnvironment(nativeHome(), repo)).not.toBe("strayhome"); }); @@ -299,9 +299,9 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => { const outer = path.join(tmp, "outer-plain"); const inner = path.join(outer, "vendor", "inner-lib"); fs.mkdirSync(inner, { recursive: true }); - spawnSync("git", ["init", "-q", outer]); // no origin — marker-only repo - spawnSync("git", ["init", "-q", inner]); - spawnSync("git", ["-C", inner, "remote", "add", "origin", "git@github.com:vendor/inner.git"]); + spawnSync("git", ["init", "-q", outer], { timeout: 30_000 }); // no origin — marker-only repo + spawnSync("git", ["init", "-q", inner], { timeout: 30_000 }); + spawnSync("git", ["-C", inner, "remote", "add", "origin", "git@github.com:vendor/inner.git"], { timeout: 30_000 }); expectBoth(inner, "vendor-inner"); }); @@ -313,8 +313,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => { fs.mkdirSync(path.join(strayHome, ".git"), { recursive: true }); const repo = path.join(strayHome, "git", "proj"); fs.mkdirSync(repo, { recursive: true }); - spawnSync("git", ["init", "-q", repo]); - spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/garrytan/gstack"]); + spawnSync("git", ["init", "-q", repo], { timeout: 30_000 }); + spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/garrytan/gstack"], { timeout: 30_000 }); const cacheDir = path.join(nativeHome(), "slug-cache"); fs.mkdirSync(cacheDir, { recursive: true }); @@ -335,8 +335,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => { const inner = path.join(wrapper, "apps", "web"); fs.mkdirSync(inner, { recursive: true }); fs.writeFileSync(path.join(wrapper, "package.json"), '{"name":"wrapper"}\n'); - spawnSync("git", ["init", "-q", inner]); - spawnSync("git", ["-C", inner, "remote", "add", "origin", "https://github.com/acme/web.git"]); + spawnSync("git", ["init", "-q", inner], { timeout: 30_000 }); + spawnSync("git", ["-C", inner, "remote", "add", "origin", "https://github.com/acme/web.git"], { timeout: 30_000 }); const cacheDir = path.join(nativeHome(), "slug-cache"); fs.mkdirSync(cacheDir, { recursive: true }); @@ -361,8 +361,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => { // fire even though cached == basename(project root). const repo = path.join(tmp, "stickyproj"); fs.mkdirSync(repo, { recursive: true }); - spawnSync("git", ["init", "-q", repo]); - spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/x/y.git"]); + spawnSync("git", ["init", "-q", repo], { timeout: 30_000 }); + spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/x/y.git"], { timeout: 30_000 }); const cacheDir = path.join(nativeHome(), "slug-cache"); fs.mkdirSync(cacheDir, { recursive: true }); @@ -379,8 +379,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => { // implementations must reject it and fall through to the basename. const repo = path.join(tmp, "dotty"); fs.mkdirSync(repo, { recursive: true }); - spawnSync("git", ["init", "-q", repo]); - spawnSync("git", ["-C", repo, "remote", "add", "origin", ".."]); + spawnSync("git", ["init", "-q", repo], { timeout: 30_000 }); + spawnSync("git", ["-C", repo, "remote", "add", "origin", ".."], { timeout: 30_000 }); expectBoth(repo, "dotty"); }); diff --git a/test/binding-template-drift.test.ts b/test/binding-template-drift.test.ts index 53f47f6a9..41e3c9d1c 100644 --- a/test/binding-template-drift.test.ts +++ b/test/binding-template-drift.test.ts @@ -102,7 +102,7 @@ describe('content-binding template drift', () => { const abort = spawnSync('bash', ['-c', scriptFor( 'clean body\n', 'body with UNTRUSTED TRACKER CONTENT banner leak\n', - )], { encoding: 'utf-8' }); + )], { encoding: 'utf-8', timeout: 30_000 }); expect(abort.stderr).toContain('ABORT: envelope banner leaked'); expect(abort.stdout).not.toContain('banner tripwire clean'); @@ -110,7 +110,7 @@ describe('content-binding template drift', () => { const clean = spawnSync('bash', ['-c', scriptFor( 'clean body\n', 'also clean body\n', - )], { encoding: 'utf-8' }); + )], { encoding: 'utf-8', timeout: 30_000 }); expect(clean.stdout).toContain('banner tripwire clean'); expect(clean.stderr).not.toContain('ABORT'); } finally { diff --git a/test/brain-sync.test.ts b/test/brain-sync.test.ts index 33eff441a..96e50d01c 100644 --- a/test/brain-sync.test.ts +++ b/test/brain-sync.test.ts @@ -43,12 +43,13 @@ function run(argv: string[], opts: { env?: Record; input?: strin encoding: 'utf-8', input: opts.input, cwd: ROOT, + timeout: 30_000, }); return { stdout: res.stdout || '', stderr: res.stderr || '', status: res.status ?? -1 }; } function git(args: string[], cwd?: string) { - const res = spawnSync('git', args, { cwd: cwd || tmpHome, encoding: 'utf-8' }); + const res = spawnSync('git', args, { cwd: cwd || tmpHome, encoding: 'utf-8', timeout: 30_000 }); return { stdout: res.stdout || '', stderr: res.stderr || '', status: res.status ?? -1 }; } @@ -77,7 +78,7 @@ function seedSpool(record: string): string { beforeEach(() => { tmpHome = fs.mkdtempSync(path.join(os.tmpdir(), 'brain-sync-home-')); bareRemote = fs.mkdtempSync(path.join(os.tmpdir(), 'brain-sync-remote-')); - spawnSync('git', ['init', '--bare', '-q', '-b', 'main', bareRemote]); + spawnSync('git', ['init', '--bare', '-q', '-b', 'main', bareRemote], { timeout: 30_000 }); }); afterEach(() => { @@ -198,6 +199,7 @@ describe('gstack-brain-enqueue', () => { const r = spawnSync(path.join(BIN, 'gstack-brain-enqueue'), [`file-${i}.jsonl`], { env: { ...process.env, GSTACK_HOME: tmpHome }, encoding: 'utf-8', + timeout: 30_000, }); resolve(); })); @@ -245,7 +247,7 @@ describe('gstack-jsonl-merge', () => { const lines = fs.readFileSync(ours, 'utf-8').trim().split('\n'); expect(lines.length).toBe(3); // Order is deterministic (sha256 of each line). - const again = spawnSync(path.join(BIN, 'gstack-jsonl-merge'), [base, ours, theirs]); + const again = spawnSync(path.join(BIN, 'gstack-jsonl-merge'), [base, ours, theirs], { timeout: 30_000 }); // (re-running doesn't change the order since same input → same output) }); }); @@ -271,7 +273,7 @@ describe('init + sync + restore round-trip', () => { test('refuses init on different remote', () => { run(['gstack-artifacts-init', '--remote', bareRemote]); const otherRemote = fs.mkdtempSync(path.join(os.tmpdir(), 'brain-other-')); - spawnSync('git', ['init', '--bare', '-q', '-b', 'main', otherRemote]); + spawnSync('git', ['init', '--bare', '-q', '-b', 'main', otherRemote], { timeout: 30_000 }); const r = run(['gstack-artifacts-init', '--remote', otherRemote]); expect(r.status).not.toBe(0); expect(r.stderr).toContain('already a git repo pointing at'); @@ -288,7 +290,7 @@ describe('init + sync + restore round-trip', () => { const r = run(['gstack-brain-sync', '--once']); expect(r.status).toBe(0); // Check the remote got the commit. - const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }); + const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 }); expect(log.stdout).toMatch(/sync: 1 file/); }); @@ -311,7 +313,7 @@ describe('init + sync + restore round-trip', () => { const restored = fs.readFileSync(path.join(machineB, 'projects/myproj/learnings.jsonl'), 'utf-8'); expect(restored).toContain('machine A wisdom'); // Merge drivers re-registered on B. - const cfg = spawnSync('git', ['-C', machineB, 'config', '--get', 'merge.jsonl-append.driver'], { encoding: 'utf-8' }); + const cfg = spawnSync('git', ['-C', machineB, 'config', '--get', 'merge.jsonl-append.driver'], { encoding: 'utf-8', timeout: 30_000 }); expect(cfg.stdout).toContain('gstack-jsonl-merge'); fs.rmSync(machineB, { recursive: true, force: true }); }); @@ -399,7 +401,7 @@ describe('gstack-brain-sync egress receipt gate', () => { // No local commit was created. expect(git(['rev-list', '--count', 'HEAD']).stdout.trim()).toBe(commitsBefore); // Nothing reached the remote. - const remoteLog = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }); + const remoteLog = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 }); expect(remoteLog.stdout).not.toMatch(/sync: 1 file/); const status = JSON.parse(fs.readFileSync(path.join(tmpHome, '.brain-sync-status.json'), 'utf-8')); expect(status.status).toBe('push_failed'); @@ -411,7 +413,7 @@ describe('gstack-brain-sync egress receipt gate', () => { // Next run (ledger writable again) drains the intact queue and pushes. const retry = run(['gstack-brain-sync', '--once']); expect(retry.status).toBe(0); - const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }); + const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 }); expect(log.stdout).toMatch(/sync: 1 file/); }); @@ -583,7 +585,7 @@ describe('#2549 queue integrity', () => { expect(r.status).toBe(0); expect(spoolText()).not.toContain('learnings.jsonl'); // synced, removed expect(spoolText()).toContain('timeline.jsonl'); // held, retained - const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }); + const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 }); expect(log.stdout).toMatch(/sync: 1 file/); }); @@ -619,7 +621,7 @@ describe('#2549 queue integrity', () => { fs.rmSync(hook); const retry = run(['gstack-brain-sync', '--once']); expect(retry.status).toBe(0); - const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }); + const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 }); expect(log.stdout).toMatch(/sync: 1 file/); expect(git(['rev-list', '--count', 'origin/main..HEAD']).stdout.trim()).toBe('0'); }); @@ -751,7 +753,7 @@ describe('C12 spool queue', () => { run(['gstack-config', 'set', 'artifacts_sync_mode', mode]); } const remoteLog = () => - spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }).stdout; + spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 }).stdout; test('two rapid enqueues of different paths create two spool files; one drain syncs both', () => { initWithMode('full'); diff --git a/test/branch-slug-hygiene.test.ts b/test/branch-slug-hygiene.test.ts index 58c7cccaf..409c5177b 100644 --- a/test/branch-slug-hygiene.test.ts +++ b/test/branch-slug-hygiene.test.ts @@ -37,7 +37,7 @@ const FILENAME_PREFIX = /\$\{?_BRANCH\}?[A-Za-z0-9._-]*\.(?:jsonl|json|md|txt|lo function renderedSkillFiles(): string[] { const out = execSync( `find "${ROOT}" -name 'SKILL.md' -not -path '*/node_modules/*' -not -path '*/.claude/*' ; find "${ROOT}" -path '*/sections/*.md' -not -path '*/node_modules/*' -not -path '*/.claude/*'`, - { encoding: 'utf-8' }, + { encoding: 'utf-8', timeout: 30_000 }, ); return out.split('\n').filter(Boolean); } @@ -91,18 +91,18 @@ describe('branch slug hygiene (#2550, #1851)', () => { const env = { ...process.env, GSTACK_HOME: home }; execSync( 'git init -q && git -c user.email=t@t -c user.name=t commit -q --allow-empty -m init && git checkout -q -b feat/slug-hygiene', - { cwd: repo, encoding: 'utf-8' }, + { cwd: repo, encoding: 'utf-8', timeout: 30_000 }, ); // Writer: the real gstack-review-log (canonicalizes via gstack-slug). execSync( `"${path.join(ROOT, 'bin', 'gstack-review-log')}" '{"skill":"ship","status":"ok"}'`, - { cwd: repo, env, encoding: 'utf-8' }, + { cwd: repo, env, encoding: 'utf-8', timeout: 30_000 }, ); // The slug-canonical filename must exist; the raw form must not. const slugVars = execSync(`"${path.join(ROOT, 'bin', 'gstack-slug')}"`, { - cwd: repo, env, encoding: 'utf-8', + cwd: repo, env, encoding: 'utf-8', timeout: 30_000, }); const slug = slugVars.match(/^SLUG=(.*)$/m)![1]; const branch = slugVars.match(/^BRANCH=(.*)$/m)![1]; @@ -120,7 +120,7 @@ describe('branch slug hygiene (#2550, #1851)', () => { .find((l) => l.includes('-reviews.jsonl'))!; const script = `_PROJ="${proj}"\nBRANCH="${branch}"\n${probeLine.trim()}`; const out = execSync(`bash -c '${script.replace(/'/g, `'\\''`)}'`, { - cwd: repo, encoding: 'utf-8', + cwd: repo, encoding: 'utf-8', timeout: 30_000, }); expect(out).toContain('REVIEWS: 1 entries'); diff --git a/test/builder-profile.test.ts b/test/builder-profile.test.ts index ba00b8303..ae52c2087 100644 --- a/test/builder-profile.test.ts +++ b/test/builder-profile.test.ts @@ -16,7 +16,7 @@ function runProfile(): Record { encoding: 'utf-8', timeout: 15000, }; - const stdout = execSync(`${BIN}/gstack-builder-profile`, execOpts).trim(); + const stdout = execSync(`${BIN}/gstack-builder-profile`, execOpts).trim(); // timeout via execOpts const result: Record = {}; for (const line of stdout.split('\n')) { const idx = line.indexOf(':'); diff --git a/test/ci-image-cli-pin.test.ts b/test/ci-image-cli-pin.test.ts new file mode 100644 index 000000000..e07a19cba --- /dev/null +++ b/test/ci-image-cli-pin.test.ts @@ -0,0 +1,67 @@ +/** + * Provider CLIs baked into the CI image must be pinned to EXACT versions. + * + * The PTY harness (test/helpers/claude-pty-runner.ts) screen-scrapes the + * claude CLI's TUI — trust dialog, input-prompt ready marker, spinner glyphs. + * The image used to install `npm i -g @anthropic-ai/claude-code` UNPINNED and + * rebuild weekly "to pick up CLI updates", while bun sat carefully pinned at + * 1.3.13 two RUN lines above — the exact drift class the bun pin exists for. + * Receipts: TUI drift broke the harness three separate times (welcome-screen + * wedge vs CLI 2.1.233, skillify HOME discovery on 2.1.237, guard/freeze + * hooks on 2.1.162), each debugged as a "flake" before being traced to an + * unpinned weekly-latest CLI. + * + * This tripwire fails the free suite when any globally-installed npm package + * in Dockerfile.ci lacks an exact `@X.Y.Z` pin. Bumps are deliberate: edit + * the pin in a PR and run the PTY gate against the new TUI before merging. + */ +import { describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +const ROOT = path.resolve(__dirname, '..'); +const DOCKERFILE = path.join(ROOT, '.github', 'docker', 'Dockerfile.ci'); + +/** Package specs from every `npm i -g` / `npm install -g` in the Dockerfile. */ +export function globalNpmInstallSpecs(source: string): string[] { + const specs: string[] = []; + for (const match of source.matchAll(/npm\s+(?:i|install)\s+(?:-g|--global)\s+([^\n\\&|;]+)/g)) { + for (const spec of match[1].trim().split(/\s+/)) { + if (spec.startsWith('-')) continue; // flags like --no-fund + specs.push(spec); + } + } + return specs; +} + +/** Exact pin = a trailing @ with no range operator (no ^ ~ x *). */ +export function isExactlyPinned(spec: string): boolean { + // Scoped (@scope/name@1.2.3) or bare (name@1.2.3); version must be exact. + const at = spec.lastIndexOf('@'); + if (at <= 0) return false; // no version at all (or a bare scope) + const version = spec.slice(at + 1); + return /^\d+\.\d+\.\d+(?:-[\w.]+)?$/.test(version); +} + +describe('ci image provider-CLI pins', () => { + const source = fs.readFileSync(DOCKERFILE, 'utf-8'); + const specs = globalNpmInstallSpecs(source); + + test('the image installs at least the claude CLI globally (scan must not rot)', () => { + expect( + specs.some((s) => s.startsWith('@anthropic-ai/claude-code@')), + `expected a pinned @anthropic-ai/claude-code install in ${path.relative(ROOT, DOCKERFILE)}; found: ${specs.join(', ') || '(none)'}`, + ).toBe(true); + }); + + test('every global npm install carries an exact @X.Y.Z pin', () => { + const unpinned = specs.filter((s) => !isExactlyPinned(s)); + expect( + unpinned, + `unpinned global npm installs in Dockerfile.ci: ${unpinned.join(', ')}\n` + + 'Pin the exact version (name@X.Y.Z) and bump via a PR that runs the ' + + 'PTY gate against the new TUI — weekly-latest CLI drift broke the ' + + 'harness three times before this tripwire existed.', + ).toHaveLength(0); + }); +}); diff --git a/test/code-intelligence.test.ts b/test/code-intelligence.test.ts index 56dff9cba..a0adb51dc 100644 --- a/test/code-intelligence.test.ts +++ b/test/code-intelligence.test.ts @@ -153,9 +153,9 @@ describe("session-start indexing offer (suggest)", () => { home = fs.mkdtempSync(path.join(os.tmpdir(), "ci-home-")); repo = fs.mkdtempSync(path.join(os.tmpdir(), "ci-repo-")); env = { ...process.env, GSTACK_HOME: home }; - Bun.spawnSync(["git", "init", "-q", repo]); + Bun.spawnSync(["git", "init", "-q", repo], { timeout: 30_000 }); for (const name of ["a.ts", "b.ts", "c.ts"]) fs.writeFileSync(path.join(repo, name), "x\n"); - Bun.spawnSync(["git", "-C", repo, "add", "-A"]); + Bun.spawnSync(["git", "-C", repo, "add", "-A"], { timeout: 30_000 }); }); afterEach(() => { fs.rmSync(home, { recursive: true, force: true }); @@ -399,7 +399,7 @@ describe("consent unification — deny tier wins (R1)", () => { function makeRepo(dir: string, url: string): string { const repo = path.join(dir, "repo"); fs.mkdirSync(repo, { recursive: true }); - const git = (...a: string[]) => execFileSync("git", a, { cwd: repo }); + const git = (...a: string[]) => execFileSync("git", a, { cwd: repo, timeout: 30_000 }); git("init", "-q", "."); git("remote", "add", "origin", url); return repo; @@ -423,11 +423,11 @@ describe("consent unification — deny tier wins (R1)", () => { const env = { ...process.env, GSTACK_HOME: home }; const repo = makeRepo(home, URL); setConsent(repo, true, env); - execFileSync(POLICY_BIN, ["set", URL, "deny"], { env, encoding: "utf-8" }); + execFileSync(POLICY_BIN, ["set", URL, "deny"], { env, encoding: "utf-8", timeout: 30_000 }); expect(hasConsent(repo, env)).toBe(false); // Flipping the tier back restores the recorded consent — the veto is // live policy, not a destructive rewrite of the consent store. - execFileSync(POLICY_BIN, ["set", URL, "read-write"], { env, encoding: "utf-8" }); + execFileSync(POLICY_BIN, ["set", URL, "read-write"], { env, encoding: "utf-8", timeout: 30_000 }); expect(hasConsent(repo, env)).toBe(true); } finally { fs.rmSync(home, { recursive: true, force: true }); } }); @@ -439,7 +439,7 @@ describe("consent unification — deny tier wins (R1)", () => { const env = { ...process.env, GSTACK_HOME: home }; const repo = makeRepo(home, URL); setConsent(repo, true, env); - execFileSync(POLICY_BIN, ["set", URL, "read-write"], { env, encoding: "utf-8" }); + execFileSync(POLICY_BIN, ["set", URL, "read-write"], { env, encoding: "utf-8", timeout: 30_000 }); fs.chmodSync(path.join(home, "gbrain-repo-policy.json"), 0o000); try { expect(hasConsent(repo, env)).toBe(false); @@ -460,7 +460,7 @@ describe("consent unification — deny tier wins (R1)", () => { const env = { ...process.env, GSTACK_HOME: home }; const repo = makeRepo(home, URL); setConsent(repo, true, env); - execFileSync(POLICY_BIN, ["set", URL, "read-only"], { env, encoding: "utf-8" }); + execFileSync(POLICY_BIN, ["set", URL, "read-only"], { env, encoding: "utf-8", timeout: 30_000 }); // Default op class is write — a caller that doesn't say gets fail-closed. expect(hasConsent(repo, env)).toBe(false); expect(hasConsent(repo, env, "write")).toBe(false); @@ -475,7 +475,7 @@ describe("consent unification — deny tier wins (R1)", () => { const env = { ...process.env, GSTACK_HOME: home }; const repo = makeRepo(home, URL); setConsent(repo, true, env); - execFileSync(POLICY_BIN, ["set", URL, "deny"], { env, encoding: "utf-8" }); + execFileSync(POLICY_BIN, ["set", URL, "deny"], { env, encoding: "utf-8", timeout: 30_000 }); expect(hasConsent(repo, env, "write")).toBe(false); expect(hasConsent(repo, env, "read")).toBe(false); } finally { fs.rmSync(home, { recursive: true, force: true }); } @@ -971,9 +971,9 @@ exit 1 function makeRepoWithFiles(count: number): string { const repo = fs.mkdtempSync(path.join(os.tmpdir(), "ci-cli-suggest-")); - Bun.spawnSync(["git", "init", "-q", repo]); + Bun.spawnSync(["git", "init", "-q", repo], { timeout: 30_000 }); for (let i = 0; i < count; i++) fs.writeFileSync(path.join(repo, `f${i}.ts`), "x\n"); - Bun.spawnSync(["git", "-C", repo, "add", "-A"]); + Bun.spawnSync(["git", "-C", repo, "add", "-A"], { timeout: 30_000 }); return repo; } diff --git a/test/codex-e2e-plan-format.test.ts b/test/codex-e2e-plan-format.test.ts index 3bfce0fdd..4cc591b1c 100644 --- a/test/codex-e2e-plan-format.test.ts +++ b/test/codex-e2e-plan-format.test.ts @@ -43,7 +43,7 @@ const ROOT = path.resolve(import.meta.dir, '..'); const CODEX_AVAILABLE = (() => { try { - const result = Bun.spawnSync(['which', 'codex']); + const result = Bun.spawnSync(['which', 'codex'], { timeout: 30_000 }); return result.exitCode === 0; } catch { return false; } })(); diff --git a/test/codex-e2e-recommendation-substance.test.ts b/test/codex-e2e-recommendation-substance.test.ts index f4a6530c0..8d7c9404f 100644 --- a/test/codex-e2e-recommendation-substance.test.ts +++ b/test/codex-e2e-recommendation-substance.test.ts @@ -31,7 +31,7 @@ const ROOT = path.resolve(import.meta.dir, '..'); const CODEX_AVAILABLE = (() => { try { - return Bun.spawnSync(['which', 'codex']).exitCode === 0; + return Bun.spawnSync(['which', 'codex'], { timeout: 30_000 }).exitCode === 0; } catch { return false; } diff --git a/test/codex-e2e-sol-scope.test.ts b/test/codex-e2e-sol-scope.test.ts index dd3c52aee..2603cdd6e 100644 --- a/test/codex-e2e-sol-scope.test.ts +++ b/test/codex-e2e-sol-scope.test.ts @@ -21,12 +21,12 @@ import { EvalCollector } from './helpers/eval-store'; import { selectTests, detectBaseBranch, getChangedFiles, GLOBAL_TOUCHFILES } from './helpers/touchfiles'; const ROOT = path.resolve(import.meta.dir, '..'); -const CODEX_AVAILABLE = spawnSync('which', ['codex']).status === 0; +const CODEX_AVAILABLE = spawnSync('which', ['codex'], { timeout: 30_000 }).status === 0; // The run pins the model with --ignore-user-config; older codex CLIs reject // the flag with an argv error indistinguishable from a Sol regression, so // probe support and skip (not fail) on old CLIs. const IGNORE_USER_CONFIG_SUPPORTED = CODEX_AVAILABLE - && (spawnSync('codex', ['exec', '--help'], { encoding: 'utf8' }).stdout ?? '').includes('--ignore-user-config'); + && (spawnSync('codex', ['exec', '--help'], { encoding: 'utf8', timeout: 120_000 }).stdout ?? '').includes('--ignore-user-config'); const evalsEnabled = !!process.env.EVALS; // External-service test — periodic tier only (CLAUDE.md tiering rule 3). The // positive guard shape below is what classifyPaidTestFile greps to exclude diff --git a/test/codex-e2e.test.ts b/test/codex-e2e.test.ts index dc51d322d..1e4b0eeb3 100644 --- a/test/codex-e2e.test.ts +++ b/test/codex-e2e.test.ts @@ -32,7 +32,7 @@ const ROOT = path.resolve(import.meta.dir, '..'); const CODEX_AVAILABLE = (() => { try { - const result = Bun.spawnSync(['which', 'codex']); + const result = Bun.spawnSync(['which', 'codex'], { timeout: 30_000 }); return result.exitCode === 0; } catch { return false; } })(); @@ -63,11 +63,18 @@ if (!evalsEnabled) { // --- Diff-based test selection --- -// Codex E2E touchfiles — keyed by test name, same pattern as E2E_TOUCHFILES -const CODEX_E2E_TOUCHFILES: Record = { - 'codex-discover-skill': ['codex/**', '.agents/skills/**', 'test/helpers/codex-session-runner.ts'], - 'codex-review-findings': ['review/**', '.agents/skills/gstack-review/**', 'codex/**', 'test/helpers/codex-session-runner.ts'], -}; +// Codex E2E touchfiles — DERIVED from the canonical map, never a local fork. +// The old hand-copy drifted (it kept gitignored '.agents/skills/**' patterns +// that can never match a git diff, and missed deps the canonical map gained +// like lib/worktree.ts and this test file itself), so review-template edits +// silently stopped selecting these tests. Deriving keeps one source of truth +// and puts these keys under the tier-alignment + dep-existence invariants. +const CODEX_E2E_TOUCHFILES: Record = Object.fromEntries( + (['codex-discover-skill', 'codex-review-findings'] as const).map((key) => { + if (!E2E_TOUCHFILES[key]) throw new Error(`canonical E2E_TOUCHFILES lost key '${key}' — fix the map, not this file`); + return [key, E2E_TOUCHFILES[key]]; + }), +); let selectedTests: string[] | null = null; // null = run all diff --git a/test/codex-generation-model.test.ts b/test/codex-generation-model.test.ts index 9f04df296..294b9ec77 100644 --- a/test/codex-generation-model.test.ts +++ b/test/codex-generation-model.test.ts @@ -116,6 +116,7 @@ model = "gpt-5.6-terra" cwd: ROOT, encoding: 'utf8', env: { ...process.env, CODEX_HOME: home }, + timeout: 30_000, }); expect(ok.status).toBe(0); expect(ok.stdout).toBe(`gpt-5.6-sol\t${path.join(home, 'config.toml')}\n`); @@ -123,6 +124,7 @@ model = "gpt-5.6-terra" const bad = spawnSync('bun', ['run', 'scripts/resolve-codex-generation-model.ts', '--explicit', 'llama-local'], { cwd: ROOT, encoding: 'utf8', + timeout: 30_000, }); expect(bad.status).not.toBe(0); expect(bad.stderr).toContain('Unknown model'); diff --git a/test/codex-hardening.test.ts b/test/codex-hardening.test.ts index 27b6a75fd..ee170d88f 100644 --- a/test/codex-hardening.test.ts +++ b/test/codex-hardening.test.ts @@ -297,7 +297,7 @@ describe('gstack-codex-probe: timeout wrapper + namespace hygiene', () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-watchdog-')); try { const which = (tool: string) => - spawnSync('bash', ['-c', `command -v ${tool}`]).stdout.toString().trim() || `/bin/${tool}`; + spawnSync('bash', ['-c', `command -v ${tool}`], { timeout: 30_000 }).stdout.toString().trim() || `/bin/${tool}`; fs.symlinkSync(which('bash'), path.join(dir, 'bash')); fs.symlinkSync(which('sleep'), path.join(dir, 'sleep')); const r = runProbe({ diff --git a/test/codex-resume-flag-semantics.test.ts b/test/codex-resume-flag-semantics.test.ts index 9075e0489..8f702ef2b 100644 --- a/test/codex-resume-flag-semantics.test.ts +++ b/test/codex-resume-flag-semantics.test.ts @@ -15,7 +15,7 @@ import { describe, test, expect } from 'bun:test'; import { spawnSync } from 'child_process'; -const codexPath = spawnSync('which', ['codex'], { encoding: 'utf-8' }).stdout.trim(); +const codexPath = spawnSync('which', ['codex'], { encoding: 'utf-8', timeout: 30_000 }).stdout.trim(); const codexAvailable = codexPath.length > 0; describe.skipIf(!codexAvailable)( diff --git a/test/codex-web-search-flag.test.ts b/test/codex-web-search-flag.test.ts index ed7c4f612..217bc5796 100644 --- a/test/codex-web-search-flag.test.ts +++ b/test/codex-web-search-flag.test.ts @@ -23,7 +23,7 @@ function grepRepo(pattern: string, includes: string[]): string[] { const includeArgs = includes.map((i) => `--include='${i}'`).join(' '); const out = execSync( `grep -rln ${includeArgs} -e '${pattern}' "${ROOT}" || true`, - { encoding: 'utf-8' }, + { encoding: 'utf-8', timeout: 30_000 }, ); return out .split('\n') diff --git a/test/context-bill.test.ts b/test/context-bill.test.ts index 5a05db01f..82afda6d9 100644 --- a/test/context-bill.test.ts +++ b/test/context-bill.test.ts @@ -718,7 +718,7 @@ describe("CLI plumbing", () => { }); it("bin/gstack-context-bill runs standalone", () => { - const result = Bun.spawnSync([path.join(ROOT, "bin", "gstack-context-bill"), TREE_A]); + const result = Bun.spawnSync([path.join(ROOT, "bin", "gstack-context-bill"), TREE_A], { timeout: 30_000 }); expect(result.exitCode).toBe(0); expect(result.stdout.toString()).toContain("ALWAYS-ON"); expect(result.stdout.toString()).toContain("EAGER"); diff --git a/test/design-flag-utils.test.ts b/test/design-flag-utils.test.ts index c891bf039..f5882dc67 100644 --- a/test/design-flag-utils.test.ts +++ b/test/design-flag-utils.test.ts @@ -99,7 +99,7 @@ describe("normalizeIntFlag CLI wrapper (exit-1 semantics)", () => { const v = normalizeIntFlag(${rawExpr}, ${specExpr}); console.log("VALUE:" + v); `; - const res = spawnSync("bun", ["-e", script], { encoding: "utf-8", cwd: ROOT }); + const res = spawnSync("bun", ["-e", script], { encoding: "utf-8", cwd: ROOT, timeout: 30_000 }); return { status: res.status ?? -1, stderr: res.stderr ?? "" }; } diff --git a/test/diagram-render-drift.test.ts b/test/diagram-render-drift.test.ts index 231f13a42..48b1574e6 100644 --- a/test/diagram-render-drift.test.ts +++ b/test/diagram-render-drift.test.ts @@ -86,7 +86,7 @@ describe("diagram-render bundle drift", () => { "deep: fresh build reproduces committed dist", async () => { const before = await Bun.file(BUILD_INFO).json(); - const proc = Bun.spawnSync(["bun", "run", "scripts/build.ts"], { cwd: ROOT }); + const proc = Bun.spawnSync(["bun", "run", "scripts/build.ts"], { cwd: ROOT, timeout: 120_000 }); expect(proc.exitCode).toBe(0); const after = await Bun.file(BUILD_INFO).json(); expect(after.sha256).toBe(before.sha256); diff --git a/test/distill-apply.test.ts b/test/distill-apply.test.ts index e46781c21..3012776f8 100644 --- a/test/distill-apply.test.ts +++ b/test/distill-apply.test.ts @@ -60,7 +60,7 @@ function run(args: string[]): { stdout: string; stderr: string; status: number } env.GSTACK_STATE_ROOT = stateRoot; env.GSTACK_QUESTION_LOG_NO_DERIVE = '1'; delete env.GSTACK_HOME; - const res = spawnSync(BIN, args, { env, encoding: 'utf-8', cwd: fixtureCwd }); + const res = spawnSync(BIN, args, { env, encoding: 'utf-8', cwd: fixtureCwd, timeout: 30_000 }); return { stdout: res.stdout ?? '', stderr: res.stderr ?? '', diff --git a/test/distill-free-text.test.ts b/test/distill-free-text.test.ts index a79490831..b2d543cf8 100644 --- a/test/distill-free-text.test.ts +++ b/test/distill-free-text.test.ts @@ -47,6 +47,7 @@ function run(args: string[]): { stdout: string; stderr: string; status: number } env: makeEnv(), encoding: 'utf-8', cwd: fixtureCwd, + timeout: 30_000, }); return { stdout: res.stdout ?? '', @@ -75,6 +76,7 @@ function writeAuqOtherEvent(text: string): void { env: makeEnv(), cwd: fixtureCwd, encoding: 'utf-8', + timeout: 30_000, }, ); } @@ -150,7 +152,7 @@ describe('no-event paths', () => { tool_use_id: 'tu-x', }), ], - { env: makeEnv(), cwd: fixtureCwd, encoding: 'utf-8' }, + { env: makeEnv(), cwd: fixtureCwd, encoding: 'utf-8', timeout: 30_000 }, ); const r = run([]); expect(r.status).toBe(0); @@ -169,7 +171,7 @@ describe('--dry-run', () => { // Strip ANTHROPIC_API_KEY to prove no API call happens. const env = makeEnv(); delete env.ANTHROPIC_API_KEY; - const res = spawnSync(BIN, ['--dry-run'], { env, cwd: fixtureCwd, encoding: 'utf-8' }); + const res = spawnSync(BIN, ['--dry-run'], { env, cwd: fixtureCwd, encoding: 'utf-8', timeout: 30_000 }); expect(res.status).toBe(0); expect(res.stdout).toContain('DISTILL PROMPT'); expect(res.stdout).toContain('always include tests'); @@ -185,7 +187,7 @@ describe('API auth', () => { writeAuqOtherEvent('Some free text response that needs distilling'); const env = makeEnv(); delete env.ANTHROPIC_API_KEY; - const res = spawnSync(BIN, [], { env, cwd: fixtureCwd, encoding: 'utf-8' }); + const res = spawnSync(BIN, [], { env, cwd: fixtureCwd, encoding: 'utf-8', timeout: 30_000 }); expect(res.status).not.toBe(0); expect(res.stderr).toMatch(/ANTHROPIC_API_KEY/); expect(res.stderr).toMatch(/separate billing/); diff --git a/test/e2e-tier-alignment.test.ts b/test/e2e-tier-alignment.test.ts index ba6bcee10..0a95eb0ef 100644 --- a/test/e2e-tier-alignment.test.ts +++ b/test/e2e-tier-alignment.test.ts @@ -42,8 +42,8 @@ const SELF_GATE_RE = /EVALS_TIER\s*===\s*['"](gate|periodic)['"]/g; const HELPER_GATE_RE = /\b(?:describeE2ETier|e2eTierEnabled)\(\s*['"](gate|periodic)['"]/g; /** - * Ratchet, not amnesty (same contract as KNOWN_MATRIX_GAPS in - * test/evals-workflow-matrix.test.ts): skill-e2e files that are named in NO + * Ratchet, not amnesty (the contract KNOWN_MATRIX_GAPS pioneered before the + * legacy eval matrix and its test retired): skill-e2e files that are named in NO * touchfiles dep list because no E2E map key exists for them. Every entry * carries a one-line reason. Do NOT add new files here — give the test an * E2E map key (touchfiles + tier) and register the file in its dep list. @@ -141,8 +141,8 @@ describe('E2E tier alignment (touchfiles declaration vs test self-gate)', () => expect(misaligned).toEqual([]); }); - // Ratchet cleanup enforcement (same contract as evals-workflow-matrix's - // burn-down test): a KNOWN_UNREGISTERED entry whose file was deleted, or + // Ratchet cleanup enforcement (same shrink-only contract as the retired + // matrix test's burn-down): a KNOWN_UNREGISTERED entry whose file was deleted, or // whose file is now named in a dep list, is stale — delete the entry so // the set can only shrink. test('KNOWN_UNREGISTERED holds only live, still-unregistered files', () => { diff --git a/test/egress-receipt.test.ts b/test/egress-receipt.test.ts index 4bbc16d5f..07a9e90b6 100644 --- a/test/egress-receipt.test.ts +++ b/test/egress-receipt.test.ts @@ -222,12 +222,12 @@ describe('gstack-egress-receipt shell bridge', () => { fs.writeFileSync(payload, '[{"v":1}]'); const write = spawnSync(bin, ['write', '--sink', 'telemetry-sync', '--host', '127.0.0.1:8399', '--class', 'telemetry-events', '--payload-file', payload, '--consent', 'telemetry=community'], - { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } }); + { encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GSTACK_HOME: home } }); expect(write.status).toBe(0); const id = write.stdout.trim(); expect(id).toMatch(/^[0-9a-f]{64}$/); const outcome = spawnSync(bin, ['outcome', id, '204'], - { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } }); + { encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GSTACK_HOME: home } }); expect(outcome.status).toBe(0); const receipts = listReceipts(home); expect(receipts.length).toBe(1); @@ -239,7 +239,7 @@ describe('gstack-egress-receipt shell bridge', () => { test('--no-payload records sha256:null (git-class: a subprocess owns the bytes)', () => { const write = spawnSync(bin, ['write', '--sink', 'brain-sync', '--host', 'github.com', '--class', 'git-push', '--no-payload', '--consent', 'artifacts_sync_mode=auto'], - { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } }); + { encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GSTACK_HOME: home } }); expect(write.status).toBe(0); const receipts = listReceipts(home); expect(receipts.length).toBe(1); @@ -251,7 +251,7 @@ describe('gstack-egress-receipt shell bridge', () => { if (!canRevokeWrites()) return; // chmod is advisory here (win32, root, DAC-override containers) fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 }); const write = spawnSync(bin, ['write', '--sink', 's', '--host', 'h', '--class', 'c', '--no-payload'], - { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } }); + { encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GSTACK_HOME: home } }); expect(write.status).toBe(3); expect(write.stderr).toContain('EGRESS_RECEIPT_FAILED'); fs.chmodSync(path.join(home, 'security'), 0o700); diff --git a/test/empty-find-fallthrough.test.ts b/test/empty-find-fallthrough.test.ts index 10782ae9d..634c644f0 100644 --- a/test/empty-find-fallthrough.test.ts +++ b/test/empty-find-fallthrough.test.ts @@ -35,7 +35,7 @@ describe('empty find must not fall through to cwd (#2483)', () => { test('no resolver emits a bare `xargs ls -t` (must be `xargs -r ls -t`)', () => { const out = execSync( `grep -rn "xargs ls -t" "${path.join(ROOT, 'scripts')}" "${path.join(ROOT, 'bin')}" || true`, - { encoding: 'utf-8' }, + { encoding: 'utf-8', timeout: 30_000 }, ); expect(out.trim()).toBe(''); }); @@ -67,6 +67,7 @@ describe('empty find must not fall through to cwd (#2483)', () => { const out = execSync(`bash -c '${script.replace(/'/g, `'\\''`)}'`, { cwd, encoding: 'utf-8', + timeout: 30_000, }); expect(out).toContain('LATEST_CP=[]'); expect(out).not.toContain('DECOY.md'); @@ -79,7 +80,7 @@ describe('empty find must not fall through to cwd (#2483)', () => { test('no generated SKILL.md carries the unguarded form', () => { const out = execSync( `grep -rln "xargs ls -t" --include=SKILL.md "${ROOT}" || true`, - { encoding: 'utf-8' }, + { encoding: 'utf-8', timeout: 30_000 }, ); // node_modules and vendored trees are not generated output; nothing in // the repo's generated skills may carry the unguarded form. diff --git a/test/eval-budgets-policy.test.ts b/test/eval-budgets-policy.test.ts index 3d39c0ed1..6e7bff921 100644 --- a/test/eval-budgets-policy.test.ts +++ b/test/eval-budgets-policy.test.ts @@ -41,7 +41,7 @@ describe('eval budget tiers', () => { }); test('no paid-test timeout literal exceeds the ceiling tier', () => { - const out = spawnSync('git', ['ls-files', 'test/*.test.ts'], { cwd: ROOT, encoding: 'utf-8' }); + const out = spawnSync('git', ['ls-files', 'test/*.test.ts'], { cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); const files = out.stdout.split('\n').filter((f) => f && isPaidTestFile(f)); expect(files.length).toBeGreaterThan(50); // scan-rot guard diff --git a/test/eval-flake-rank.test.ts b/test/eval-flake-rank.test.ts new file mode 100644 index 000000000..661b187f4 --- /dev/null +++ b/test/eval-flake-rank.test.ts @@ -0,0 +1,70 @@ +/** + * Unit tests for the flake-rank aggregator (WS1's dial). The CLI ranks tests + * by retried passes (the flake signature) across finalized eval-store runs — + * these pin the accounting: N attempt records = 1 run of that test, the + * FINAL attempt decides pass/fail, retried passes count separately, partials + * and runner artifacts are excluded, shard dirs recurse, and the recency + * bound drops stale files. + */ +import { describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { aggregate, collectEvalFiles } from '../scripts/eval-flake-rank'; + +const entry = (name: string, passed: boolean, attempt: number) => ({ + name, suite: 's', tier: 'e2e', passed, attempt, duration_ms: 1000, cost_usd: 0.1, +}); + +const run = (tests: object[], extra: object = {}) => JSON.stringify({ + schema_version: 2, version: '1.0.0', branch: 'b', git_sha: 'x', hostname: 'h', + timestamp: '2026-08-31T00:00:00Z', tier: 'e2e', + total_tests: tests.length, passed: 0, failed: 0, total_cost_usd: 0, total_duration_ms: 0, + tests, ...extra, +}); + +describe('eval-flake-rank aggregate', () => { + test('final attempt decides; retried pass counts as retriedPass, not a fail', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flakerank-')); + fs.writeFileSync(path.join(dir, 'run1.json'), run([ + entry('flaky', false, 1), entry('flaky', true, 2), // pass on retry + entry('steady', true, 1), + entry('broken', false, 1), entry('broken', false, 2), // fails even retried + ])); + fs.writeFileSync(path.join(dir, 'run2.json'), run([ + entry('flaky', true, 1), entry('steady', true, 1), + ])); + const series = aggregate(collectEvalFiles(dir)); + expect(series.get('flaky')).toMatchObject({ runs: 2, passes: 2, fails: 0, retriedPasses: 1, totalAttempts: 3 }); + expect(series.get('steady')).toMatchObject({ runs: 2, passes: 2, fails: 0, retriedPasses: 0 }); + expect(series.get('broken')).toMatchObject({ runs: 1, passes: 0, fails: 1, retriedPasses: 0, totalAttempts: 2 }); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + test('partials and runner artifacts are excluded; shard dirs recurse', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flakerank-')); + fs.mkdirSync(path.join(dir, 'shards', 'slug-a'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'shards', 'slug-a', 'run.json'), run([entry('sharded', true, 1)])); + fs.writeFileSync(path.join(dir, '_partial-e2e.json'), run([entry('inflight', false, 1)], { _partial: true })); + fs.writeFileSync(path.join(dir, 'manifest.json'), '{"version":1}'); + fs.writeFileSync(path.join(dir, 'slice-3.json'), '{"version":1}'); + const files = collectEvalFiles(dir); + expect(files).toHaveLength(1); + const series = aggregate(files); + expect(series.has('sharded')).toBe(true); + expect(series.has('inflight')).toBe(false); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + test('recency bound drops files older than sinceDays', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flakerank-')); + const stale = path.join(dir, 'old.json'); + fs.writeFileSync(stale, run([entry('ancient', true, 1)])); + const old = new Date(Date.now() - 90 * 86_400_000); + fs.utimesSync(stale, old, old); + fs.writeFileSync(path.join(dir, 'new.json'), run([entry('recent', true, 1)])); + const files = collectEvalFiles(dir, 60); + expect(files.map((f) => path.basename(f))).toEqual(['new.json']); + fs.rmSync(dir, { recursive: true, force: true }); + }); +}); diff --git a/test/eval-list-cli.test.ts b/test/eval-list-cli.test.ts index 536e36617..cffc68e09 100644 --- a/test/eval-list-cli.test.ts +++ b/test/eval-list-cli.test.ts @@ -65,6 +65,7 @@ function runEvalList(...args: string[]): { stdout: string; stderr: string; statu GSTACK_HOME: path.join(tmpHome, '.gstack'), }, encoding: 'utf-8', + timeout: 30_000, }); return { stdout: result.stdout ?? '', diff --git a/test/evals-workflow-matrix.test.ts b/test/evals-workflow-matrix.test.ts deleted file mode 100644 index b59353a08..000000000 --- a/test/evals-workflow-matrix.test.ts +++ /dev/null @@ -1,197 +0,0 @@ -/** - * CI eval-matrix completeness tripwire — kills the silent-skip class where a - * gate-tier test exists in the repo but the hand-enumerated matrix in - * .github/workflows/evals.yml never runs it, so "gate tier blocks merge" is - * quietly false in CI. This has happened before (see the "rehomed from the - * deleted pre-split monolith" comment in evals.yml) and was found again on - * PR #2700: nine gate-hosting files absent from the matrix, plus matrix rows - * whose whole-file tier guards can never fire because the Run step exported - * no EVALS_TIER. - * - * Ratchet, not amnesty: the KNOWN_* lists below enumerate the PRE-EXISTING - * gaps with reasons, so no NEW gap can land while the backlog burns down - * (same pattern as SCANNER_EXEMPT in egress-receipt-wiring). If you fix a - * listed gap (add its matrix row / tier property), this test FAILS until you - * remove the entry — stale exemptions are enforced, not decorative. - * - * Wiring pinned: - * - every matrix `file:` path exists on disk (no stale rows), - * - every gate-hosting paid file (whole-file gate self-gate, or named in the - * dep list of a gate-tier E2E_TOUCHFILES key) appears in the matrix or in - * KNOWN_MATRIX_GAPS, - * - every matrix file with a whole-file tier guard has a matching row-level - * `tier:` property (else the suite self-skips and the job is hollow-green) - * or sits in KNOWN_TIER_UNSET. - */ -import { describe, test, expect } from 'bun:test'; -import * as fs from 'fs'; -import * as path from 'path'; -import { E2E_TOUCHFILES, E2E_TIERS } from './helpers/touchfiles-data'; -import { isPaidTestFile } from './helpers/paid-test-set'; - -const ROOT = path.join(import.meta.dir, '..'); -const WORKFLOW = path.join(ROOT, '.github', 'workflows', 'evals.yml'); - -/** - * Pre-existing gate-hosting files with no matrix row (found 2026-08-26, - * PR #2700). Adding a row activates real paid runs on every PR — a cost and - * flake-surface decision per file, tracked in TODOS.md ("CI gate-lane - * hollow-coverage burn-down"). Fix = add a matrix row (plus `tier: gate` when - * the file is whole-file gated), then DELETE the entry here. - */ -const KNOWN_MATRIX_GAPS = new Set([ - 'test/skill-e2e-ask-user-question-format-compliance.test.ts', - 'test/skill-e2e-hermetic-canary.test.ts', - 'test/skill-e2e-ios.test.ts', - 'test/skill-e2e-plan-ceo-finding-floor.test.ts', - 'test/skill-e2e-plan-ceo-plan-mode.test.ts', - 'test/skill-e2e-plan-design-with-ui.test.ts', - 'test/skill-e2e-plan-devex-finding-floor.test.ts', - 'test/skill-e2e-plan-devex-plan-mode.test.ts', - // Exposed by the 2026-08 dep-list self-registration sweep: these eight had - // zero gate-key dep-list membership before it, so the census never saw - // them as gate-hosting. Their gate tests run in NO CI lane today. The - // paid-lane re-platform (test-paid-shards.ts as the CI engine) runs every - // gate-tier file by construction and retires this whole ratchet. - 'test/skill-e2e-cso.test.ts', - 'test/skill-e2e-diagram.test.ts', - 'test/skill-e2e-learnings.test.ts', - 'test/skill-e2e-plan-tune.test.ts', - 'test/skill-e2e-plan-tune-cathedral.test.ts', - 'test/skill-e2e-review-army.test.ts', - 'test/skill-e2e-session-intelligence.test.ts', - 'test/skill-e2e-skillify.test.ts', -]); - -/** - * Matrix files whose whole-file tier guard has no matching row `tier:` - * property. Burned down to empty 2026-08-29: the vestigial codex/gemini rows - * were deleted (periodic-tier files, zero tests per PR) and - * e2e-pty-plan-smoke gained its `tier: gate`. The ratchet stays so a future - * row/file tier mismatch fails the suite instead of shipping hollow green. - */ -const KNOWN_TIER_UNSET = new Map([]); - -interface MatrixRow { - name: string; - files: string[]; - tier?: string; -} - -/** Parse the `matrix: suite:` rows (name / file / optional tier) from evals.yml. */ -function parseMatrixRows(source: string): MatrixRow[] { - const rows: MatrixRow[] = []; - let current: MatrixRow | null = null; - for (const line of source.split('\n')) { - const name = line.match(/^\s+- name: (\S+)\s*$/); - if (name) { - if (current) rows.push(current); - current = { name: name[1], files: [] }; - continue; - } - if (!current) continue; - const file = line.match(/^\s+file: (.+?)\s*$/); - if (file) current.files.push(...file[1].trim().split(/\s+/)); - const tier = line.match(/^\s+tier: (\S+)\s*$/); - if (tier) current.tier = tier[1]; - // `steps:` ends the strategy block — stop before step-level keys leak in. - if (/^\s{4}steps:\s*$/.test(line)) break; - } - if (current) rows.push(current); - return rows.filter((r) => r.files.length > 0); -} - -const wholeFileTier = (source: string): string | null => { - const m = - /\b(?:describeE2ETier|e2eTierEnabled)\(\s*['"`](gate|periodic)['"`]/.exec(source) || - /EVALS_TIER\s*===\s*['"`](gate|periodic)['"`]/.exec(source); - return m ? m[1] : null; -}; - -const workflowSource = fs.readFileSync(WORKFLOW, 'utf-8'); -const rows = parseMatrixRows(workflowSource); -const matrixFiles = new Map(); -for (const row of rows) for (const f of row.files) matrixFiles.set(f, row); - -const paidFiles = fs - .readdirSync(path.join(ROOT, 'test')) - .filter((f) => f.endsWith('.test.ts')) - .map((f) => `test/${f}`) - .filter(isPaidTestFile); - -describe('evals.yml matrix completeness (gate-lane silent-skip tripwire)', () => { - test('matrix parse sanity: rows and known suites present', () => { - expect(rows.length).toBeGreaterThanOrEqual(15); - expect(matrixFiles.has('test/skill-e2e-workflow.test.ts')).toBe(true); - expect(matrixFiles.has('test/skill-e2e-ship-docsync.test.ts')).toBe(true); - }); - - test('every matrix file exists on disk', () => { - const missing = [...matrixFiles.keys()].filter( - (f) => !fs.existsSync(path.join(ROOT, f)) - ); - expect(missing).toEqual([]); - }); - - test('every gate-hosting paid file is in the matrix (or the documented backlog)', () => { - const gaps: string[] = []; - for (const file of paidFiles) { - const source = fs.readFileSync(path.join(ROOT, file), 'utf-8'); - const declaresGate = wholeFileTier(source) === 'gate'; - const inGateDeps = Object.entries(E2E_TOUCHFILES).some( - ([key, deps]) => - (E2E_TIERS as Record)[key] === 'gate' && - (deps as string[]).includes(file) - ); - if (!declaresGate && !inGateDeps) continue; - if (matrixFiles.has(file) || KNOWN_MATRIX_GAPS.has(file)) continue; - gaps.push(file); - } - expect( - gaps, - `Gate-hosting test file(s) missing from the evals.yml matrix — CI will ` + - `never run them and "gate tier blocks merge" becomes silently false. ` + - `Add a matrix row (with tier: gate when the file is whole-file gated). ` + - `Do NOT extend KNOWN_MATRIX_GAPS for new files.` - ).toEqual([]); - }); - - test('matrix rows for whole-file-gated files carry a matching tier property', () => { - const mismatches: string[] = []; - for (const [file, row] of matrixFiles) { - if (!fs.existsSync(path.join(ROOT, file))) continue; - const declared = wholeFileTier(fs.readFileSync(path.join(ROOT, file), 'utf-8')); - if (!declared) continue; - if (row.tier === declared) continue; - if (KNOWN_TIER_UNSET.get(file) === declared && row.tier === undefined) continue; - mismatches.push(`${file} declares '${declared}' but row '${row.name}' has tier: ${row.tier ?? 'unset'}`); - } - expect( - mismatches, - `A whole-file tier guard with no matching row tier means the suite ` + - `self-skips and the CI job reports a hollow green. Set tier: ` + - `on the row (the Run step exports it as EVALS_TIER).` - ).toEqual([]); - }); - - test('burn-down lists hold only live gaps (ratchet cleanup enforcement)', () => { - const staleGaps = [...KNOWN_MATRIX_GAPS].filter( - (f) => matrixFiles.has(f) || !fs.existsSync(path.join(ROOT, f)) - ); - expect( - staleGaps, - 'Entry fixed or file removed — delete it from KNOWN_MATRIX_GAPS.' - ).toEqual([]); - const staleTiers = [...KNOWN_TIER_UNSET.entries()].filter(([f, declared]) => { - const row = matrixFiles.get(f); - if (!row) return true; // row deleted — entry no longer applies - if (row.tier === declared) return true; // fixed — entry must go - if (!fs.existsSync(path.join(ROOT, f))) return true; - return wholeFileTier(fs.readFileSync(path.join(ROOT, f), 'utf-8')) !== declared; - }); - expect( - staleTiers.map(([f]) => f), - 'Entry fixed, row removed, or guard changed — delete it from KNOWN_TIER_UNSET.' - ).toEqual([]); - }); -}); diff --git a/test/evals-workflow-wiring.test.ts b/test/evals-workflow-wiring.test.ts new file mode 100644 index 000000000..921f45881 --- /dev/null +++ b/test/evals-workflow-wiring.test.ts @@ -0,0 +1,169 @@ +/** + * Sliced-lane wiring pins for the paid CI workflows — the successor to + * evals-workflow-matrix.test.ts, which enforced completeness of a + * hand-enumerated 17-row matrix (and carried KNOWN_MATRIX_GAPS / + * KNOWN_TIER_UNSET burn-down ratchets for the files that matrix missed). + * The matrix is deleted: the sliced lane's planner derives the gate census + * from the runner itself (collectPaidTestFiles + tier selection), so "every + * gate-hosting file is in the census" is true BY CONSTRUCTION and the + * burn-down ratchets retired with the rows. + * + * What still needs pinning is the WIRING — the yml plumbing that free tests + * are the only guard for: + * - the legacy matrix (and its `needs: evals` serialization) stays deleted, + * - planner/executor/report all run tier=gate and agree on the slice count, + * - both surviving lanes register skills through the SHARED composite that + * carries the fail-fast dangling-symlink/frontmatter verification loop + * (the sliced + periodic copies had silently dropped it — the loop was + * written after a silent "Unknown command" + 35-min-timeout incident), + * - the PR comment survives the matrix-report deletion (it moved into + * slices-report, keyed on the same "## E2E Evals" upsert marker). + */ +import { describe, test, expect } from 'bun:test'; +import * as fs from 'fs'; +import * as path from 'path'; + +const ROOT = path.join(import.meta.dir, '..'); +const read = (rel: string) => fs.readFileSync(path.join(ROOT, rel), 'utf-8'); + +const evalsYml = read('.github/workflows/evals.yml'); +const periodicYml = read('.github/workflows/evals-periodic.yml'); +const registerAction = read('.github/actions/register-gstack-skills/action.yml'); + +/** Slice count the planner emits (`--slices N`) in a workflow source. */ +function plannedSlices(source: string): number[] { + return [...source.matchAll(/--emit-plan\s+\S+\s+--slices\s+(\d+)/g)].map((m) => Number(m[1])); +} + +/** The executor matrix's slice list (`slice: [1, 2, ...]`). */ +function matrixSlices(source: string): number[][] { + return [...source.matchAll(/^\s+slice: \[([\d,\s]+)\]\s*$/gm)].map((m) => + m[1].split(',').map((n) => Number(n.trim())), + ); +} + +describe('evals.yml sliced-lane wiring (post-matrix)', () => { + test('the legacy matrix job stays deleted', () => { + // Row-enumeration shapes from the deleted matrix. Any reappearance means + // someone is re-growing a hand-maintained enumeration next to a lane + // whose census is derived — the drift class the deletion killed. + expect(evalsYml).not.toMatch(/^\s+suite:\s*$/m); + expect(evalsYml).not.toMatch(/^\s+file: test\//m); + expect(evalsYml).not.toContain('needs: [build-image, evals]'); + expect(evalsYml).not.toMatch(/^\s+needs: evals\s*$/m); + }); + + test('no workflow-level EVALS_TIER env (each command sets its own)', () => { + // The workflow-level `EVALS_TIER: gate` was dead config once every + // consumer set its own; a resurrected copy would silently leak gate + // semantics into steps that must choose explicitly. + expect(evalsYml).not.toMatch(/^env:[\s\S]{0,120}^\s+EVALS_TIER:/m); + }); + + test('planner, executors, and report all run tier=gate on the shared runner', () => { + expect(evalsYml).toMatch(/EVALS_TIER=gate bun run scripts\/test-paid-shards\.ts --tier gate --emit-plan/); + expect(evalsYml).toMatch(/EVALS_TIER=gate bun run scripts\/test-paid-shards\.ts --tier gate --plan .* --slice /); + expect(evalsYml).toMatch(/EVALS_TIER=gate bun run scripts\/test-paid-shards\.ts --tier gate --report /); + }); + + test('executor matrix slice list matches the planner --slices count', () => { + const planned = plannedSlices(evalsYml); + const matrices = matrixSlices(evalsYml); + expect(planned, 'expected exactly one --emit-plan site in evals.yml').toHaveLength(1); + expect(matrices, 'expected exactly one slice matrix in evals.yml').toHaveLength(1); + const n = planned[0]; + expect(matrices[0]).toEqual(Array.from({ length: n }, (_, i) => i + 1)); + }); + + test('reconcile exit is captured via PIPESTATUS, never $? after a pipe', () => { + // GitHub's default run-step shell is `bash -e {0}` with NO pipefail, so + // `$?` after `... | tee` is tee's exit — always 0. That made the + // fail-closed reconcile gate silently fail-open (ship review army, + // 2026-08-31). Both lanes must read PIPESTATUS[0]. + for (const [name, source] of [['evals.yml', evalsYml], ['evals-periodic.yml', periodicYml]] as const) { + const reconcileBlocks = [...source.matchAll(/--report[^\n]*\| tee[^\n]*\n([\s\S]{0,400}?)GITHUB_OUTPUT/g)]; + expect(reconcileBlocks.length, `${name}: expected a tee'd reconcile step`).toBeGreaterThanOrEqual(1); + for (const block of reconcileBlocks) { + expect(block[1], `${name} reconcile captures tee's exit, not the runner's`).toContain('PIPESTATUS[0]'); + expect(block[1]).not.toMatch(/exit=\$\?/); + } + } + }); + + test('the PR comment survived the matrix-report deletion (moved to slices-comment)', () => { + // Keyed on the upsert marker so the migration keeps updating the SAME + // comment; and the job holding it needs the issues permission (#1802). + expect(evalsYml).toContain('## E2E Evals'); + expect(evalsYml).toMatch(/pull-requests: write/); + expect(evalsYml).toMatch(/issues: write/); + }); + + test('the write-token job runs ZERO repo code (token/exec separation)', () => { + // slices-report executes PR-authored code (bun install + the reconcile + // runner), so it must hold contents:read ONLY; the write token lives in + // slices-comment, which may only download artifacts and run jq/gh — + // $GITHUB_ENV persistence is job-scoped, so this split IS the trust + // boundary (codex adversarial, 2026-08-31; the matrix-era report job had + // this property and the consolidation briefly regressed it). + const commentJob = evalsYml.slice(evalsYml.indexOf(' slices-comment:')); + expect(commentJob.length).toBeGreaterThan(100); + expect(commentJob).not.toContain('actions/checkout'); + expect(commentJob).not.toContain('bun install'); + expect(commentJob).not.toMatch(/run: .*bun run/); + expect(commentJob).not.toContain('uses: ./'); + // No checkout also means no git context: `gh pr comment` resolves the + // repo FROM git and dies with "not a git repository" here (PR #2746's + // first run). Every comment call must be explicit-repo REST (gh api). + expect(commentJob).not.toContain('gh pr comment'); + // And the code-executing report job must NOT hold write scopes. + const reportJob = evalsYml.slice(evalsYml.indexOf(' slices-report:'), evalsYml.indexOf(' slices-comment:')); + expect(reportJob).not.toMatch(/pull-requests: write/); + expect(reportJob).not.toMatch(/issues: write/); + }); +}); + +describe('evals-periodic.yml sliced-lane wiring', () => { + test('planner/executor/report tier=periodic and slice counts agree', () => { + expect(periodicYml).toMatch(/EVALS_TIER=periodic bun run scripts\/test-paid-shards\.ts --tier periodic --emit-plan/); + expect(periodicYml).toMatch(/EVALS_TIER=periodic bun run scripts\/test-paid-shards\.ts --tier periodic --plan .* --slice /); + expect(periodicYml).toMatch(/EVALS_TIER=periodic bun run scripts\/test-paid-shards\.ts --tier periodic --report /); + const planned = plannedSlices(periodicYml); + const matrices = matrixSlices(periodicYml); + expect(planned).toHaveLength(1); + expect(matrices).toHaveLength(1); + expect(matrices[0]).toEqual(Array.from({ length: planned[0] }, (_, i) => i + 1)); + }); +}); + +describe('shared setup composites (both surviving lanes)', () => { + test('both lanes register skills through the shared composite', () => { + for (const [name, source] of [['evals.yml', evalsYml], ['evals-periodic.yml', periodicYml]] as const) { + expect(source, `${name} must use the register-gstack-skills composite`) + .toContain('uses: ./.github/actions/register-gstack-skills'); + // No inline re-implementation creeping back beside the composite. + expect(source, `${name} re-inlines the skill registry instead of using the composite`) + .not.toContain('ln -snf "$REPO" "$SKILLS_DIR/gstack"'); + } + }); + + test('the register composite carries the fail-fast verification loop', () => { + // The loop is the POINT of the composite: a dangling symlink or renamed + // committed target fails in seconds with a named path, never as a wedged + // PTY session at the shard wall. Pin its load-bearing markers. + expect(registerAction).toContain('skill registry OK'); + expect(registerAction).toContain('skill-registry target missing'); + expect(registerAction).toContain('gstack root symlink dangles'); + expect(registerAction).toMatch(/grep -m1 "\^name: \$s\\\$"/); + }); + + test('seed/deps/temp composites exist and both lanes use them', () => { + for (const action of ['seed-claude-config', 'restore-deps', 'fix-bun-temp']) { + expect(fs.existsSync(path.join(ROOT, '.github', 'actions', action, 'action.yml')), `missing composite: ${action}`).toBe(true); + } + for (const [name, source] of [['evals.yml', evalsYml], ['evals-periodic.yml', periodicYml]] as const) { + expect(source, `${name} must use seed-claude-config`).toContain('uses: ./.github/actions/seed-claude-config'); + expect(source, `${name} must use restore-deps`).toContain('uses: ./.github/actions/restore-deps'); + expect(source, `${name} must use fix-bun-temp`).toContain('uses: ./.github/actions/fix-bun-temp'); + } + }); +}); diff --git a/test/flake-ledger.test.ts b/test/flake-ledger.test.ts new file mode 100644 index 000000000..8c2da4b5a --- /dev/null +++ b/test/flake-ledger.test.ts @@ -0,0 +1,64 @@ +/** + * WS1 flake-ledger unit tests: the free runner's FLAKY-PASS events become a + * durable JSONL series (single writer: the parent runner). Fail-open is the + * contract — a broken ledger warns loudly but must never turn a real verdict + * into a failure on the only required lane. + */ +import { describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { appendFlakeLedger, flakeLedgerPath, type FlakeLedgerEntry } from '../scripts/test-free-shards'; + +const entry = (file: string): FlakeLedgerEntry => ({ + ts: '2026-08-31T00:00:00.000Z', + runner: 'free', + kind: 'flaky-pass', + file, + shard: 2, +}); + +describe('flake ledger', () => { + test('appends one JSONL line per entry, creating parent dirs', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flake-ledger-')); + const ledger = path.join(dir, 'nested', 'ledger.jsonl'); + expect(appendFlakeLedger([entry('test/a.test.ts')], ledger)).toBe(true); + expect(appendFlakeLedger([entry('test/b.test.ts'), entry('test/c.test.ts')], ledger)).toBe(true); + const lines = fs.readFileSync(ledger, 'utf-8').trim().split('\n'); + expect(lines).toHaveLength(3); + expect(JSON.parse(lines[0])).toMatchObject({ runner: 'free', kind: 'flaky-pass', file: 'test/a.test.ts', shard: 2 }); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + test('empty entry list is a no-op success (no file created)', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flake-ledger-')); + const ledger = path.join(dir, 'ledger.jsonl'); + expect(appendFlakeLedger([], ledger)).toBe(true); + expect(fs.existsSync(ledger)).toBe(false); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + test('FAIL-OPEN: an unwritable path warns and returns false, never throws', () => { + const warnings: string[] = []; + // A path whose parent is a FILE cannot be mkdir'd — deterministic EEXIST/ENOTDIR. + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flake-ledger-')); + const blocker = path.join(dir, 'blocker'); + fs.writeFileSync(blocker, 'not a dir'); + const ledger = path.join(blocker, 'ledger.jsonl'); + const ok = appendFlakeLedger([entry('test/a.test.ts')], ledger, (l) => warnings.push(l)); + expect(ok).toBe(false); + expect(warnings).toHaveLength(1); + expect(warnings[0]).toContain('verdict unaffected'); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + test('env override wins; local default is project-scoped, never machine-global', () => { + expect(flakeLedgerPath({ GSTACK_FLAKE_LEDGER: '/x/y.jsonl' } as NodeJS.ProcessEnv)).toBe('/x/y.jsonl'); + // Without the env override, the default lives under the PROJECT dir + // (sibling worktrees of different repos must not interleave one series); + // tmpdir is only the last-resort fallback when slug detection fails. + const local = flakeLedgerPath({} as NodeJS.ProcessEnv); + expect(local).toMatch(/flake-ledger\.jsonl$/); + expect(local.includes(path.join('.gstack', 'projects')) || local.startsWith(os.tmpdir())).toBe(true); + }); +}); diff --git a/test/founder-resources-optout.test.ts b/test/founder-resources-optout.test.ts index 46993f0bb..8af21a970 100644 --- a/test/founder-resources-optout.test.ts +++ b/test/founder-resources-optout.test.ts @@ -30,6 +30,7 @@ afterEach(() => { fs.rmSync(tmpHome, { recursive: true, force: true }); }); function cfg(args: string[]): string { return execFileSync(CONFIG_BIN, args, { + timeout: 30_000, encoding: "utf-8", env: { ...process.env, GSTACK_HOME: tmpHome }, }).trim(); @@ -47,6 +48,7 @@ describe("founder_resources config key (#538)", () => { test("invalid values are rejected to the default, never persisted as-is", () => { execFileSync(CONFIG_BIN, ["set", "founder_resources", "banana"], { + timeout: 30_000, encoding: "utf-8", stdio: ["ignore", "pipe", "pipe"], env: { ...process.env, GSTACK_HOME: tmpHome }, }); diff --git a/test/free-tests-workflow-wiring.test.ts b/test/free-tests-workflow-wiring.test.ts index 736b391b9..30e774ef9 100644 --- a/test/free-tests-workflow-wiring.test.ts +++ b/test/free-tests-workflow-wiring.test.ts @@ -54,6 +54,17 @@ describe('free-tests workflow wiring', () => { } }); + test('flake telemetry stays wired: retry flag, single-writer ledger, unconditional artifact', () => { + // WS1: a timing flake must not red the required lane, but every + // flaky-pass must be recorded and uploaded — a green run is exactly when + // the evidence matters. Removing any of these silently returns flakes to + // either merge-blocking (flag off) or invisibility (ledger/artifact off). + expect(source).toMatch(/GSTACK_FREE_RETRY_FLAKY:\s*"1"/); + expect(source).toMatch(/GSTACK_FLAKE_LEDGER:\s*\$\{\{ runner\.temp \}\}\/flake-ledger\.jsonl/); + expect(source).toContain('name: flake-ledger'); + expect(source).toMatch(/name: Upload flake ledger\s*\n\s*if: always\(\)/); + }); + test('least-privilege token: contents read-only, credentials not persisted', () => { // The job executes PR-controlled code (install lifecycle scripts + the // suite itself). A default-grant GITHUB_TOKEN persisted into .git/config diff --git a/test/fs-utils.test.ts b/test/fs-utils.test.ts index 6959b8a57..f15c45199 100644 --- a/test/fs-utils.test.ts +++ b/test/fs-utils.test.ts @@ -74,11 +74,11 @@ describe("swept mkdirp sites under bun-on-Windows EEXIST semantics (#2635)", () fs.mkdirSync(work, { recursive: true }); const payload = '{"decision":"eexist probe","rationale":"r","scope":"repo","source":"user"}'; const env = { ...process.env, HOME: base }; - const first = spawnSync("bun", [DECISION_LOG, payload], { cwd: work, encoding: "utf8", env }); + const first = spawnSync("bun", [DECISION_LOG, payload], { cwd: work, encoding: "utf8", env, timeout: 30_000 }); expect(first.status).toBe(0); const second = spawnSync( "bun", ["--preload", EEXIST_PRELOAD, DECISION_LOG, payload], - { cwd: work, encoding: "utf8", env }, + { cwd: work, encoding: "utf8", env, timeout: 30_000 }, ); expect(second.status).toBe(0); expect(second.stderr ?? "").not.toContain("EEXIST"); @@ -93,7 +93,7 @@ describe("install-prepush-hook under bun-on-Windows EEXIST semantics (#2635)", ( const base = tmpdir(); try { const repo = path.join(base, "repo"); - spawnSync("git", ["init", "-q", repo]); + spawnSync("git", ["init", "-q", repo], { timeout: 30_000 }); const hookDir = path.join(repo, ".git", "hooks"); fs.mkdirSync(hookDir, { recursive: true }); const hookPath = path.join(hookDir, "pre-push"); @@ -105,6 +105,7 @@ describe("install-prepush-hook under bun-on-Windows EEXIST semantics (#2635)", ( const r = spawnSync("bun", ["--preload", EEXIST_PRELOAD, REDACT, "install-prepush-hook"], { cwd: repo, encoding: "utf8", + timeout: 30_000, }); expect(r.status).toBe(0); expect(r.stderr ?? "").not.toContain("EEXIST"); diff --git a/test/gbrain-detect-install.test.ts b/test/gbrain-detect-install.test.ts index 725eb9bdc..9109ec304 100644 --- a/test/gbrain-detect-install.test.ts +++ b/test/gbrain-detect-install.test.ts @@ -51,6 +51,7 @@ function run(bin: string, args: string[], opts: RunOpts = {}) { env, cwd: opts.cwd, encoding: 'utf-8', + timeout: 30_000, }); return { stdout: (res.stdout || '').trim(), diff --git a/test/gbrain-exec-invariant.test.ts b/test/gbrain-exec-invariant.test.ts index a0d962b4a..205fbf319 100644 --- a/test/gbrain-exec-invariant.test.ts +++ b/test/gbrain-exec-invariant.test.ts @@ -35,9 +35,12 @@ const GUARDED_FILES = [ // Patterns that would bypass lib/gbrain-exec.ts. Match the literal `"gbrain"` // as the first argument since these helpers are the failure mode. const BANNED_PATTERNS: Array<{ name: string; regex: RegExp }> = [ + // tripwire-exempt: grep-needle STRING for this invariant, not a process spawn { name: 'spawnSync("gbrain", ...)', regex: /spawnSync\s*\(\s*["']gbrain["']/g }, { name: 'spawn("gbrain", ...)', regex: /\bspawn\s*\(\s*["']gbrain["']/g }, + // tripwire-exempt: grep-needle STRING for this invariant, not a process spawn { name: 'execFileSync("gbrain", ...)', regex: /execFileSync\s*\(\s*["']gbrain["']/g }, + // tripwire-exempt: grep-needle STRING for this invariant, not a process spawn { name: 'execSync("...gbrain...")', regex: /execSync\s*\(\s*["'`][^"'`]*\bgbrain\b/g }, ]; diff --git a/test/gbrain-init-rollback.test.ts b/test/gbrain-init-rollback.test.ts index 39777e03a..747cac8a9 100644 --- a/test/gbrain-init-rollback.test.ts +++ b/test/gbrain-init-rollback.test.ts @@ -122,6 +122,7 @@ echo "ok" HOME: env.home, PATH: `${env.bindir}:/usr/bin:/bin`, }, + timeout: 30_000, }); return { exitCode: result.status ?? 1, diff --git a/test/gbrain-init-voyage-code-3.test.ts b/test/gbrain-init-voyage-code-3.test.ts index 7365d8e1c..be73e26b3 100644 --- a/test/gbrain-init-voyage-code-3.test.ts +++ b/test/gbrain-init-voyage-code-3.test.ts @@ -121,6 +121,7 @@ gbrain init --pglite --json "$@" const result = spawnSync(shell, ["-c", script], { encoding: "utf-8", env: baseEnv, + timeout: 30_000, }); if (result.status !== 0) { throw new Error(`init script exited ${result.status}: ${result.stderr}`); @@ -133,7 +134,7 @@ function lastArgc(env: FakeEnv): number { return parseInt(lines[lines.length - 1], 10); } -const HAVE_ZSH = spawnSync("zsh", ["-c", "true"]).status === 0; +const HAVE_ZSH = spawnSync("zsh", ["-c", "true"], { timeout: 30_000 }).status === 0; describe("voyage-code-3 default for gstack-driven PGLite init", () => { it("passes voyage-code-3 flags when VOYAGE_API_KEY is set", () => { @@ -193,6 +194,7 @@ gbrain init --pglite --json $GBRAIN_EMBED_FLAGS const result = spawnSync("zsh", ["-c", brokenShape], { encoding: "utf-8", env: { ...process.env, HOME: env.home, PATH: `${env.bindir}:/usr/bin:/bin` }, + timeout: 30_000, }); expect(result.status).toBe(0); expect(lastArgc(env)).toBe(4); // init, --pglite, --json, "" diff --git a/test/gbrain-lib-verify.test.ts b/test/gbrain-lib-verify.test.ts index 64c88e8f3..2a29e5af7 100644 --- a/test/gbrain-lib-verify.test.ts +++ b/test/gbrain-lib-verify.test.ts @@ -28,6 +28,7 @@ function runVerify(arg: string, stdin?: string) { const res = spawnSync(VERIFY, arg === '' ? [] : [arg], { input: stdin, encoding: 'utf-8', + timeout: 30_000, }); return { stdout: (res.stdout || '').trim(), @@ -43,6 +44,7 @@ function runLibSnippet(snippet: string, stdin: string = '') { const res = spawnSync('bash', ['-c', script], { input: stdin, encoding: 'utf-8', + timeout: 30_000, }); return { stdout: (res.stdout || '').trim(), diff --git a/test/gbrain-local-status.test.ts b/test/gbrain-local-status.test.ts index 85bebd2a0..e310362ce 100644 --- a/test/gbrain-local-status.test.ts +++ b/test/gbrain-local-status.test.ts @@ -210,6 +210,7 @@ describe("lib/gbrain-local-status — status classification", () => { ); expect(source).not.toContain('command -v gbrain'); + // tripwire-exempt: string assertion on lib source text, not a call expect(source).toContain('execFileSync("gbrain", ["--version"]'); }); @@ -538,6 +539,7 @@ describe("lib/gbrain-local-status — thin-client (#2051)", () => { GSTACK_HOME: env.gstackHome, GSTACK_DETECT_NO_CACHE: "1", }, + timeout: 30_000, }); expect(r.status).toBe(0); }); @@ -554,6 +556,7 @@ describe("lib/gbrain-local-status — thin-client (#2051)", () => { GSTACK_HOME: env.gstackHome, GSTACK_DETECT_NO_CACHE: "1", }, + timeout: 30_000, }); expect(r.status).toBe(1); }); @@ -790,6 +793,7 @@ describe("lib/gbrain-local-status — bearer-token thin-client (#2520)", () => { GSTACK_HOME: env.gstackHome, GSTACK_DETECT_NO_CACHE: "1", }, + timeout: 30_000, }); expect(r.status).toBe(0); }); diff --git a/test/gbrain-repo-policy-client.test.ts b/test/gbrain-repo-policy-client.test.ts index fd87bf227..de130e29c 100644 --- a/test/gbrain-repo-policy-client.test.ts +++ b/test/gbrain-repo-policy-client.test.ts @@ -32,7 +32,7 @@ function env(): NodeJS.ProcessEnv { } function run(args: string[], input?: string) { - const res = spawnSync(BIN, args, { env: env(), encoding: "utf-8", input }); + const res = spawnSync(BIN, args, { env: env(), encoding: "utf-8", input, timeout: 30_000 }); return { stdout: res.stdout || "", stderr: res.stderr || "", diff --git a/test/gbrain-repo-policy.test.ts b/test/gbrain-repo-policy.test.ts index 05c25f974..db6b58c2d 100644 --- a/test/gbrain-repo-policy.test.ts +++ b/test/gbrain-repo-policy.test.ts @@ -29,6 +29,7 @@ function run(args: string[], opts: { env?: Record } = {}) { const res = spawnSync(BIN, args, { env: { ...process.env, GSTACK_HOME: tmpHome, ...(opts.env || {}) }, encoding: 'utf-8', + timeout: 30_000, }); return { stdout: (res.stdout || '').trim(), @@ -263,6 +264,7 @@ describe('get without arg (auto-detect from current dir)', () => { env: { ...process.env, GSTACK_HOME: tmpHome }, cwd: cwdTmp, encoding: 'utf-8', + timeout: 30_000, }); expect((res.stdout || '').trim()).toBe('unset'); } finally { @@ -288,7 +290,7 @@ describe('gstack-gbrain-sync code stage honors the repo policy (#2140 sync path) function makeRepo(): void { repoDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-policy-repo-')); const git = (...args: string[]) => - spawnSync('git', args, { cwd: repoDir, encoding: 'utf-8' }); + spawnSync('git', args, { cwd: repoDir, encoding: 'utf-8', timeout: 30_000 }); git('init', '-q', '.'); git('remote', 'add', 'origin', REPO_URL); fs.writeFileSync(path.join(repoDir, 'README.md'), 'fixture\n'); diff --git a/test/gbrain-source-worktree-advance.test.ts b/test/gbrain-source-worktree-advance.test.ts index 3763bcf57..eaeaa4a57 100644 --- a/test/gbrain-source-worktree-advance.test.ts +++ b/test/gbrain-source-worktree-advance.test.ts @@ -23,12 +23,13 @@ function run(argv: string[], env: Record = {}) { env: { ...process.env, HOME: tmpHome, GSTACK_HOME: tmpHome, ...env }, encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); return { stdout: res.stdout || '', stderr: res.stderr || '', status: res.status ?? -1 }; } function git(args: string[], cwd: string) { - const res = spawnSync('git', args, { cwd, encoding: 'utf-8' }); + const res = spawnSync('git', args, { cwd, encoding: 'utf-8', timeout: 30_000 }); return { stdout: (res.stdout || '').trim(), status: res.status ?? -1 }; } diff --git a/test/gbrain-sync-skip.test.ts b/test/gbrain-sync-skip.test.ts index 902256b3b..ce8866a2f 100644 --- a/test/gbrain-sync-skip.test.ts +++ b/test/gbrain-sync-skip.test.ts @@ -25,7 +25,7 @@ import { join } from "path"; import { execFileSync, spawnSync } from "child_process"; const SCRIPT = join(import.meta.dir, "..", "bin", "gstack-gbrain-sync.ts"); -const BUN_BIN = execFileSync("sh", ["-c", "command -v bun"], { encoding: "utf-8" }).trim(); +const BUN_BIN = execFileSync("sh", ["-c", "command -v bun"], { encoding: "utf-8", timeout: 30_000 }).trim(); interface FakeEnv { tmp: string; @@ -112,9 +112,10 @@ function runOrchestrator( ): { stdout: string; stderr: string; exitCode: number } { // Initialize a git repo in the sandbox so repoRoot() finds it (otherwise // code stage skips with "not in git repo" before our check ever fires). - spawnSync("git", ["init", "-q", env.home], { encoding: "utf-8" }); + spawnSync("git", ["init", "-q", env.home], { encoding: "utf-8", timeout: 30_000 }); spawnSync("git", ["-C", env.home, "commit", "--allow-empty", "-m", "init", "-q"], { encoding: "utf-8", + timeout: 30_000, env: { ...process.env, GIT_AUTHOR_NAME: "T", GIT_AUTHOR_EMAIL: "t@t", GIT_COMMITTER_NAME: "T", GIT_COMMITTER_EMAIL: "t@t" }, }); diff --git a/test/gbrain-sync-voyage-code-3-integration.test.ts b/test/gbrain-sync-voyage-code-3-integration.test.ts index 268e5ec5b..f2064032b 100644 --- a/test/gbrain-sync-voyage-code-3-integration.test.ts +++ b/test/gbrain-sync-voyage-code-3-integration.test.ts @@ -44,7 +44,7 @@ import { tmpdir } from "os"; import { join } from "path"; import { spawnSync } from "child_process"; -const gbrainPath = spawnSync("which", ["gbrain"], { encoding: "utf-8" }).stdout.trim(); +const gbrainPath = spawnSync("which", ["gbrain"], { encoding: "utf-8", timeout: 30_000 }).stdout.trim(); const gbrainAvailable = gbrainPath.length > 0; const voyageKey = process.env.VOYAGE_API_KEY?.trim() ?? ""; const voyageKeyPresent = voyageKey.length > 0; @@ -114,14 +114,14 @@ The queue module is a simple FIFO job queue. ); // Make it a git repo because gbrain's code-sync strategy expects one. - const gitInit = spawnSync("git", ["init", "-q"], { cwd: fixtureDir, encoding: "utf-8" }); + const gitInit = spawnSync("git", ["init", "-q"], { cwd: fixtureDir, encoding: "utf-8", timeout: 30_000 }); if (gitInit.status !== 0) { throw new Error(`git init failed: ${gitInit.stderr}`); } - spawnSync("git", ["config", "user.email", "test@example.invalid"], { cwd: fixtureDir }); - spawnSync("git", ["config", "user.name", "test"], { cwd: fixtureDir }); - spawnSync("git", ["add", "."], { cwd: fixtureDir }); - spawnSync("git", ["commit", "-q", "-m", "fixture"], { cwd: fixtureDir }); + spawnSync("git", ["config", "user.email", "test@example.invalid"], { cwd: fixtureDir, timeout: 30_000 }); + spawnSync("git", ["config", "user.name", "test"], { cwd: fixtureDir, timeout: 30_000 }); + spawnSync("git", ["add", "."], { cwd: fixtureDir, timeout: 30_000 }); + spawnSync("git", ["commit", "-q", "-m", "fixture"], { cwd: fixtureDir, timeout: 30_000 }); return { root, diff --git a/test/gemini-e2e.test.ts b/test/gemini-e2e.test.ts index eb3c49b00..ed1270a61 100644 --- a/test/gemini-e2e.test.ts +++ b/test/gemini-e2e.test.ts @@ -19,7 +19,7 @@ import { JUDGE_MS } from './helpers/eval-budgets'; import { runGeminiSkill } from './helpers/gemini-session-runner'; import type { GeminiResult } from './helpers/gemini-session-runner'; import { EvalCollector } from './helpers/eval-store'; -import { selectTests, detectBaseBranch, getChangedFiles, GLOBAL_TOUCHFILES } from './helpers/touchfiles'; +import { selectTests, detectBaseBranch, getChangedFiles, E2E_TOUCHFILES, GLOBAL_TOUCHFILES } from './helpers/touchfiles'; import { createTestWorktree, harvestAndCleanup } from './helpers/e2e-helpers'; import * as path from 'path'; @@ -29,7 +29,7 @@ const ROOT = path.resolve(import.meta.dir, '..'); const GEMINI_AVAILABLE = (() => { try { - const result = Bun.spawnSync(['which', 'gemini']); + const result = Bun.spawnSync(['which', 'gemini'], { timeout: 30_000 }); return result.exitCode === 0; } catch { return false; } })(); @@ -74,10 +74,16 @@ if (!evalsEnabled) { // --- Diff-based test selection --- -// Gemini E2E touchfiles — keyed by test name -const GEMINI_E2E_TOUCHFILES: Record = { - 'gemini-smoke': ['.agents/skills/**', 'test/helpers/gemini-session-runner.ts'], -}; +// Gemini E2E touchfiles — DERIVED from the canonical map, never a local fork +// (the old hand-copy kept a gitignored '.agents/skills/**' pattern that can +// never match a git diff and missed canonical deps — same drift class as the +// codex copy). +const GEMINI_E2E_TOUCHFILES: Record = Object.fromEntries( + (['gemini-smoke'] as const).map((key) => { + if (!E2E_TOUCHFILES[key]) throw new Error(`canonical E2E_TOUCHFILES lost key '${key}' — fix the map, not this file`); + return [key, E2E_TOUCHFILES[key]]; + }), +); let selectedTests: string[] | null = null; // null = run all diff --git a/test/gen-skill-docs-import-purity.test.ts b/test/gen-skill-docs-import-purity.test.ts index e109b73ba..f3f104798 100644 --- a/test/gen-skill-docs-import-purity.test.ts +++ b/test/gen-skill-docs-import-purity.test.ts @@ -39,7 +39,7 @@ describe('gen-skill-docs import purity', () => { } console.log('IMPORT_PURE'); `; - const out = Bun.spawnSync(['bun', '-e', probe], { cwd: ROOT }); + const out = Bun.spawnSync(['bun', '-e', probe], { cwd: ROOT, timeout: 120_000 }); const stdout = out.stdout.toString(); const stderr = out.stderr.toString(); expect(stderr, stderr).not.toContain('import mutated'); diff --git a/test/gen-skill-docs-out-dir.test.ts b/test/gen-skill-docs-out-dir.test.ts index 5cbff0fb2..f957d89be 100644 --- a/test/gen-skill-docs-out-dir.test.ts +++ b/test/gen-skill-docs-out-dir.test.ts @@ -21,7 +21,7 @@ describe('gen-skill-docs --out-dir (B2 render isolation)', () => { } function porcelain(): string { - const r = spawnSync('git', ['status', '--porcelain'], { cwd: ROOT, encoding: 'utf-8' }); + const r = spawnSync('git', ['status', '--porcelain'], { cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); return r.status === 0 ? r.stdout : ''; } diff --git a/test/gen-skill-docs.test.ts b/test/gen-skill-docs.test.ts index 961fa2a5e..dc7c66cae 100644 --- a/test/gen-skill-docs.test.ts +++ b/test/gen-skill-docs.test.ts @@ -139,7 +139,7 @@ const EXTERNAL_OUT = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-gen-docs-out- { const render = Bun.spawnSync( ['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--out-dir', EXTERNAL_OUT], - { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }, + { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 }, ); if (render.exitCode !== 0) { throw new Error( @@ -315,6 +315,7 @@ describe('gen-skill-docs', () => { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + timeout: 120_000, }); expect(result.exitCode).toBe(0); const output = result.stdout.toString(); @@ -1965,7 +1966,7 @@ describe('Codex generation (--host codex)', () => { '/tmp/gstack-claude-error-XXXXXX', '/tmp/gstack-claude-diff-XXXXXX', ]) { - const result = spawnSync('mktemp', [template], { encoding: 'utf-8' }); + const result = spawnSync('mktemp', [template], { encoding: 'utf-8', timeout: 30_000 }); expect(result.status).toBe(0); const created = result.stdout.trim(); expect(created.startsWith(template.replace('XXXXXX', ''))).toBe(true); @@ -1990,6 +1991,7 @@ describe('Codex generation (--host codex)', () => { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + timeout: 120_000, }); expect(result.exitCode).toBe(0); const output = result.stdout.toString(); @@ -2005,11 +2007,13 @@ describe('Codex generation (--host codex)', () => { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + timeout: 120_000, }); const agentsResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'agents', '--dry-run', '--out-dir', EXTERNAL_OUT], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + timeout: 120_000, }); expect(codexResult.exitCode).toBe(0); expect(agentsResult.exitCode).toBe(0); @@ -2224,6 +2228,7 @@ describe('Codex generation (--host codex)', () => { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + timeout: 120_000, }); expect(override.exitCode).toBe(0); const content = fs.readFileSync(path.join(overrideOut, '.agents', 'skills', 'gstack-ship', 'SKILL.md'), 'utf-8'); @@ -2342,10 +2347,10 @@ describe('Factory generation (--host factory)', () => { test('--host droid alias works', () => { const factoryResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], { - cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000, }); const droidResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'droid', '--dry-run', '--out-dir', EXTERNAL_OUT], { - cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000, }); expect(factoryResult.exitCode).toBe(0); expect(droidResult.exitCode).toBe(0); @@ -2354,7 +2359,7 @@ describe('Factory generation (--host factory)', () => { test('--host factory --dry-run freshness', () => { const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], { - cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000, }); expect(result.exitCode).toBe(0); const output = result.stdout.toString(); @@ -2432,7 +2437,7 @@ describe('Parameterized host smoke tests', () => { test('--dry-run freshness check passes', () => { const result = Bun.spawnSync( ['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', hostConfig.name, '--dry-run', '--out-dir', EXTERNAL_OUT], - { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' } + { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 } ); expect(result.exitCode).toBe(0); const output = result.stdout.toString(); @@ -2457,7 +2462,7 @@ describe('--host all', () => { // claude host plus every external host regenerate deterministically. test('--host all generates for all registered hosts', () => { const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--dry-run', '--out-dir', EXTERNAL_OUT], { - cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000, }); expect(result.exitCode).toBe(0); const output = result.stdout.toString(); diff --git a/test/git-ref-fixture-tripwire.test.ts b/test/git-ref-fixture-tripwire.test.ts new file mode 100644 index 000000000..bffc3f581 --- /dev/null +++ b/test/git-ref-fixture-tripwire.test.ts @@ -0,0 +1,64 @@ +/** + * Git-ref fixture tripwire: no test or helper may pin repo content to a raw + * commit SHA (the `git show :path` fixture pattern). + * + * The class: test/helpers/auq-sdk-capture.ts defaulted verboseSkill() to + * `git show ab66193e^:plan-ceo-review/SKILL.md` — a BRANCH-LOCAL ref. That + * fixture dies the day the branch is pruned, and already failed on shallow + * clones (CI executors fetch-depth-0 exists precisely because self-derived + * selection crashed on shallow checkouts). The v1.75 precedent is to VENDOR + * the frozen content under test/fixtures/ instead — content-addressed by the + * repo itself, immune to ref pruning and clone depth. + * + * Scans test trees + helpers for two shapes: + * - a quoted `{7,40}[^]?:` rev-path (the `git show SHA:path` form) + * - a gitRef-style default parameter carrying a raw hex SHA + */ +import { describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +const ROOT = path.resolve(import.meta.dir, '..'); +const SCAN_ROOTS = ['test', 'browse/test', 'design/test', 'make-pdf/test']; +const SELF = path.join('test', 'git-ref-fixture-tripwire.test.ts'); + +// Quoted `SHA:` rev-path (7-40 hex chars, optional ^/~ suffix, then colon) — +// requires >= 2 digits among the hex so ordinary words ('deadbeef' aside) +// and pure-alpha identifiers don't false-positive. +const REV_PATH = /['"`]([0-9a-f]{7,40})[\^~]?:/g; +const GIT_REF_DEFAULT = /gitRef\s*=\s*['"`][0-9a-f]{7,40}/; + +const looksLikeSha = (s: string): boolean => /[0-9]/.test(s) && /[a-f]/.test(s); + +describe('git-ref fixture tripwire', () => { + test('no raw-SHA fixture refs in the test trees (vendor the content instead)', () => { + const hits: string[] = []; + for (const root of SCAN_ROOTS) { + const abs = path.join(ROOT, root); + if (!fs.existsSync(abs)) continue; + const stack = [abs]; + while (stack.length > 0) { + const dir = stack.pop()!; + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) { stack.push(full); continue; } + if (!/\.(?:[cm]?[jt]s|tsx)$/.test(entry.name)) continue; + const rel = path.relative(ROOT, full); + if (rel === SELF) continue; + const src = fs.readFileSync(full, 'utf-8'); + src.split('\n').forEach((line, i) => { + for (const m of line.matchAll(REV_PATH)) { + if (looksLikeSha(m[1])) hits.push(`${rel}:${i + 1} ${line.trim().slice(0, 100)}`); + } + if (GIT_REF_DEFAULT.test(line)) hits.push(`${rel}:${i + 1} ${line.trim().slice(0, 100)}`); + }); + } + } + } + expect( + hits, + `raw-SHA fixture reference(s) — these die on branch prune and fail on shallow clones. ` + + `Vendor the frozen content under test/fixtures/ instead (v1.75 precedent):\n ${hits.join('\n ')}`, + ).toEqual([]); + }); +}); diff --git a/test/global-discover.test.ts b/test/global-discover.test.ts index f433da8c4..48b09e72e 100644 --- a/test/global-discover.test.ts +++ b/test/global-discover.test.ts @@ -180,10 +180,11 @@ describe("gstack-global-discover", () => { // Create a git repo as the session target const repoDir = join(tmpDir, "fake-repo"); mkdirSync(repoDir); - spawnSync("git", ["init"], { cwd: repoDir, stdio: "pipe" }); + spawnSync("git", ["init"], { cwd: repoDir, stdio: "pipe", timeout: 30_000 }); spawnSync("git", ["commit", "--allow-empty", "-m", "init"], { cwd: repoDir, stdio: "pipe", + timeout: 30_000, }); // Write a session with a 20KB first line (simulates Codex v0.117+) diff --git a/test/gstack-artifacts-init.test.ts b/test/gstack-artifacts-init.test.ts index a125562c3..b1dd3c84b 100644 --- a/test/gstack-artifacts-init.test.ts +++ b/test/gstack-artifacts-init.test.ts @@ -119,7 +119,7 @@ exit 0 * test focused on artifacts-init's branching logic, not git plumbing. */ function makeFakeGit() { - const realGit = spawnSync('which', ['git'], { encoding: 'utf-8' }).stdout.trim(); + const realGit = spawnSync('which', ['git'], { encoding: 'utf-8', timeout: 30_000 }).stdout.trim(); const script = `#!/bin/bash # Walk argv past leading -C and similar flags to find the real subcommand. args=("$@") @@ -157,6 +157,7 @@ function run(argv: string[], opts: { env?: Record; input?: strin encoding: 'utf-8', input: opts.input, cwd: ROOT, + timeout: 30_000, }); return { stdout: res.stdout || '', @@ -176,7 +177,7 @@ beforeEach(() => { fakeBinDir = fs.mkdtempSync(path.join(os.tmpdir(), 'artifacts-fake-bin-')); ghCallLog = path.join(fakeBinDir, 'gh-calls.log'); glabCallLog = path.join(fakeBinDir, 'glab-calls.log'); - spawnSync('git', ['init', '--bare', '-q', '-b', 'main', bareRemote]); + spawnSync('git', ['init', '--bare', '-q', '-b', 'main', bareRemote], { timeout: 30_000 }); makeFakeGit(); }); @@ -277,7 +278,7 @@ describe('gstack-artifacts-init canonical URL storage (codex Finding #10)', () = makeFakeGh({ webUrl: 'https://github.com/testuser/gstack-artifacts-testuser' }); const r = run(['--host', 'github']); expect(r.status).toBe(0); - const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' }); + const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 }); expect(remote.stdout.trim()).toBe('https://github.com/testuser/gstack-artifacts-testuser'); }); @@ -288,7 +289,7 @@ describe('gstack-artifacts-init canonical URL storage (codex Finding #10)', () = }); const r = run(['--host', 'github']); expect(r.status).toBe(0); - const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' }); + const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 }); expect(remote.stdout.trim()).toBe('git@github.com:testuser/gstack-artifacts-testuser.git'); }); @@ -296,7 +297,7 @@ describe('gstack-artifacts-init canonical URL storage (codex Finding #10)', () = makeFakeGh({ gitProtocol: 'unset' }); const r = run(['--host', 'github']); expect(r.status).toBe(0); - const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' }); + const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 }); expect(remote.stdout.trim()).toBe('https://github.com/testuser/gstack-artifacts-testuser'); }); @@ -304,7 +305,7 @@ describe('gstack-artifacts-init canonical URL storage (codex Finding #10)', () = makeFakeGlab({ gitProtocol: 'ssh' }); const r = run(['--host', 'gitlab']); expect(r.status).toBe(0); - const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' }); + const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 }); expect(remote.stdout.trim()).toBe('git@gitlab.com:testuser/gstack-artifacts-testuser.git'); }); }); @@ -359,7 +360,7 @@ describe('gstack-artifacts-init idempotency', () => { makeFakeGh({ gitProtocol: 'ssh' }); const r = run(['--remote', 'https://github.com/testuser/gstack-artifacts-testuser']); expect(r.status).toBe(0); - const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' }); + const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 }); expect(remote.stdout.trim()).toBe('https://github.com/testuser/gstack-artifacts-testuser'); }); @@ -372,7 +373,7 @@ describe('gstack-artifacts-init idempotency', () => { 'ssh', ]); expect(r.status).toBe(0); - const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' }); + const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 }); expect(remote.stdout.trim()).toBe('git@github.com:testuser/gstack-artifacts-testuser.git'); }); diff --git a/test/gstack-artifacts-url.test.ts b/test/gstack-artifacts-url.test.ts index 133ed7fc8..76f3e2cc0 100644 --- a/test/gstack-artifacts-url.test.ts +++ b/test/gstack-artifacts-url.test.ts @@ -13,7 +13,7 @@ const ROOT = path.resolve(import.meta.dir, '..'); const URL_BIN = path.join(ROOT, 'bin', 'gstack-artifacts-url'); function run(args: string[]): { code: number; stdout: string; stderr: string } { - const r = spawnSync(URL_BIN, args, { encoding: 'utf-8' }); + const r = spawnSync(URL_BIN, args, { encoding: 'utf-8', timeout: 30_000 }); return { code: r.status ?? -1, stdout: (r.stdout || '').trim(), diff --git a/test/gstack-codex-session-import.test.ts b/test/gstack-codex-session-import.test.ts index 7cd32e949..be48c75ae 100644 --- a/test/gstack-codex-session-import.test.ts +++ b/test/gstack-codex-session-import.test.ts @@ -68,7 +68,7 @@ function runImport(sessionPath: string): { stdout: string; stderr: string; statu env.GSTACK_STATE_ROOT = stateRoot; env.GSTACK_QUESTION_LOG_NO_DERIVE = '1'; delete env.GSTACK_HOME; - const res = spawnSync(BIN, [sessionPath], { env, encoding: 'utf-8', cwd: ROOT }); + const res = spawnSync(BIN, [sessionPath], { env, encoding: 'utf-8', cwd: ROOT, timeout: 30_000 }); return { stdout: res.stdout ?? '', stderr: res.stderr ?? '', @@ -196,7 +196,7 @@ describe('default mode (no args → latest)', () => { } env.GSTACK_STATE_ROOT = stateRoot; env.CODEX_SESSIONS_ROOT = emptyDir; - const res = spawnSync(BIN, [], { env, encoding: 'utf-8', cwd: ROOT }); + const res = spawnSync(BIN, [], { env, encoding: 'utf-8', cwd: ROOT, timeout: 30_000 }); expect(res.status).toBe(0); expect(res.stdout).toMatch(/NO_SESSIONS/); } finally { diff --git a/test/gstack-config-defaults.test.ts b/test/gstack-config-defaults.test.ts index cb8febadd..781ee64b9 100644 --- a/test/gstack-config-defaults.test.ts +++ b/test/gstack-config-defaults.test.ts @@ -45,6 +45,7 @@ const STATE = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-config-test-')); function get(key: string): { out: string; code: number } { const r = spawnSync('bash', [CONFIG_BIN, 'get', key], { encoding: 'utf-8', + timeout: 30_000, env: { ...process.env, GSTACK_STATE_ROOT: STATE }, }); return { out: r.stdout ?? '', code: r.status ?? -1 }; diff --git a/test/gstack-config-key-locale.test.ts b/test/gstack-config-key-locale.test.ts index 94d60cd6b..599a0c168 100644 --- a/test/gstack-config-key-locale.test.ts +++ b/test/gstack-config-key-locale.test.ts @@ -28,6 +28,7 @@ function run(args: string[]) { // live tree (observed in the free-tests CI job). Relink behavior itself is // covered in isolation by test/relink.test.ts's mock install. env: { ...process.env, GSTACK_STATE_ROOT: stateRoot, GSTACK_SETUP_RUNNING: "1" }, + timeout: 30_000, }); return { diff --git a/test/gstack-config-redact-keys.test.ts b/test/gstack-config-redact-keys.test.ts index 9290d478d..9a73571a5 100644 --- a/test/gstack-config-redact-keys.test.ts +++ b/test/gstack-config-redact-keys.test.ts @@ -16,6 +16,7 @@ function cfg(args: string[]): { code: number; out: string; err: string } { const r = spawnSync(CONFIG, args, { encoding: "utf8", env: { ...process.env, GSTACK_HOME: home }, + timeout: 30_000, }); return { code: r.status ?? 0, out: r.stdout ?? "", err: r.stderr ?? "" }; } diff --git a/test/gstack-decision-bins.test.ts b/test/gstack-decision-bins.test.ts index a99209039..82c28ac23 100644 --- a/test/gstack-decision-bins.test.ts +++ b/test/gstack-decision-bins.test.ts @@ -20,18 +20,18 @@ function opts(): ExecSyncOptionsWithStringEncoding { } function log(arg: string, expectFail = false): { out: string; code: number } { try { - return { out: execSync(`${LOG} '${arg.replace(/'/g, "'\\''")}'`, opts()).trim(), code: 0 }; + return { out: execSync(`${LOG} '${arg.replace(/'/g, "'\\''")}'`, opts()).trim(), code: 0 }; // timeout via opts() } catch (e: any) { if (expectFail) return { out: (e.stderr?.toString() || "").trim(), code: e.status || 1 }; throw e; } } function logFlag(flag: string): string { - return execSync(`${LOG} ${flag}`, opts()).trim(); + return execSync(`${LOG} ${flag}`, opts()).trim(); // timeout via opts() } function search(args = ""): string { try { - return execSync(`${SEARCH} ${args}`, opts()).trim(); + return execSync(`${SEARCH} ${args}`, opts()).trim(); // timeout via opts() } catch { return ""; } diff --git a/test/gstack-detach.test.ts b/test/gstack-detach.test.ts index 926bf253a..517bf6d45 100644 --- a/test/gstack-detach.test.ts +++ b/test/gstack-detach.test.ts @@ -15,13 +15,13 @@ const ROOT = path.resolve(import.meta.dir, '..'); const DETACH = path.join(ROOT, 'bin', 'gstack-detach'); function ownPgid(): string { - return (spawnSync('ps', ['-o', 'pgid=', '-p', String(process.pid)], { encoding: 'utf-8' }).stdout || '').trim(); + return (spawnSync('ps', ['-o', 'pgid=', '-p', String(process.pid)], { encoding: 'utf-8', timeout: 30_000 }).stdout || '').trim(); } function waitFor(pred: () => boolean, ms: number): boolean { const end = Date.now() + ms; while (Date.now() < end) { if (pred()) return true; - spawnSync('sleep', ['0.2']); + spawnSync('sleep', ['0.2'], { timeout: 30_000 }); } return pred(); } @@ -69,6 +69,59 @@ describe('gstack-detach', () => { } finally { fs.rmSync(dir, { recursive: true, force: true }); } }, 16000); + test('watchdog group-SIGKILLs TERM-immune grandchildren (no orphan survives)', () => { + // Regression pin for the 2026-08 escalation change: the watchdog used to + // follow its killpg(SIGTERM) + 5s grace with a DIRECT proc.kill() — a + // grandchild that ignores TERM survived and burned cores/API for hours + // (the observed 15-hour-orphan class). Now the grace escalates to + // killpg(SIGKILL). The child here traps TERM and spawns a TERM-immune + // grandchild; only a GROUP SIGKILL clears both. Markers are per-run + // unique (pid) so concurrent worktree suites can't cross-kill. + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gd-')); + const log = path.join(dir, 'run.log'); + const g1 = `6091.${process.pid}`; + const g2 = `6092.${process.pid}`; + const alive = (m: string) => spawnSync('pgrep', ['-f', `sleep ${m.replace('.', '\\.')}`], { stdio: 'pipe', timeout: 5_000 }).status === 0; + try { + spawnSync(DETACH, ['--log', log, '--timeout', '1', '--', 'bash', '-c', + `trap '' TERM; (trap '' TERM; sleep ${g1}) & exec sleep ${g2}`], + { encoding: 'utf-8', timeout: 10000 }); + expect(waitFor(() => logHas(log, '### gstack-detach EXIT=timeout ###'), 15000)).toBe(true); + // Grace is 5s after the TERM that both processes ignore — the SIGKILL + // escalation must clear the whole group shortly after the sentinel. + expect(waitFor(() => !alive(g1) && !alive(g2), 10000), + 'TERM-immune child/grandchild survived the watchdog — killpg(SIGKILL) escalation regressed').toBe(true); + } finally { + spawnSync('pkill', ['-9', '-f', `sleep 609[12]\\.${process.pid}`], { stdio: 'ignore', timeout: 5_000 }); + fs.rmSync(dir, { recursive: true, force: true }); + } + }, 30000); + + test('watchdog kills the grandchild even when the LEADER dies on the SIGTERM', () => { + // The pgid-after-grace bug: killpg(getpgid(proc.pid), SIGKILL) raised + // ESRCH once the leader had honored the TERM, and the except fell back + // to proc.kill() on a corpse — the TERM-immune grandchild lived forever. + // The fix captures the pgid AT SPAWN. This variant is the one the + // TERM-immune-leader test above cannot see. + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gd-')); + const log = path.join(dir, 'run.log'); + const g = `6093.${process.pid}`; + const alive = () => spawnSync('pgrep', ['-f', `sleep ${g.replace('.', '\\.')}`], { stdio: 'pipe', timeout: 5_000 }).status === 0; + try { + // Leader: no trap — dies on the watchdog's SIGTERM. Grandchild: + // TERM-immune, same group — only a saved-pgid SIGKILL reaches it. + spawnSync(DETACH, ['--log', log, '--timeout', '1', '--', 'bash', '-c', + `(trap '' TERM; sleep ${g}) & sleep 60`], + { encoding: 'utf-8', timeout: 10000 }); + expect(waitFor(() => logHas(log, '### gstack-detach EXIT=timeout ###'), 15000)).toBe(true); + expect(waitFor(() => !alive(), 10000), + 'grandchild survived a dead leader — the pgid must be captured at spawn, not resolved after the grace').toBe(true); + } finally { + spawnSync('pkill', ['-9', '-f', `sleep 6093\\.${process.pid}`], { stdio: 'ignore', timeout: 5_000 }); + fs.rmSync(dir, { recursive: true, force: true }); + } + }, 30000); + test('machine --lock serializes concurrent runs (second WAITS for the first)', () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gd-')); const lock = `gstack-detach-test-${process.pid}`; @@ -90,7 +143,7 @@ describe('gstack-detach', () => { }, 20000); test('rejects missing command (exit 2)', () => { - const r = spawnSync(DETACH, ['--label', 'x'], { encoding: 'utf-8' }); + const r = spawnSync(DETACH, ['--label', 'x'], { encoding: 'utf-8', timeout: 30_000 }); expect(r.status).toBe(2); }); }); diff --git a/test/gstack-developer-profile.test.ts b/test/gstack-developer-profile.test.ts index 507706509..1af3eef18 100644 --- a/test/gstack-developer-profile.test.ts +++ b/test/gstack-developer-profile.test.ts @@ -36,6 +36,7 @@ function runDev(...args: string[]): { stdout: string; stderr: string; status: nu env: { ...process.env, GSTACK_HOME: tmpHome }, encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); return { stdout: res.stdout ?? '', @@ -49,6 +50,7 @@ function logQuestion(payload: Record): number { env: { ...process.env, GSTACK_HOME: tmpHome }, encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); return res.status ?? -1; } diff --git a/test/gstack-egress-cli.test.ts b/test/gstack-egress-cli.test.ts index 6d9f19f8d..7b0398375 100644 --- a/test/gstack-egress-cli.test.ts +++ b/test/gstack-egress-cli.test.ts @@ -36,6 +36,7 @@ function run(args: string[]) { const result = spawnSync(BIN, args, { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home }, + timeout: 30_000, }); return { code: result.status ?? -1, stdout: result.stdout || '', stderr: result.stderr || '' }; } @@ -122,11 +123,13 @@ describe('gstack-egress grants', () => { const config = spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'telemetry', 'community'], { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home }, + timeout: 30_000, }); expect(config.status).toBe(0); spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'artifacts_sync_mode', 'full'], { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home }, + timeout: 30_000, }); const r = run(['grants', '--json']); expect(r.code).toBe(0); diff --git a/test/gstack-gbrain-detect-mcp-mode.test.ts b/test/gstack-gbrain-detect-mcp-mode.test.ts index c4793e6fc..e1cfba204 100644 --- a/test/gstack-gbrain-detect-mcp-mode.test.ts +++ b/test/gstack-gbrain-detect-mcp-mode.test.ts @@ -66,6 +66,7 @@ function runDetect(extraEnv: Record = {}): { code: number; json: ...extraEnv, }, encoding: 'utf-8', + timeout: 30_000, }); let json: any = null; try { diff --git a/test/gstack-gbrain-mcp-verify.test.ts b/test/gstack-gbrain-mcp-verify.test.ts index 4461d4dd5..57f036ebe 100644 --- a/test/gstack-gbrain-mcp-verify.test.ts +++ b/test/gstack-gbrain-mcp-verify.test.ts @@ -94,6 +94,7 @@ function runVerify(token: string, url: string): { code: number; stdout: string; GSTACK_HOME: tmpDir, }, encoding: 'utf-8', + timeout: 30_000, }); return { code: result.status ?? -1, @@ -249,6 +250,7 @@ describe('gstack-gbrain-mcp-verify', () => { const r = spawnSync(VERIFY_BIN, ['https://example.com/mcp'], { env: { ...process.env, PATH: `${fakeBinDir}:${process.env.PATH}`, GBRAIN_MCP_TOKEN: '' }, encoding: 'utf-8', + timeout: 30_000, }); expect(r.status).toBe(2); expect(r.stderr).toContain('GBRAIN_MCP_TOKEN'); @@ -259,6 +261,7 @@ describe('gstack-gbrain-mcp-verify', () => { const r = spawnSync(VERIFY_BIN, [], { env: { ...process.env, PATH: `${fakeBinDir}:${process.env.PATH}`, GBRAIN_MCP_TOKEN: 'x' }, encoding: 'utf-8', + timeout: 30_000, }); expect(r.status).toBe(2); }); diff --git a/test/gstack-gbrain-source-wireup.test.ts b/test/gstack-gbrain-source-wireup.test.ts index 71e2d8b17..4b7858c5f 100644 --- a/test/gstack-gbrain-source-wireup.test.ts +++ b/test/gstack-gbrain-source-wireup.test.ts @@ -145,6 +145,7 @@ function run( env, encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); } @@ -163,13 +164,13 @@ function gbrainCalls(): string[] { function setupGstackRepo(remoteUrl: string) { // Real git repo at gstackHome with at least one commit + an origin remote. fs.mkdirSync(gstackHome, { recursive: true }); - spawnSync('git', ['-C', gstackHome, 'init', '-q', '-b', 'main'], { stdio: 'pipe' }); - spawnSync('git', ['-C', gstackHome, 'config', 'user.email', 'test@example.com'], { stdio: 'pipe' }); - spawnSync('git', ['-C', gstackHome, 'config', 'user.name', 'test'], { stdio: 'pipe' }); + spawnSync('git', ['-C', gstackHome, 'init', '-q', '-b', 'main'], { stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['-C', gstackHome, 'config', 'user.email', 'test@example.com'], { stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['-C', gstackHome, 'config', 'user.name', 'test'], { stdio: 'pipe', timeout: 30_000 }); fs.writeFileSync(path.join(gstackHome, '.brain-allowlist'), '# allowlist\n'); - spawnSync('git', ['-C', gstackHome, 'add', '.'], { stdio: 'pipe' }); - spawnSync('git', ['-C', gstackHome, 'commit', '-q', '-m', 'init'], { stdio: 'pipe' }); - spawnSync('git', ['-C', gstackHome, 'remote', 'add', 'origin', remoteUrl], { stdio: 'pipe' }); + spawnSync('git', ['-C', gstackHome, 'add', '.'], { stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['-C', gstackHome, 'commit', '-q', '-m', 'init'], { stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['-C', gstackHome, 'remote', 'add', 'origin', remoteUrl], { stdio: 'pipe', timeout: 30_000 }); } beforeEach(() => { @@ -316,6 +317,7 @@ describe('gstack-gbrain-source-wireup — wireup mode', () => { const check = spawnSync('bash', ['-c', `command -v gbrain && gbrain --version`], { env: { PATH: `${hostLikeDir}:${process.env.PATH || '/usr/bin:/bin'}` }, encoding: 'utf-8', + timeout: 30_000, }); expect(check.status).toBe(0); expect(check.stdout).toContain('gbrain 0.18.2'); @@ -526,13 +528,15 @@ describe('gstack-gbrain-source-wireup — defensive paths', () => { run([], { env: { GSTACK_BRAIN_NO_SYNC: '1' } }); // Make a new commit on parent so worktree HEAD is "behind" fs.writeFileSync(path.join(gstackHome, 'newfile.md'), 'new'); - spawnSync('git', ['-C', gstackHome, 'add', '.'], { stdio: 'pipe' }); - spawnSync('git', ['-C', gstackHome, 'commit', '-q', '-m', 'second commit'], { stdio: 'pipe' }); + spawnSync('git', ['-C', gstackHome, 'add', '.'], { stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['-C', gstackHome, 'commit', '-q', '-m', 'second commit'], { stdio: 'pipe', timeout: 30_000 }); const parentHeadAfter = spawnSync('git', ['-C', gstackHome, 'rev-parse', 'HEAD'], { encoding: 'utf-8', + timeout: 30_000, }).stdout.trim(); const worktreeHeadBefore = spawnSync('git', ['-C', worktreeDir, 'rev-parse', 'HEAD'], { encoding: 'utf-8', + timeout: 30_000, }).stdout.trim(); expect(parentHeadAfter).not.toBe(worktreeHeadBefore); // sanity: parent advanced // --no-pull should leave worktree HEAD where it was @@ -540,6 +544,7 @@ describe('gstack-gbrain-source-wireup — defensive paths', () => { expect(r.status).toBe(0); const worktreeHeadAfter = spawnSync('git', ['-C', worktreeDir, 'rev-parse', 'HEAD'], { encoding: 'utf-8', + timeout: 30_000, }).stdout.trim(); expect(worktreeHeadAfter).toBe(worktreeHeadBefore); expect(worktreeHeadAfter).not.toBe(parentHeadAfter); diff --git a/test/gstack-gbrain-sync.test.ts b/test/gstack-gbrain-sync.test.ts index 9948da51c..7f16772bd 100644 --- a/test/gstack-gbrain-sync.test.ts +++ b/test/gstack-gbrain-sync.test.ts @@ -137,7 +137,7 @@ describe("gstack-gbrain-sync CLI", () => { const repo = mkdtempSync(join(tmpdir(), "gstack-pinned-source-repo-")); const commandLog = join(home, "gbrain-commands.log"); mkdirSync(gstackHome, { recursive: true }); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); writeFileSync(join(repo, ".gbrain-source"), "client-acme-app\n"); writeFileSync(join(bindir, "gbrain"), `#!/bin/sh printf '%s\\n' "$*" >> "$GSTACK_TEST_GBRAIN_LOG" @@ -179,7 +179,7 @@ exit 99 mkdirSync(gstackHome, { recursive: true }); mkdirSync(join(home, ".gbrain"), { recursive: true }); writeFileSync(join(home, ".gbrain", "config.json"), JSON.stringify({ engine: "pglite", database_url: "pglite:///test" })); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); writeFileSync(join(repo, ".gbrain-source"), "client-acme-app\n"); symlinkSync(repo, link, "dir"); writeFileSync(join(bindir, "gbrain"), `#!/bin/sh @@ -223,7 +223,7 @@ esac const bindir = mkdtempSync(join(tmpdir(), "gstack-pinned-dream-bin-")); const repo = mkdtempSync(join(tmpdir(), "gstack-pinned-dream-repo-")); mkdirSync(gstackHome, { recursive: true }); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); writeFileSync(join(repo, ".gbrain-source"), "client-acme-app\n"); writeFileSync(join(bindir, "gbrain"), "#!/bin/sh\nexit 99\n"); chmodSync(join(bindir, "gbrain"), 0o755); @@ -247,7 +247,7 @@ esac const gstackHome = join(home, ".gstack"); const repo = mkdtempSync(join(tmpdir(), "gstack-unreadable-pin-repo-")); mkdirSync(gstackHome, { recursive: true }); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); mkdirSync(join(repo, ".gbrain-source")); const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { @@ -282,8 +282,8 @@ esac const gstackHome = join(home, ".gstack"); mkdirSync(gstackHome, { recursive: true }); const repo = mkdtempSync(join(tmpdir(), "gstack-source-id-repo-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", remote], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", remote], { cwd: repo, timeout: 30_000 }); const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { encoding: "utf-8", @@ -312,7 +312,7 @@ esac const gstackHome = join(home, ".gstack"); mkdirSync(gstackHome, { recursive: true }); const repo = mkdtempSync(join(tmpdir(), "gstack-no-origin-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); // No `git remote add origin` — this is the no-remote case. const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { @@ -347,7 +347,7 @@ esac const parent = mkdtempSync(join(tmpdir(), "gstack-empty-base-")); const repo = join(parent, "___"); mkdirSync(repo); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); // No `origin` remote — forces the basename-fallback path. const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { @@ -381,8 +381,8 @@ esac const gstackHome = join(home, ".gstack"); mkdirSync(gstackHome, { recursive: true }); const repo = mkdtempSync(join(tmpdir(), "gstack-host-collide-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", "https://github.com/example/multihost.git"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", "https://github.com/example/multihost.git"], { cwd: repo, timeout: 30_000 }); // Dry-run still gates the code stage on `command -v gbrain`. Drop a no-op // shim on PATH so the stage runs (we only assert the preview line, never @@ -569,8 +569,8 @@ esac const repoA = mkdtempSync(join(tmpdir(), "gstack-worktree-a-")); const repoB = mkdtempSync(join(tmpdir(), "gstack-worktree-b-")); for (const repo of [repoA, repoB]) { - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", remote], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", remote], { cwd: repo, timeout: 30_000 }); } const idOf = (cwd: string): string => { @@ -606,8 +606,8 @@ esac const gstackHome = join(home, ".gstack"); mkdirSync(gstackHome, { recursive: true }); const repo = mkdtempSync(join(tmpdir(), "gstack-worktree-stable-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", remote], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", remote], { cwd: repo, timeout: 30_000 }); const idOf = (): string => { const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { @@ -635,8 +635,8 @@ esac const gstackHome = join(home, ".gstack"); mkdirSync(gstackHome, { recursive: true }); const repo = mkdtempSync(join(tmpdir(), "gstack-legacy-cleanup-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", "https://github.com/garrytan/gstack.git"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", "https://github.com/garrytan/gstack.git"], { cwd: repo, timeout: 30_000 }); const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { encoding: "utf-8", @@ -671,8 +671,8 @@ esac const gstackHome = join(home, ".gstack"); mkdirSync(gstackHome, { recursive: true }); const repo = mkdtempSync(join(tmpdir(), "gstack-attach-preview-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", "https://github.com/garrytan/gstack.git"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", "https://github.com/garrytan/gstack.git"], { cwd: repo, timeout: 30_000 }); const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { encoding: "utf-8", @@ -727,8 +727,8 @@ describe("derivePathOnlyHashLegacyId", () => { // legacy id regardless of $GSTACK_HOSTNAME, because the pre-#1468 hash // didn't include hostname. const repo = mkdtempSync(join(tmpdir(), "gstack-legacy-id-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", "https://github.com/example/legacy-test.git"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", "https://github.com/example/legacy-test.git"], { cwd: repo, timeout: 30_000 }); const cwd = process.cwd(); try { @@ -754,8 +754,8 @@ describe("derivePathOnlyHashLegacyId", () => { // host-fold id must differ for any non-empty hostname, so the migration // can detect + clean up the orphan. const repo = mkdtempSync(join(tmpdir(), "gstack-legacy-id-distinct-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", "https://github.com/example/distinct.git"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", "https://github.com/example/distinct.git"], { cwd: repo, timeout: 30_000 }); const cwd = process.cwd(); try { @@ -890,10 +890,10 @@ describe("constrainSourceId truncation (hyphen-boundary cut)", () => { const gstackHome = join(home, ".gstack"); mkdirSync(gstackHome, { recursive: true }); const repo = mkdtempSync(join(tmpdir(), "gstack-hyphen-cut-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); // Remote chosen to be long enough that constrainSourceId truncates and // the boundary lands inside the word `skill`. - spawnSync("git", ["remote", "add", "origin", "https://github.com/drummerms-av-sow-wiz/skill-270c0001.git"], { cwd: repo }); + spawnSync("git", ["remote", "add", "origin", "https://github.com/drummerms-av-sow-wiz/skill-270c0001.git"], { cwd: repo, timeout: 30_000 }); const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { encoding: "utf-8", @@ -924,8 +924,8 @@ describe("constrainSourceId truncation (hyphen-boundary cut)", () => { const gstackHome = join(home, ".gstack"); mkdirSync(gstackHome, { recursive: true }); const repo = mkdtempSync(join(tmpdir(), "gstack-https-period-")); - spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo }); - spawnSync("git", ["remote", "add", "origin", "https://github.com/foo/bar.git"], { cwd: repo }); + spawnSync("git", ["init", "--quiet", "-b", "main"], { cwd: repo, timeout: 30_000 }); + spawnSync("git", ["remote", "add", "origin", "https://github.com/foo/bar.git"], { cwd: repo, timeout: 30_000 }); const r = spawnSync("bun", [SCRIPT, "--dry-run", "--code-only", "--quiet"], { encoding: "utf-8", diff --git a/test/gstack-home-module-scope.test.ts b/test/gstack-home-module-scope.test.ts index 842e13247..c28c8d220 100644 --- a/test/gstack-home-module-scope.test.ts +++ b/test/gstack-home-module-scope.test.ts @@ -26,7 +26,7 @@ const ROOT = path.resolve(__dirname, '..'); function trackedTestFiles(): string[] { const out = spawnSync('git', ['ls-files', '*.test.ts'], { - cwd: ROOT, encoding: 'utf-8', + cwd: ROOT, encoding: 'utf-8', timeout: 30_000, }); if (out.status !== 0) throw new Error(`git ls-files failed: ${out.stderr}`); return out.stdout.split('\n').filter(Boolean); diff --git a/test/gstack-learnings-search.test.ts b/test/gstack-learnings-search.test.ts index 489e52607..ffa0227ae 100644 --- a/test/gstack-learnings-search.test.ts +++ b/test/gstack-learnings-search.test.ts @@ -16,6 +16,7 @@ const otherProjDir = path.join(tmpHome, 'projects', 'other-project'); function run(args: string[]): string { return execFileSync(BIN, args, { + timeout: 30_000, env: { ...process.env, GSTACK_HOME: tmpHome }, cwd: tmpCwd, encoding: 'utf-8', diff --git a/test/gstack-memory-ingest.test.ts b/test/gstack-memory-ingest.test.ts index 56523a0b6..a0220e279 100644 --- a/test/gstack-memory-ingest.test.ts +++ b/test/gstack-memory-ingest.test.ts @@ -500,6 +500,7 @@ describe("gstack-memory-ingest writer (gbrain v0.20+ batch `import` interface)", const POLICY = join(import.meta.dir, "..", "bin", "gstack-gbrain-repo-policy"); const seeded = spawnSync("bash", [POLICY, "set", "_unattributed", "deny"], { encoding: "utf-8", + timeout: 30_000, env: { ...process.env, HOME: home, GSTACK_HOME: gstackHome }, }); expect(seeded.status).toBe(0); @@ -622,6 +623,7 @@ esac expect(existsSync(stagingCopy)).toBe(true); const findMd = spawnSync("find", [stagingCopy, "-name", "*.md", "-type", "f"], { encoding: "utf-8", + timeout: 30_000, }); const mdPaths = (findMd.stdout || "").trim().split("\n").filter(Boolean); expect(mdPaths.length).toBeGreaterThan(0); @@ -685,6 +687,7 @@ esac // walk to find a .md and read its head.) const findMd = spawnSync("find", [stagingCopy, "-name", "*.md", "-type", "f"], { encoding: "utf-8", + timeout: 30_000, }); const mdPaths = (findMd.stdout || "").trim().split("\n").filter(Boolean); expect(mdPaths.length).toBeGreaterThan(0); @@ -913,8 +916,8 @@ describe("#2394: probe applies the same attribution gate as prepare", () => { function makeAttributableCwd(home: string): string { const repo = join(home, "work", "attributable-repo"); mkdirSync(repo, { recursive: true }); - spawnSync("git", ["-C", repo, "init", "-q"], { encoding: "utf-8" }); - spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/foo/bar.git"], { encoding: "utf-8" }); + spawnSync("git", ["-C", repo, "init", "-q"], { encoding: "utf-8", timeout: 30_000 }); + spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/foo/bar.git"], { encoding: "utf-8", timeout: 30_000 }); return repo; } @@ -993,8 +996,8 @@ describe("#2394: probe applies the same attribution gate as prepare", () => { mkdirSync(gstackHome, { recursive: true }); const attributableCwd = join(home, "work", "attributable-repo"); mkdirSync(attributableCwd, { recursive: true }); - spawnSync("git", ["-C", attributableCwd, "init", "-q"], { encoding: "utf-8" }); - spawnSync("git", ["-C", attributableCwd, "remote", "add", "origin", "https://github.com/foo/bar.git"], { encoding: "utf-8" }); + spawnSync("git", ["-C", attributableCwd, "init", "-q"], { encoding: "utf-8", timeout: 30_000 }); + spawnSync("git", ["-C", attributableCwd, "remote", "add", "origin", "https://github.com/foo/bar.git"], { encoding: "utf-8", timeout: 30_000 }); const ts = new Date().toISOString(); const cwdLine = `{"type":"user","message":{"role":"user","content":"hello"},"timestamp":"${ts}","cwd":"${attributableCwd.replace(/\\/g, "\\\\")}"}\n`; @@ -1077,8 +1080,8 @@ describe("#2392: transcript ingest honors per-remote trust policy", () => { function makeRepoWithRemote(home: string, name: string, remoteUrl: string): string { const repo = join(home, "work", name); mkdirSync(repo, { recursive: true }); - spawnSync("git", ["-C", repo, "init", "-q"], { encoding: "utf-8" }); - spawnSync("git", ["-C", repo, "remote", "add", "origin", remoteUrl], { encoding: "utf-8" }); + spawnSync("git", ["-C", repo, "init", "-q"], { encoding: "utf-8", timeout: 30_000 }); + spawnSync("git", ["-C", repo, "remote", "add", "origin", remoteUrl], { encoding: "utf-8", timeout: 30_000 }); return repo; } @@ -1095,6 +1098,7 @@ describe("#2392: transcript ingest honors per-remote trust policy", () => { function setPolicy(gstackHome: string, url: string, tier: string): void { const r = spawnSync(POLICY_BIN, ["set", url, tier], { encoding: "utf-8", + timeout: 30_000, env: { ...process.env, GSTACK_HOME: gstackHome }, }); expect(r.status).toBe(0); diff --git a/test/gstack-next-version.test.ts b/test/gstack-next-version.test.ts index 319047287..9099fd626 100644 --- a/test/gstack-next-version.test.ts +++ b/test/gstack-next-version.test.ts @@ -326,7 +326,7 @@ describe("default-base detection (no --base)", () => { function runWithoutBase(cwd: string): { exitCode: number; parsed: any } { const proc = Bun.spawnSync( ["bun", "run", SCRIPT, "--bump", "patch", "--workspace-root", "null"], - { cwd }, + { cwd, timeout: 30_000 }, ); const out = new TextDecoder().decode(proc.stdout); return { exitCode: proc.exitCode, parsed: JSON.parse(out) }; @@ -403,11 +403,11 @@ describe("offline output contract (what /ship branches on, #2545)", () => { // host:"unknown" on CI) while keeping ls-remote/fetch fully local. const bare = join(root, "github.com", "origin.git"); mkdirSync(bare, { recursive: true }); - Bun.spawnSync(["git", "init", "-q", "--bare", "-b", "main", bare]); + Bun.spawnSync(["git", "init", "-q", "--bare", "-b", "main", bare], { timeout: 30_000 }); const work = join(root, "work"); mkdirSync(work); const git = (...args: string[]) => - Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd: work }); + Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd: work, timeout: 30_000 }); git("init", "-q", "-b", "main"); writeFileSync(join(work, "VERSION"), "1.0.0.0\n"); git("add", "-A"); @@ -427,7 +427,7 @@ describe("offline output contract (what /ship branches on, #2545)", () => { const proc = Bun.spawnSync( ["bun", "run", NEXTVER, "--base", "main", "--bump", "patch", "--current-version", "1.0.0.0", "--workspace-root", "null"], - { cwd: work, env: { ...process.env, PATH: `${stubDir}:${process.env.PATH}` } }, + { cwd: work, env: { ...process.env, PATH: `${stubDir}:${process.env.PATH}` }, timeout: 30_000 }, ); rmSync(stubDir, { recursive: true, force: true }); rmSync(root, { recursive: true, force: true }); @@ -454,7 +454,7 @@ describe("offline output contract (what /ship branches on, #2545)", () => { const proc = Bun.spawnSync( ["bun", "run", NEXTVER, "--base", "main", "--bump", "patch", "--current-version", "1.0.0.0", "--workspace-root", "null"], - { cwd: work, env: { ...process.env, PATH: `${stubDir}:${process.env.PATH}` } }, + { cwd: work, env: { ...process.env, PATH: `${stubDir}:${process.env.PATH}` }, timeout: 30_000 }, ); rmSync(stubDir, { recursive: true, force: true }); rmSync(root, { recursive: true, force: true }); @@ -475,7 +475,7 @@ describe("fetchGitClaimed (offline allocation — the anti-duplicate fallback, # // (plus three earlier pairs found in the same audit). Git knows what the API // was asked for, so offline now degrades the QUEUE VIEW, not the ALLOCATION. function git(cwd: string, ...args: string[]) { - return Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd }); + return Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd, timeout: 30_000 }); } function fixture(): string { @@ -600,7 +600,7 @@ describe("fetchGitClaimed — non-mutating live remote query (ls-remote first)", // remote's LIVE branch list with zero local mutation — a path/file remote // answers it offline, which is exactly what these fixtures use. function git(cwd: string, ...args: string[]) { - return Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd }); + return Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd, timeout: 30_000 }); } // Local origin with: main (0.1.66.0), sibling (0.1.67.0, live claim), and @@ -718,7 +718,7 @@ describe("fetchGitClaimed — unfetched live claims (G2: ls-remote advertises SH // VERSION reads fail. The old `continue` silently dropped that LIVE claim — // the exact duplicate-allocation this fallback exists to prevent. function git(cwd: string, ...args: string[]) { - return Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd }); + return Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd, timeout: 30_000 }); } function cloneFixture(): { root: string; origin: string; clone: string } { @@ -908,14 +908,14 @@ describe("width pinned on failed base read (3-digit repos)", () => { // the base read fails too, which is the path under test. writeFileSync(join(stubDir, "gh"), "#!/bin/sh\nexit 1\n", { mode: 0o755 }); writeFileSync(join(stubDir, "glab"), "#!/bin/sh\nexit 1\n", { mode: 0o755 }); - Bun.spawnSync(["git", "init", "-q", "-b", "main"], { cwd: dir }); + Bun.spawnSync(["git", "init", "-q", "-b", "main"], { cwd: dir, timeout: 30_000 }); writeFileSync(join(dir, "VERSION"), "0.99.2\n"); - Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", "add", "-A"], { cwd: dir }); - Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "init"], { cwd: dir }); + Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", "add", "-A"], { cwd: dir, timeout: 30_000 }); + Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "init"], { cwd: dir, timeout: 30_000 }); const proc = Bun.spawnSync( ["bun", "run", SCRIPT, "--base", "main", "--bump", "patch", "--workspace-root", "null"], - { cwd: dir, env: { ...process.env, PATH: `${stubDir}:${process.env.PATH}` } }, + { cwd: dir, env: { ...process.env, PATH: `${stubDir}:${process.env.PATH}` }, timeout: 30_000 }, ); const out = JSON.parse(new TextDecoder().decode(proc.stdout)); // Zero base at the repo's OWN width — never "0.0.0.0" in a 3-digit repo. @@ -946,7 +946,7 @@ describe("integration (smoke)", () => { "1.6.3.0", "--workspace-root", "null", // skip sibling scan in CI - ]); + ], { timeout: 30_000 }); const out = new TextDecoder().decode(proc.stdout); const parsed = JSON.parse(out); expect(parsed).toHaveProperty("version"); @@ -976,7 +976,7 @@ describe("integration (smoke)", () => { "null", "--version-path", "Tinas Second Brain/health-tracker/VERSION", - ]); + ], { timeout: 30_000 }); const out = new TextDecoder().decode(proc.stdout); const parsed = JSON.parse(out); expect(parsed).toHaveProperty("version_path", "Tinas Second Brain/health-tracker/VERSION"); @@ -1003,7 +1003,7 @@ describe("fetchGitClaimed — laundered ls-remote (exit 0, empty output) is neve chmodSync(join(stubDir, "git"), 0o755); const git = (cwd: string, ...args: string[]) => - Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd }); + Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd, timeout: 30_000 }); const cwd = process.cwd(); const oldPath = process.env.PATH; @@ -1052,7 +1052,7 @@ describe("fetchGitClaimed — laundered ls-remote (exit 0, empty output) is neve chmodSync(join(stubDir, "git"), 0o755); const git = (cwd: string, ...args: string[]) => - Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd }); + Bun.spawnSync(["git", "-c", "user.email=t@t", "-c", "user.name=t", ...args], { cwd, timeout: 30_000 }); const cwd = process.cwd(); const oldPath = process.env.PATH; diff --git a/test/gstack-paths.test.ts b/test/gstack-paths.test.ts index 628f59919..50c1c7ed1 100644 --- a/test/gstack-paths.test.ts +++ b/test/gstack-paths.test.ts @@ -19,6 +19,7 @@ function run(env: Record): Record { const result = spawnSync('bash', [BIN], { env: { PATH: process.env.PATH, USERPROFILE: '', ...env } as Record, encoding: 'utf-8', + timeout: 30_000, }); if (result.status !== 0) { throw new Error(`gstack-paths failed (status ${result.status}): ${result.stderr}`); @@ -117,6 +118,7 @@ describe('gstack-paths', () => { { env: { PATH: process.env.PATH, USERPROFILE: '', ...env } as Record, encoding: 'utf-8', + timeout: 30_000, }, ); if (result.status !== 0) { @@ -159,6 +161,7 @@ describe('gstack-paths', () => { const result = spawnSync('bash', [BIN], { env: { PATH: process.env.PATH, USERPROFILE: '', HOME: '/tmp/h' } as Record, encoding: 'utf-8', + timeout: 30_000, }); const lines = result.stdout.split('\n').filter(Boolean); for (const line of lines) { diff --git a/test/gstack-question-log.test.ts b/test/gstack-question-log.test.ts index c99edf3d2..2be6772bc 100644 --- a/test/gstack-question-log.test.ts +++ b/test/gstack-question-log.test.ts @@ -26,6 +26,7 @@ function run(payload: string): { stdout: string; stderr: string; status: number env: { ...process.env, GSTACK_HOME: tmpHome }, encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); return { stdout: res.stdout ?? '', diff --git a/test/gstack-question-preference.test.ts b/test/gstack-question-preference.test.ts index 9b4f3c4a3..1b1e03063 100644 --- a/test/gstack-question-preference.test.ts +++ b/test/gstack-question-preference.test.ts @@ -31,6 +31,7 @@ function run(...args: string[]): { stdout: string; stderr: string; status: numbe env: { ...process.env, GSTACK_HOME: tmpHome }, encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); return { stdout: res.stdout ?? '', @@ -45,6 +46,7 @@ function runWithStdin(input: string, ...args: string[]): { stdout: string; stder encoding: 'utf-8', cwd: ROOT, input, + timeout: 30_000, }); return { stdout: res.stdout ?? '', diff --git a/test/gstack-redact-cli.test.ts b/test/gstack-redact-cli.test.ts index b41294b64..129125574 100644 --- a/test/gstack-redact-cli.test.ts +++ b/test/gstack-redact-cli.test.ts @@ -22,6 +22,7 @@ function run( ): { code: number; stdout: string; stderr: string } { const proc = Bun.spawnSync(["bun", BIN, ...args], { stdin: Buffer.from(stdin), + timeout: 30_000, }); return { code: proc.exitCode, @@ -88,7 +89,7 @@ describe("gstack-redact --from-file", () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "redact-file-")); const f = path.join(dir, "spec.md"); fs.writeFileSync(f, "leaked ghp_" + "a".repeat(36)); - const proc = Bun.spawnSync(["bun", BIN, "--from-file", f, "--json"]); + const proc = Bun.spawnSync(["bun", BIN, "--from-file", f, "--json"], { timeout: 30_000 }); const parsed = JSON.parse(proc.stdout.toString()); expect(parsed.findings[0].id).toBe("github.pat"); fs.rmSync(dir, { recursive: true, force: true }); diff --git a/test/gstack-repo-mode.test.ts b/test/gstack-repo-mode.test.ts index 62088a217..47b382684 100644 --- a/test/gstack-repo-mode.test.ts +++ b/test/gstack-repo-mode.test.ts @@ -25,6 +25,7 @@ function run(command: string, args: string[], cwd: string, home: string): Comman cwd, encoding: 'utf8', env: { ...process.env, HOME: home, GSTACK_HOME: path.join(home, '.gstack') }, + timeout: 30_000, }); return { stdout: result.stdout ?? '', diff --git a/test/gstack-session-kind.test.ts b/test/gstack-session-kind.test.ts index 7fab33871..310daeee0 100644 --- a/test/gstack-session-kind.test.ts +++ b/test/gstack-session-kind.test.ts @@ -22,6 +22,7 @@ function kind(env: Record): string { return execFileSync(BIN, [], { env: { PATH: process.env.PATH ?? '/usr/bin:/bin', ...env }, encoding: 'utf-8', + timeout: 30_000, }).trim(); } diff --git a/test/gstack-settings-hook-schema-aware.test.ts b/test/gstack-settings-hook-schema-aware.test.ts index a6dbb252a..7a0ce7b83 100644 --- a/test/gstack-settings-hook-schema-aware.test.ts +++ b/test/gstack-settings-hook-schema-aware.test.ts @@ -919,6 +919,7 @@ describe('prune-stale', () => { const live = mkCanon(tmpDir, 'live-worktree'); execSync(`'${path.join(ROOT, 'bin', 'gstack-config')}' set plan_tune_hooks no`, { env: { ...process.env, GSTACK_STATE_ROOT: tmpDir }, + timeout: 30_000, }); fs.writeFileSync(settingsFile, JSON.stringify({ hooks: { diff --git a/test/gstack-skill-start.test.ts b/test/gstack-skill-start.test.ts index 860352dc7..fcc069d64 100644 --- a/test/gstack-skill-start.test.ts +++ b/test/gstack-skill-start.test.ts @@ -31,6 +31,7 @@ let tmpGstackHome: string; function runStart(args: string[] = [], env: Record = {}): string { return execFileSync(START, ['--skill', 'testskill', ...args], { + timeout: 30_000, encoding: 'utf-8', cwd: tmpHome, // no CLAUDE.md/AGENTS.md, not the repo — routing detection stays cold env: { @@ -150,6 +151,7 @@ describe('gstack-skill-start behavior', () => { fs.copyFileSync(START, path.join(fakeBin, 'gstack-skill-start')); fs.chmodSync(path.join(fakeBin, 'gstack-skill-start'), 0o755); const out = execFileSync(path.join(fakeBin, 'gstack-skill-start'), ['--skill', 't'], { + timeout: 30_000, encoding: 'utf-8', cwd: tmpHome, env: { PATH: process.env.PATH!, HOME: tmpHome, GSTACK_HOME: tmpGstackHome }, @@ -317,7 +319,7 @@ describe('gstack-skill-end', () => { const out = execFileSync( END, ['--skill', 't', '--outcome', 'success', '--session-id', 'sid-1', '--tel-start', String(start)], - { encoding: 'utf-8', cwd: tmpHome, env: { PATH: process.env.PATH!, HOME: tmpHome, GSTACK_HOME: tmpGstackHome } }, + { timeout: 30_000, encoding: 'utf-8', cwd: tmpHome, env: { PATH: process.env.PATH!, HOME: tmpHome, GSTACK_HOME: tmpGstackHome } }, ); const m = out.match(/SKILL_END: recorded outcome=success duration_s=(\d+)/); expect(m).not.toBeNull(); @@ -338,6 +340,7 @@ describe('gstack-skill-end', () => { const pending = path.join(tmpGstackHome, 'analytics', '.pending-sid-2'); fs.writeFileSync(pending, 'x'); execFileSync(END, ['--skill', 't', '--outcome', 'abort', '--session-id', 'sid-2', '--tel-start', 'bogus'], { + timeout: 30_000, encoding: 'utf-8', cwd: tmpHome, env: { PATH: process.env.PATH!, HOME: tmpHome, GSTACK_HOME: tmpGstackHome }, diff --git a/test/gstack-slug-cwd-walk-up.test.ts b/test/gstack-slug-cwd-walk-up.test.ts index b129b8e79..4280c2107 100644 --- a/test/gstack-slug-cwd-walk-up.test.ts +++ b/test/gstack-slug-cwd-walk-up.test.ts @@ -172,12 +172,13 @@ describe('gstack-slug — outermost project-root resolution', () => { // succeeds. (The script's step 2 reads the remote when there's no cache.) const gitInit = spawnSync('git', ['init', '-q', '-b', 'main', projectRoot], { encoding: 'utf8', + timeout: 30_000, }); expect(gitInit.status).toBe(0); const gitRemote = spawnSync( 'git', ['-C', projectRoot, 'remote', 'add', 'origin', 'https://github.com/foo/bar.git'], - { encoding: 'utf8' }, + { encoding: 'utf8', timeout: 30_000 }, ); expect(gitRemote.status).toBe(0); diff --git a/test/gstack-slug-sanitize.test.ts b/test/gstack-slug-sanitize.test.ts index ab0eb0b7f..bc3aee3d1 100644 --- a/test/gstack-slug-sanitize.test.ts +++ b/test/gstack-slug-sanitize.test.ts @@ -26,6 +26,7 @@ function runSlug(cwd: string, home: string) { return spawnSync([SLUG_BIN], { cwd, env: { ...process.env, HOME: home }, + timeout: 30_000, }); } @@ -77,6 +78,7 @@ describe('slug cache hygiene', () => { const r = spawnSync(['bash', SLUG_BIN], { cwd: os.tmpdir(), env: { ...process.env, GSTACK_HOME: home, GSTACK_PROJECT_SLUG: 'override-slug' }, + timeout: 30_000, }); expect(r.stdout.toString()).toContain('SLUG=override-slug'); expect(fs.existsSync(path.join(home, 'slug-cache'))).toBe(false); @@ -95,6 +97,7 @@ describe('slug cache hygiene', () => { const r = spawnSync(['bash', SLUG_BIN], { cwd: os.tmpdir(), env: { ...ambient, GSTACK_HOME: home }, + timeout: 30_000, }); expect(r.exitCode).toBe(0); const entries = fs.readdirSync(path.join(home, 'slug-cache')); diff --git a/test/gstack-state-root-override.test.ts b/test/gstack-state-root-override.test.ts index cc2e672d6..75a130220 100644 --- a/test/gstack-state-root-override.test.ts +++ b/test/gstack-state-root-override.test.ts @@ -54,6 +54,7 @@ function runBin( env: cleaned, encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); return { stdout: res.stdout ?? '', diff --git a/test/gstack-team-init-hook-schema.test.ts b/test/gstack-team-init-hook-schema.test.ts index 9ce2f0cfc..f5ff08a4f 100644 --- a/test/gstack-team-init-hook-schema.test.ts +++ b/test/gstack-team-init-hook-schema.test.ts @@ -42,6 +42,7 @@ describe('gstack-team-init required: PreToolUse hook schema (#2413)', () => { const stdout = execSync(`bash "${hookPath}"`, { env: { ...process.env, HOME: home }, encoding: 'utf-8', + timeout: 30_000, }); return { status: 0, stdout, stderr: '' }; } catch (err) { diff --git a/test/gstack-upgrade-migration-v1_17_0_0.test.ts b/test/gstack-upgrade-migration-v1_17_0_0.test.ts index e1d20a95d..c5ffefe65 100644 --- a/test/gstack-upgrade-migration-v1_17_0_0.test.ts +++ b/test/gstack-upgrade-migration-v1_17_0_0.test.ts @@ -64,6 +64,7 @@ function run(opts: { env?: Record } = {}) { env, encoding: 'utf-8', cwd: tmpHome, + timeout: 30_000, }); } diff --git a/test/gstack-upgrade-migration-v1_40_0_0.test.ts b/test/gstack-upgrade-migration-v1_40_0_0.test.ts index f76815d83..b09fd44f2 100644 --- a/test/gstack-upgrade-migration-v1_40_0_0.test.ts +++ b/test/gstack-upgrade-migration-v1_40_0_0.test.ts @@ -136,6 +136,7 @@ function run(opts: { path?: string } = {}) { env, encoding: "utf-8", cwd: tmpHome, + timeout: 30_000, }); } diff --git a/test/gstack-version-bump.test.ts b/test/gstack-version-bump.test.ts index 89d525803..2c8a55835 100644 --- a/test/gstack-version-bump.test.ts +++ b/test/gstack-version-bump.test.ts @@ -61,7 +61,7 @@ describe('write (FRESH bump)', () => { test('writes VERSION + package.json.version, preserving other pkg fields', () => { fs.writeFileSync(path.join(dir, 'VERSION'), '1.0.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '1.0.0.0', scripts: { t: 'y' } }, null, 2) + '\n'); - const out = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: dir, timeout: 30_000 }).toString(); expect(JSON.parse(out)).toEqual({ wrote: '1.1.0.0', packageJson: true, packageJsonPath: 'package.json', packageJsonVersion: '1.1.0', packageLock: false, agentsDigest: null, @@ -76,7 +76,7 @@ describe('write (FRESH bump)', () => { test('rejects a malformed version with exit 2', () => { let code = 0; - try { execFileSync('bun', [BIN, 'write', '--version', '1.2.3.4.5'], { cwd: dir, stdio: 'pipe' }); } + try { execFileSync('bun', [BIN, 'write', '--version', '1.2.3.4.5'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); } catch (e: any) { code = e.status; } expect(code).toBe(2); }); @@ -84,7 +84,7 @@ describe('write (FRESH bump)', () => { test('VERSION-only repo (no package.json) writes just VERSION', () => { const d2 = fs.mkdtempSync(path.join(os.tmpdir(), 'vbump-noPkg-')); fs.writeFileSync(path.join(d2, 'VERSION'), '0.1.0.0\n'); - const out = execFileSync('bun', [BIN, 'write', '--version', '0.2.0.0'], { cwd: d2 }).toString(); + const out = execFileSync('bun', [BIN, 'write', '--version', '0.2.0.0'], { cwd: d2, timeout: 30_000 }).toString(); expect(JSON.parse(out)).toEqual({ wrote: '0.2.0.0', packageJson: false, packageJsonPath: null, packageJsonVersion: null, packageLock: false, agentsDigest: null, @@ -101,7 +101,7 @@ describe('repair (DRIFT_STALE_PKG)', () => { test('syncs package.json.version up to VERSION, no re-bump', () => { fs.writeFileSync(path.join(dir, 'VERSION'), '2.0.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '1.9.0.0' }, null, 2) + '\n'); - const out = execFileSync('bun', [BIN, 'repair'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'repair'], { cwd: dir, timeout: 30_000 }).toString(); expect(JSON.parse(out)).toEqual({ repaired: '2.0.0.0', packageJsonPath: 'package.json', packageJsonVersion: '2.0.0', }); @@ -112,7 +112,7 @@ describe('repair (DRIFT_STALE_PKG)', () => { test('refuses to propagate an invalid VERSION (exit 2)', () => { fs.writeFileSync(path.join(dir, 'VERSION'), 'not-a-version\n'); let code = 0; - try { execFileSync('bun', [BIN, 'repair'], { cwd: dir, stdio: 'pipe' }); } + try { execFileSync('bun', [BIN, 'repair'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); } catch (e: any) { code = e.status; } expect(code).toBe(2); }); @@ -131,7 +131,7 @@ describe('write/repair sync npm lockfiles (both version fields, #2567)', () => { fs.writeFileSync(path.join(dir, 'VERSION'), '1.0.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '1.0.0' }, null, 2) + '\n'); fs.writeFileSync(path.join(dir, 'package-lock.json'), lock('1.0.0')); - const out = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: dir, timeout: 30_000 }).toString(); expect(JSON.parse(out)).toEqual({ wrote: '1.1.0.0', packageJson: true, packageJsonPath: 'package.json', packageJsonVersion: '1.1.0', packageLock: true, agentsDigest: null, @@ -146,7 +146,7 @@ describe('write/repair sync npm lockfiles (both version fields, #2567)', () => { fs.writeFileSync(path.join(dir, 'VERSION'), '2.0.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '1.9.0' }, null, 2) + '\n'); fs.writeFileSync(path.join(dir, 'package-lock.json'), lock('1.9.0')); - execFileSync('bun', [BIN, 'repair'], { cwd: dir }); + execFileSync('bun', [BIN, 'repair'], { cwd: dir, timeout: 30_000 }); const l = JSON.parse(fs.readFileSync(path.join(dir, 'package-lock.json'), 'utf-8')); expect(l.version).toBe('2.0.0'); expect(l.packages[''].version).toBe('2.0.0'); @@ -156,7 +156,7 @@ describe('write/repair sync npm lockfiles (both version fields, #2567)', () => { fs.writeFileSync(path.join(dir, 'VERSION'), '3.0.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '2.9.0' }, null, 2) + '\n'); fs.writeFileSync(path.join(dir, 'package-lock.json'), JSON.stringify({ name: 'x', version: '2.9.0', lockfileVersion: 1 }, null, 2) + '\n'); - execFileSync('bun', [BIN, 'repair'], { cwd: dir }); + execFileSync('bun', [BIN, 'repair'], { cwd: dir, timeout: 30_000 }); const l = JSON.parse(fs.readFileSync(path.join(dir, 'package-lock.json'), 'utf-8')); expect(l.version).toBe('3.0.0'); expect(l.packages).toBeUndefined(); @@ -167,7 +167,7 @@ describe('write/repair sync npm lockfiles (both version fields, #2567)', () => { fs.writeFileSync(path.join(d2, 'VERSION'), '1.0.0.0\n'); fs.writeFileSync(path.join(d2, 'package.json'), JSON.stringify({ name: 'x', version: '1.0.0' }, null, 2) + '\n'); fs.writeFileSync(path.join(d2, 'npm-shrinkwrap.json'), lock('1.0.0').replace('package-lock', 'npm-shrinkwrap')); - const out = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: d2 }).toString(); + const out = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: d2, timeout: 30_000 }).toString(); expect(JSON.parse(out).packageLock).toBe(true); const l = JSON.parse(fs.readFileSync(path.join(d2, 'npm-shrinkwrap.json'), 'utf-8')); expect(l.version).toBe('1.1.0'); @@ -183,7 +183,7 @@ describe('write/repair sync npm lockfiles (both version fields, #2567)', () => { fs.writeFileSync(path.join(d3, 'package.json'), JSON.stringify({ name: 'x', version: '1.0.0.0' }, null, 2) + '\n'); fs.writeFileSync(path.join(d3, 'package-lock.json'), '{ not json'); let code = 0; - try { execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: d3, stdio: 'pipe' }); } + try { execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: d3, stdio: 'pipe', timeout: 30_000 }); } catch (e: any) { code = e.status; } expect(code).toBe(3); // VERSION was written before the failure — exactly the half-write the @@ -198,26 +198,26 @@ describe('classify (idempotency over a real git base)', () => { afterAll(() => { try { fs.rmSync(dir, { recursive: true, force: true }); } catch { /* noop */ } }); // Build a tiny repo with an "origin/main" carrying VERSION=1.0.0.0. - const git = (...a: string[]) => execFileSync('git', a, { cwd: dir, stdio: 'pipe' }); + const git = (...a: string[]) => execFileSync('git', a, { cwd: dir, stdio: 'pipe', timeout: 30_000 }); fs.writeFileSync(path.join(dir, 'VERSION'), '1.0.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '1.0.0.0' }, null, 2) + '\n'); git('init', '-q', '-b', 'main'); git('config', 'user.email', 't@t'); git('config', 'user.name', 't'); git('add', '-A'); git('commit', '-q', '-m', 'base'); // Fake an "origin/main" remote-tracking ref pointing at this commit. - const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: dir }).toString().trim(); + const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: dir, timeout: 30_000 }).toString().trim(); fs.mkdirSync(path.join(dir, '.git', 'refs', 'remotes', 'origin'), { recursive: true }); fs.writeFileSync(path.join(dir, '.git', 'refs', 'remotes', 'origin', 'main'), head + '\n'); test('reports FRESH before any bump', () => { - const out = execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: dir, timeout: 30_000 }).toString(); expect(JSON.parse(out).state).toBe('FRESH'); }); test('reports ALREADY_BUMPED after VERSION+pkg move together', () => { fs.writeFileSync(path.join(dir, 'VERSION'), '1.1.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '1.1.0.0' }, null, 2) + '\n'); - const out = execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: dir, timeout: 30_000 }).toString(); const parsed = JSON.parse(out); expect(parsed.state).toBe('ALREADY_BUMPED'); expect(parsed.baseVersion).toBe('1.0.0.0'); @@ -246,17 +246,17 @@ describe('package.json as the version source (monorepo, 3-digit, #2501)', () => fs.mkdirSync(path.join(dir, 'frontend'), { recursive: true }); fs.writeFileSync(pkgAbs, JSON.stringify({ name: 'frontend', version: '0.99.2', private: true, scripts: { dev: 'next dev' } }, null, 2) + '\n'); - execFileSync('git', ['init', '-q', '-b', 'main'], { cwd: dir }); - execFileSync('git', ['config', 'user.email', 't@e.com'], { cwd: dir }); - execFileSync('git', ['config', 'user.name', 't'], { cwd: dir }); - execFileSync('git', ['add', '-A'], { cwd: dir }); - execFileSync('git', ['commit', '-qm', 'v0.99.2 base'], { cwd: dir }); - const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: dir }).toString().trim(); + execFileSync('git', ['init', '-q', '-b', 'main'], { cwd: dir, timeout: 30_000 }); + execFileSync('git', ['config', 'user.email', 't@e.com'], { cwd: dir, timeout: 30_000 }); + execFileSync('git', ['config', 'user.name', 't'], { cwd: dir, timeout: 30_000 }); + execFileSync('git', ['add', '-A'], { cwd: dir, timeout: 30_000 }); + execFileSync('git', ['commit', '-qm', 'v0.99.2 base'], { cwd: dir, timeout: 30_000 }); + const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: dir, timeout: 30_000 }).toString().trim(); fs.mkdirSync(path.join(dir, '.git', 'refs', 'remotes', 'origin'), { recursive: true }); fs.writeFileSync(path.join(dir, '.git', 'refs', 'remotes', 'origin', 'main'), head + '\n'); test('classify reads the real version from the package.json version-path', () => { - const out = execFileSync('bun', [BIN, 'classify', '--base', 'main', '--version-path', pkgRel], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'classify', '--base', 'main', '--version-path', pkgRel], { cwd: dir, timeout: 30_000 }).toString(); const parsed = JSON.parse(out); expect(parsed.state).toBe('FRESH'); expect(parsed.baseVersion).toBe('0.99.2'); // was "0.0.0.0" @@ -265,7 +265,7 @@ describe('package.json as the version source (monorepo, 3-digit, #2501)', () => }); test('write updates the package.json in place and creates no VERSION file', () => { - const out = execFileSync('bun', [BIN, 'write', '--version', '0.99.3', '--version-path', pkgRel], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'write', '--version', '0.99.3', '--version-path', pkgRel], { cwd: dir, timeout: 30_000 }).toString(); expect(JSON.parse(out)).toEqual({ wrote: '0.99.3', versionPath: pkgRel, packageJson: true, packageLock: false, agentsDigest: null }); const pkg = JSON.parse(fs.readFileSync(pkgAbs, 'utf-8')); expect(pkg.version).toBe('0.99.3'); @@ -275,7 +275,7 @@ describe('package.json as the version source (monorepo, 3-digit, #2501)', () => }); test('classify reports ALREADY_BUMPED after that write, not a drift state', () => { - const out = execFileSync('bun', [BIN, 'classify', '--base', 'main', '--version-path', pkgRel], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'classify', '--base', 'main', '--version-path', pkgRel], { cwd: dir, timeout: 30_000 }).toString(); const parsed = JSON.parse(out); expect(parsed.state).toBe('ALREADY_BUMPED'); expect(parsed.baseVersion).toBe('0.99.2'); @@ -283,14 +283,14 @@ describe('package.json as the version source (monorepo, 3-digit, #2501)', () => }); test('repair is a no-op: there is no second file to drift from', () => { - const out = execFileSync('bun', [BIN, 'repair', '--version-path', pkgRel], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'repair', '--version-path', pkgRel], { cwd: dir, timeout: 30_000 }).toString(); expect(JSON.parse(out).repaired).toBeNull(); }); test('write refuses a version-path that does not exist', () => { let code = 0; try { - execFileSync('bun', [BIN, 'write', '--version', '1.0.0', '--version-path', 'nope/package.json'], { cwd: dir, stdio: 'pipe' }); + execFileSync('bun', [BIN, 'write', '--version', '1.0.0', '--version-path', 'nope/package.json'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); } catch (e: any) { code = e.status; } expect(code).toBe(2); }); @@ -315,12 +315,12 @@ describe('.gstack/version-path pin, no --version-path flag (#2462)', () => { }; const commitBase = (d: string): void => { - execFileSync('git', ['init', '-q', '-b', 'main'], { cwd: d }); - execFileSync('git', ['config', 'user.email', 't@e.com'], { cwd: d }); - execFileSync('git', ['config', 'user.name', 't'], { cwd: d }); - execFileSync('git', ['add', '-A'], { cwd: d }); - execFileSync('git', ['commit', '-qm', 'base'], { cwd: d }); - const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: d }).toString().trim(); + execFileSync('git', ['init', '-q', '-b', 'main'], { cwd: d, timeout: 30_000 }); + execFileSync('git', ['config', 'user.email', 't@e.com'], { cwd: d, timeout: 30_000 }); + execFileSync('git', ['config', 'user.name', 't'], { cwd: d, timeout: 30_000 }); + execFileSync('git', ['add', '-A'], { cwd: d, timeout: 30_000 }); + execFileSync('git', ['commit', '-qm', 'base'], { cwd: d, timeout: 30_000 }); + const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: d, timeout: 30_000 }).toString().trim(); fs.mkdirSync(path.join(d, '.git', 'refs', 'remotes', 'origin'), { recursive: true }); fs.writeFileSync(path.join(d, '.git', 'refs', 'remotes', 'origin', 'main'), head + '\n'); }; @@ -332,7 +332,7 @@ describe('.gstack/version-path pin, no --version-path flag (#2462)', () => { commitBase(d); // Move the pinned file past base — NO root VERSION file exists at all. fs.writeFileSync(path.join(d, pinRel), '1.5.0.0\n'); - const out = JSON.parse(execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: d }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: d, timeout: 30_000 }).toString()); // Before the fix: baseVersion read root VERSION → "0.0.0.0" and the // branch misclassified as... current 1.5.0.0 vs base 0.0.0.0. The REAL // base is the pinned file's committed value. @@ -347,7 +347,7 @@ describe('.gstack/version-path pin, no --version-path flag (#2462)', () => { const d = mkPinned(pinRel); fs.writeFileSync(path.join(d, pinRel), JSON.stringify({ name: 'f', version: '0.99.2' }, null, 2) + '\n'); commitBase(d); - const out = JSON.parse(execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: d }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: d, timeout: 30_000 }).toString()); // Before the fix: versionRel="VERSION" → the pinned JSON was read as raw // text → currentVersion "0.0.0.0", pkgExists false, base from a // nonexistent root VERSION. @@ -362,7 +362,7 @@ describe('.gstack/version-path pin, no --version-path flag (#2462)', () => { const pinRel = 'frontend/package.json'; const d = mkPinned(pinRel); fs.writeFileSync(path.join(d, pinRel), JSON.stringify({ name: 'f', version: '0.99.2' }, null, 2) + '\n'); - const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '0.99.3'], { cwd: d }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '0.99.3'], { cwd: d, timeout: 30_000 }).toString()); expect(out).toEqual({ wrote: '0.99.3', versionPath: pinRel, packageJson: true, packageLock: false, agentsDigest: null }); expect(JSON.parse(fs.readFileSync(path.join(d, pinRel), 'utf-8')).version).toBe('0.99.3'); // Before the fix, write treated versionRel as "VERSION" and overwrote the @@ -375,7 +375,7 @@ describe('.gstack/version-path pin, no --version-path flag (#2462)', () => { const pinRel = 'frontend/package.json'; const d = mkPinned(pinRel); fs.writeFileSync(path.join(d, pinRel), JSON.stringify({ name: 'f', version: '0.99.2' }, null, 2) + '\n'); - const out = JSON.parse(execFileSync('bun', [BIN, 'repair'], { cwd: d }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'repair'], { cwd: d, timeout: 30_000 }).toString()); expect(out.repaired).toBeNull(); fs.rmSync(d, { recursive: true, force: true }); }); @@ -386,7 +386,7 @@ describe('.gstack/version-path pin, no --version-path flag (#2462)', () => { fs.writeFileSync(path.join(d, 'OTHER_VERSION'), '2.0.0.0\n'); commitBase(d); const out = JSON.parse( - execFileSync('bun', [BIN, 'classify', '--base', 'main', '--version-path', 'OTHER_VERSION'], { cwd: d }).toString(), + execFileSync('bun', [BIN, 'classify', '--base', 'main', '--version-path', 'OTHER_VERSION'], { cwd: d, timeout: 30_000 }).toString(), ); expect(out.currentVersion).toBe('2.0.0.0'); expect(out.baseVersion).toBe('2.0.0.0'); @@ -414,7 +414,7 @@ describe('subdirectory manifest (no root package.json, #2531)', () => { const d = mk(); fs.writeFileSync(path.join(d, 'web', 'package.json'), JSON.stringify({ name: 'w', version: '0.1.0' }, null, 2) + '\n'); - const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '0.2.0.0'], { cwd: d }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '0.2.0.0'], { cwd: d, timeout: 30_000 }).toString()); expect(out.packageJson).toBe(true); expect(out.packageJsonPath).toBe('web/package.json'); expect(out.packageJsonVersion).toBe('0.2.0'); @@ -428,7 +428,7 @@ describe('subdirectory manifest (no root package.json, #2531)', () => { fs.writeFileSync(path.join(d, 'web', 'package.json'), JSON.stringify({ version: '0.1.0' }, null, 2) + '\n'); fs.writeFileSync(path.join(d, 'app', 'package.json'), JSON.stringify({ version: '0.1.0' }, null, 2) + '\n'); const out = JSON.parse(execFileSync('bun', - [BIN, 'write', '--version', '0.3.0.0', '--package-json-path', 'app/package.json'], { cwd: d }).toString()); + [BIN, 'write', '--version', '0.3.0.0', '--package-json-path', 'app/package.json'], { cwd: d, timeout: 30_000 }).toString()); expect(out.packageJsonPath).toBe('app/package.json'); expect(JSON.parse(fs.readFileSync(path.join(d, 'app', 'package.json'), 'utf-8')).version).toBe('0.3.0'); // the pinned one is untouched @@ -440,16 +440,16 @@ describe('subdirectory manifest (no root package.json, #2531)', () => { const d = mk(); fs.writeFileSync(path.join(d, 'web', 'package.json'), JSON.stringify({ name: 'w', version: '0.1.0' }, null, 2) + '\n'); - execFileSync('git', ['init', '-q', '-b', 'main'], { cwd: d }); - execFileSync('git', ['config', 'user.email', 't@e.com'], { cwd: d }); - execFileSync('git', ['config', 'user.name', 't'], { cwd: d }); - execFileSync('git', ['add', '-A'], { cwd: d }); - execFileSync('git', ['commit', '-qm', 'base'], { cwd: d }); - const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: d }).toString().trim(); + execFileSync('git', ['init', '-q', '-b', 'main'], { cwd: d, timeout: 30_000 }); + execFileSync('git', ['config', 'user.email', 't@e.com'], { cwd: d, timeout: 30_000 }); + execFileSync('git', ['config', 'user.name', 't'], { cwd: d, timeout: 30_000 }); + execFileSync('git', ['add', '-A'], { cwd: d, timeout: 30_000 }); + execFileSync('git', ['commit', '-qm', 'base'], { cwd: d, timeout: 30_000 }); + const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: d, timeout: 30_000 }).toString().trim(); fs.mkdirSync(path.join(d, '.git', 'refs', 'remotes', 'origin'), { recursive: true }); fs.writeFileSync(path.join(d, '.git', 'refs', 'remotes', 'origin', 'main'), head + '\n'); - const out = JSON.parse(execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: d }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: d, timeout: 30_000 }).toString()); // 0.1.0 IS the npm-valid translation of 0.1.0.0 — in sync, no drift. expect(out.state).toBe('FRESH'); expect(out.pkgExists).toBe(true); @@ -462,7 +462,7 @@ describe('subdirectory manifest (no root package.json, #2531)', () => { const d = mk(); fs.writeFileSync(path.join(d, 'web', 'package.json'), JSON.stringify({ name: 'w', version: '0.0.9' }, null, 2) + '\n'); - const out = JSON.parse(execFileSync('bun', [BIN, 'repair'], { cwd: d }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'repair'], { cwd: d, timeout: 30_000 }).toString()); expect(out).toEqual({ repaired: '0.1.0.0', packageJsonPath: 'web/package.json', packageJsonVersion: '0.1.0' }); expect(JSON.parse(fs.readFileSync(path.join(d, 'web', 'package.json'), 'utf-8')).version).toBe('0.1.0'); fs.rmSync(d, { recursive: true, force: true }); @@ -504,7 +504,7 @@ describe('path containment: pins and flags cannot escape the repo', () => { function runFail(args: string[]): { code: number; stderr: string } { try { - execFileSync('bun', [BIN, ...args], { cwd: dir, stdio: 'pipe' }); + execFileSync('bun', [BIN, ...args], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); return { code: 0, stderr: '' }; } catch (e: any) { return { code: e.status, stderr: (e.stderr || '').toString() }; @@ -562,7 +562,7 @@ describe('path containment: pins and flags cannot escape the repo', () => { fs.writeFileSync(outerLock, JSON.stringify({ version: '1.0.0', packages: { '': { version: '1.0.0' } } }, null, 2) + '\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '1.0.0' }, null, 2) + '\n'); fs.symlinkSync(outerLock, path.join(dir, 'package-lock.json')); - const res = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: dir, stdio: 'pipe' }); + const res = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); expect(JSON.parse(res.toString()).packageLock).toBe(false); expect(JSON.parse(fs.readFileSync(outerLock, 'utf-8')).version).toBe('1.0.0'); }); @@ -572,7 +572,7 @@ describe('path containment: pins and flags cannot escape the repo', () => { fs.mkdirSync(path.join(dir, 'frontend'), { recursive: true }); fs.writeFileSync(path.join(dir, 'frontend', 'package.json'), JSON.stringify({ name: 'x', version: '1.0.0' }, null, 2) + '\n'); fs.writeFileSync(path.join(dir, '.gstack', 'version-path'), 'frontend/package.json\n'); - const out = execFileSync('bun', [BIN, 'write', '--version', '1.1.0'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'write', '--version', '1.1.0'], { cwd: dir, timeout: 30_000 }).toString(); expect(JSON.parse(out).wrote).toBe('1.1.0'); expect(JSON.parse(fs.readFileSync(path.join(dir, 'frontend', 'package.json'), 'utf-8')).version).toBe('1.1.0'); }); @@ -602,7 +602,7 @@ describe('#2600: repair must not write fabricated 0.0.0.0 when VERSION is missin let code = 0; let stderr = ''; try { - execFileSync('bun', [BIN, 'repair'], { cwd: dir, stdio: 'pipe' }); + execFileSync('bun', [BIN, 'repair'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); } catch (e: any) { code = e.status; stderr = (e.stderr || '').toString(); @@ -621,7 +621,7 @@ describe('#2600: repair must not write fabricated 0.0.0.0 when VERSION is missin fs.writeFileSync(path.join(dir, 'VERSION'), '2.0.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '1.9.0' }, null, 2) + '\n'); - const out = execFileSync('bun', [BIN, 'repair'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'repair'], { cwd: dir, timeout: 30_000 }).toString(); const result = JSON.parse(out); expect(result.repaired).toBe('2.0.0.0'); @@ -639,7 +639,7 @@ describe('#2600: repair must not write fabricated 0.0.0.0 when VERSION is missin let code = 0; let stderr = ''; try { - execFileSync('bun', [BIN, 'repair'], { cwd: dir, stdio: 'pipe' }); + execFileSync('bun', [BIN, 'repair'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); } catch (e: any) { code = e.status; stderr = (e.stderr || '').toString(); @@ -660,7 +660,7 @@ describe('#2600: repair must not write fabricated 0.0.0.0 when VERSION is missin fs.writeFileSync(path.join(dir, 'VERSION'), '0.0.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '0.5.0' }, null, 2) + '\n'); - const out = execFileSync('bun', [BIN, 'repair'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'repair'], { cwd: dir, timeout: 30_000 }).toString(); const result = JSON.parse(out); expect(result.repaired).toBe('0.0.0.0'); expect(result.packageJsonVersion).toBe('0.0.0'); @@ -673,7 +673,7 @@ describe('#2600: repair must not write fabricated 0.0.0.0 when VERSION is missin fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '0.5.0' }, null, 2) + '\n'); let code = 0; - try { execFileSync('bun', [BIN, 'repair'], { cwd: dir, stdio: 'pipe' }); } + try { execFileSync('bun', [BIN, 'repair'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); } catch (e: any) { code = e.status; } expect(code).toBe(2); expect(JSON.parse(fs.readFileSync(path.join(dir, 'package.json'), 'utf-8')).version).toBe('0.5.0'); @@ -692,7 +692,7 @@ describe('#2600: repair must not write fabricated 0.0.0.0 when VERSION is missin let code = 0; let stderr = ''; try { - execFileSync('bun', [BIN, 'repair'], { cwd: path.join(rootDir, 'app'), stdio: 'pipe' }); + execFileSync('bun', [BIN, 'repair'], { cwd: path.join(rootDir, 'app'), stdio: 'pipe', timeout: 30_000 }); } catch (e: any) { code = e.status; stderr = (e.stderr || '').toString(); @@ -717,13 +717,13 @@ describe('#2600: classify must surface versionFileExists=false when VERSION is m function makeRepoDir(): string { const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vbump-2600-classify-')); dirs.push(dir); - const git = (...a: string[]) => execFileSync('git', a, { cwd: dir, stdio: 'pipe' }); + const git = (...a: string[]) => execFileSync('git', a, { cwd: dir, stdio: 'pipe', timeout: 30_000 }); git('init', '-q', '-b', 'main'); git('config', 'user.email', 't@t'); git('config', 'user.name', 't'); // Commit with no VERSION file fs.writeFileSync(path.join(dir, 'README.md'), 'test\n'); git('add', '-A'); git('commit', '-q', '-m', 'base'); - const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: dir }).toString().trim(); + const head = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: dir, timeout: 30_000 }).toString().trim(); fs.mkdirSync(path.join(dir, '.git', 'refs', 'remotes', 'origin'), { recursive: true }); fs.writeFileSync(path.join(dir, '.git', 'refs', 'remotes', 'origin', 'main'), head + '\n'); return dir; @@ -734,7 +734,7 @@ describe('#2600: classify must surface versionFileExists=false when VERSION is m // No package.json: pkgExists=false, pkgAgrees=true, current===base → FRESH. // (A package.json with a non-zero version would cause DRIFT_UNEXPECTED.) - const out = execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: dir, timeout: 30_000 }).toString(); const result = JSON.parse(out); expect(result.versionFileExists).toBe(false); @@ -748,7 +748,7 @@ describe('#2600: classify must surface versionFileExists=false when VERSION is m fs.writeFileSync(path.join(dir, 'VERSION'), '0.2.0.0\n'); fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', version: '0.2.0.0' }, null, 2) + '\n'); - const out = execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: dir }).toString(); + const out = execFileSync('bun', [BIN, 'classify', '--base', 'main'], { cwd: dir, timeout: 30_000 }).toString(); const result = JSON.parse(out); expect(result.versionFileExists).toBe(true); @@ -784,7 +784,7 @@ describe('write --regen-digest regenerates the gstack agents digest (explicit op const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vbump-digest-')); fs.writeFileSync(path.join(dir, 'VERSION'), '1.0.0.0\n'); stubGenerator(dir); - const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0', '--regen-digest'], { cwd: dir }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0', '--regen-digest'], { cwd: dir, timeout: 30_000 }).toString()); expect(out.agentsDigest).toBe(true); expect(fs.readFileSync(path.join(dir, 'agents-digest', 'gstack-AGENTS.md'), 'utf-8')) .toBe('# gstack digest v1.1.0.0\n'); @@ -795,7 +795,7 @@ describe('write --regen-digest regenerates the gstack agents digest (explicit op const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vbump-digest-noflag-')); fs.writeFileSync(path.join(dir, 'VERSION'), '1.0.0.0\n'); stubGenerator(dir); - const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: dir }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0'], { cwd: dir, timeout: 30_000 }).toString()); expect(out.agentsDigest).toBe(null); // Digest untouched — the stub would have stamped v1.1.0.0 had it run. expect(fs.readFileSync(path.join(dir, 'agents-digest', 'gstack-AGENTS.md'), 'utf-8')) @@ -811,7 +811,7 @@ describe('write --regen-digest regenerates the gstack agents digest (explicit op fs.writeFileSync(path.join(d2, 'scripts', 'gen-agents-digest.ts'), 'process.exit(1);\n'); fs.writeFileSync(path.join(d2, 'agents-digest', 'gstack-AGENTS.md'), '# gstack digest v1.0.0.0\n'); - const res = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0', '--regen-digest'], { cwd: d2, stdio: 'pipe' }); + const res = execFileSync('bun', [BIN, 'write', '--version', '1.1.0.0', '--regen-digest'], { cwd: d2, stdio: 'pipe', timeout: 30_000 }); const out = JSON.parse(res.toString()); expect(out.wrote).toBe('1.1.0.0'); // the bump itself still lands expect(out.agentsDigest).toBe(false); @@ -834,7 +834,7 @@ describe('write --regen-digest regenerates the gstack agents digest (explicit op path.join(dir, 'agents-digest', 'gstack-AGENTS.md'), ); fs.writeFileSync(path.join(dir, 'VERSION'), '9.9.9.9\n'); - const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '9.9.10.0', '--regen-digest'], { cwd: dir }).toString()); + const out = JSON.parse(execFileSync('bun', [BIN, 'write', '--version', '9.9.10.0', '--regen-digest'], { cwd: dir, timeout: 30_000 }).toString()); expect(out.agentsDigest).toBe(true); const first = fs.readFileSync(path.join(dir, 'agents-digest', 'gstack-AGENTS.md'), 'utf-8').split('\n')[0]; expect(first).toContain('v9.9.10.0'); diff --git a/test/helpers/auq-sdk-capture.ts b/test/helpers/auq-sdk-capture.ts index 9bb08acf7..05603b9fa 100644 --- a/test/helpers/auq-sdk-capture.ts +++ b/test/helpers/auq-sdk-capture.ts @@ -281,13 +281,21 @@ export function carvedSkill(): { skillMd: string; sectionsFrom: string | null } }; } -/** Read the pre-carve verbose monolith plan-ceo SKILL.md from git. */ -export function verboseSkill(gitRef = 'ab66193e^'): string { - return execGit(['show', `${gitRef}:plan-ceo-review/SKILL.md`]); +/** Read the pre-carve verbose monolith plan-ceo SKILL.md. + * VENDORED fixture (v1.75 precedent), not a git ref: the old default + * `git show ab66193e^:...` pinned a BRANCH-LOCAL commit — it dies the day + * that branch is pruned and already fails on shallow clones. The fixture + * is the frozen pre-cut render; test/git-ref-fixture-tripwire.test.ts + * keeps this class from coming back. */ +export function verboseSkill(): string { + return fs.readFileSync( + path.join(ROOT, 'test', 'fixtures', 'auq-pre-cut-plan-ceo-review-SKILL.md'), + 'utf-8', + ); } function execGit(args: string[]): string { - const r = spawnSync('git', args, { cwd: ROOT, encoding: 'utf-8', maxBuffer: 64 * 1024 * 1024 }); + const r = spawnSync('git', args, { cwd: ROOT, encoding: 'utf-8', maxBuffer: 64 * 1024 * 1024, timeout: 30_000 }); if (r.status !== 0) throw new Error(`git ${args.join(' ')} failed: ${r.stderr}`); return r.stdout; } diff --git a/test/helpers/capture-parity-baseline.ts b/test/helpers/capture-parity-baseline.ts index 9971e266b..7bb794a80 100644 --- a/test/helpers/capture-parity-baseline.ts +++ b/test/helpers/capture-parity-baseline.ts @@ -141,8 +141,8 @@ function discoverEvalCoverage(repoRoot: string, skills: string[]): { function getGitInfo(repoRoot: string): { commit: string; branch: string } { try { - const commit = execSync('git rev-parse --short HEAD', { cwd: repoRoot, encoding: 'utf-8' }).trim(); - const branch = execSync('git rev-parse --abbrev-ref HEAD', { cwd: repoRoot, encoding: 'utf-8' }).trim(); + const commit = execSync('git rev-parse --short HEAD', { cwd: repoRoot, encoding: 'utf-8', timeout: 30_000 }).trim(); + const branch = execSync('git rev-parse --abbrev-ref HEAD', { cwd: repoRoot, encoding: 'utf-8', timeout: 30_000 }).trim(); return { commit, branch }; } catch { return { commit: 'unknown', branch: 'unknown' }; diff --git a/test/helpers/codex-session-runner.ts b/test/helpers/codex-session-runner.ts index 7d24248fd..4246c6c01 100644 --- a/test/helpers/codex-session-runner.ts +++ b/test/helpers/codex-session-runner.ts @@ -15,8 +15,11 @@ import * as fs from 'fs'; import * as path from 'path'; import * as os from 'os'; +import { spawn } from 'child_process'; +import { Readable } from 'node:stream'; import { hermeticChildEnv } from './hermetic-env'; import { extractSkillSections } from './skill-fixture'; +import { killProcessGroup } from '../../scripts/test-strict-output'; // --- Interfaces --- @@ -180,7 +183,7 @@ export async function runCodexSkill(opts: { const name = skillName || path.basename(skillDir) || 'gstack'; // Check if codex binary exists - const whichResult = Bun.spawnSync(['which', 'codex']); + const whichResult = Bun.spawnSync(['which', 'codex'], { timeout: 30_000 }); if (whichResult.exitCode !== 0) { return { output: 'SKIP: codex binary not found', @@ -232,28 +235,42 @@ export async function runCodexSkill(opts: { // Hermetic scrub (test/helpers/hermetic-env.ts) with codex's auth surface // re-admitted: codex auths from $HOME/.codex (copied into tempHome above) // plus OPENAI_API_KEY/CODEX_* when present. HOME override merges last. - const proc = Bun.spawn(['codex', ...args], { + // node:child_process spawn with `detached` (own process group) — mirrors + // session-runner.ts. Bun.spawn's bare proc.kill() signalled only codex + // itself; command subprocesses codex spawned survived as orphans holding + // our pipes open (the same blocked-drain hang the claude runner fixed — + // this copy never inherited that fix until now). + const proc = spawn('codex', args, { cwd: cwd || skillDir, - stdout: 'pipe', - stderr: 'pipe', + stdio: ['ignore', 'pipe', 'pipe'], + detached: process.platform !== 'win32', env: hermeticChildEnv( { HOME: tempHome, CODEX_HOME: tempCodexDir }, { extraAllow: ['OPENAI_API_KEY', 'CODEX_*'] }, ), }); + const stdoutWeb = Readable.toWeb(proc.stdout!) as ReadableStream; + const stderrWeb = Readable.toWeb(proc.stderr!) as ReadableStream; + const procExited: Promise = new Promise((resolve) => { + proc.on('close', (code) => resolve(code ?? 1)); + proc.on('error', () => resolve(1)); + }); // Race against timeout let timedOut = false; const timeoutId = setTimeout(() => { timedOut = true; - proc.kill(); + // Group SIGKILL + reader cancel: kill the whole tree AND unblock the + // read loop even if a stray grandchild survives the group kill. + killProcessGroup(proc, 'SIGKILL'); + reader.cancel().catch(() => { /* stream already closed */ }); }, timeoutMs); // Stream and collect JSONL from stdout const collectedLines: string[] = []; - const stderrPromise = new Response(proc.stderr).text(); + const stderrPromise = new Response(stderrWeb).text(); - const reader = proc.stdout.getReader(); + const reader = stdoutWeb.getReader(); const decoder = new TextDecoder(); let buf = ''; @@ -291,8 +308,18 @@ export async function runCodexSkill(opts: { collectedLines.push(buf); } - const stderr = await stderrPromise; - const exitCode = await proc.exited; + // Same orphan hazard as stdout: a grandchild holding stderr open would + // block this drain forever. Race it against child exit + a short grace + // window (ported from session-runner.ts — the codex copy lacked it). + const stderr = await Promise.race([ + stderrPromise, + (async () => { + await procExited; + await new Promise((r) => setTimeout(r, 5_000)); + return ''; + })(), + ]); + const exitCode = await procExited; clearTimeout(timeoutId); const durationMs = Date.now() - startTime; diff --git a/test/helpers/eval-store.test.ts b/test/helpers/eval-store.test.ts index c0769225b..cf02e28d8 100644 --- a/test/helpers/eval-store.test.ts +++ b/test/helpers/eval-store.test.ts @@ -114,6 +114,36 @@ describe('EvalCollector', () => { expect(data.total_duration_ms).toBe(3000); expect(data.timestamp).toBeTruthy(); expect(data.hostname).toBeTruthy(); + // CLI version stamping: always a non-empty string ('unknown' when the + // claude binary is absent — the field must exist either way so flake + // investigations can correlate runs with the TUI they exercised). + expect(typeof data.claude_cli_version).toBe('string'); + expect(data.claude_cli_version!.length).toBeGreaterThan(0); + }); + + test('a same-name re-record stamps attempts and surfaces flaky_retries', async () => { + // bun --retry re-runs the test BODY, so recordE2E fires again under the + // same name — the only reliable retry signal (bun's own output hides + // retried passes: fail→pass recaps as a clean pass, probed on 1.3.10). + const collector = new EvalCollector('e2e', tmpDir); + collector.addTest(makeEntry({ name: 'flaky-one', passed: false })); + collector.addTest(makeEntry({ name: 'flaky-one', passed: true })); + collector.addTest(makeEntry({ name: 'steady', passed: true })); + const filepath = await collector.finalize(); + + const data: EvalResult = JSON.parse(fs.readFileSync(filepath, 'utf-8')); + const attempts = data.tests.filter((t) => t.name === 'flaky-one').map((t) => t.attempt); + expect(attempts).toEqual([1, 2]); + expect(data.tests.find((t) => t.name === 'steady')?.attempt).toBe(1); + expect(data.flaky_retries).toEqual([{ name: 'flaky-one', attempts: 2 }]); + }); + + test('no retries → no flaky_retries field (absent, not empty)', async () => { + const collector = new EvalCollector('e2e', tmpDir); + collector.addTest(makeEntry({ name: 'only-once' })); + const filepath = await collector.finalize(); + const data: EvalResult = JSON.parse(fs.readFileSync(filepath, 'utf-8')); + expect('flaky_retries' in data).toBe(false); }); test('finalize creates directory if missing', async () => { diff --git a/test/helpers/eval-store.ts b/test/helpers/eval-store.ts index 06b68ceff..4542cbbd3 100644 --- a/test/helpers/eval-store.ts +++ b/test/helpers/eval-store.ts @@ -63,6 +63,12 @@ export interface EvalTestEntry { passed: boolean; duration_ms: number; cost_usd: number; + /** 1-based record attempt for this name in this run. bun's --retry leaves + * retried passes INVISIBLE in its text output (a fail→pass prints no + * (fail) line and recaps as a clean pass — probed on 1.3.10), so the ONLY + * reliable attempt signal is this in-process record: a retried test runs + * its body again and re-records under the same name. Set by addTest. */ + attempt?: number; // E2E transcript?: any[]; @@ -118,6 +124,10 @@ export interface EvalResult { git_sha: string; timestamp: string; hostname: string; + /** `claude --version` first line at run time (schema-additive, optional). + * TUI drift broke the PTY harness three times before runs recorded which + * CLI they actually exercised. */ + claude_cli_version?: string; tier: 'e2e' | 'llm-judge'; total_tests: number; passed: number; @@ -128,6 +138,11 @@ export interface EvalResult { tests: EvalTestEntry[]; /** Shard slug when the run was collected under /shards//. */ shard?: string; + /** Tests recorded more than once this run — the flake ledger for the paid + * lane. A test passing on attempt 2 every week used to read permanently + * green (the retry's entry was indistinguishable and bun's output hides + * retries entirely). Present only when non-empty. */ + flaky_retries?: Array<{ name: string; attempts: number }>; _partial?: boolean; // true for incremental saves, absent in final } @@ -177,6 +192,21 @@ export function isPartialEval(data: unknown, filename: string): boolean { return Boolean((data as { _partial?: unknown } | null)?._partial); } +/** + * Is this path a FINALIZED eval-store result file? Single owner of the + * filename taxonomy (manifest.json / slice-N.json are runner artifacts, + * _partial* are in-progress accumulators) — the paid runner's report mode + * and eval-flake-rank both consume this instead of re-encoding the rule + * (review finding: the rule lived in three places). + */ +export function isFinalizedEvalResultFile(relPath: string): boolean { + const base = path.basename(relPath); + if (!base.endsWith('.json')) return false; + if (base === 'manifest.json' || /^slice-\d+\.json$/.test(base)) return false; + if (base.startsWith('_partial')) return false; + return true; +} + /** * List eval JSON files in `evalDir` plus one level of `/shards//` * subdirectories (where the sharded paid runner points each shard's collector). @@ -777,6 +807,35 @@ function getVersion(): string { } } +// Cached per process: savePartial runs after EVERY test and must not pay a +// CLI spawn each time. Three separate harness breakages were traced to +// claude-CLI TUI drift only after long flake hunts — stamping the version +// into every run record makes that correlation a grep instead of an +// archaeology dig. +// +// GSTACK_CLAUDE_CLI_VERSION short-circuits the spawn entirely: the paid +// runner's parent resolves the version once and passes it to every shard, +// so test processes never block on it. The fallback spawn is SYNCHRONOUS on +// the same thread that polls PTY sessions — the judgePtyState blocking +// class — so its budget is a tight 3s, not a generous one: a slow/hung CLI +// costs one bounded stall per process and records 'unknown'. +let claudeCliVersionCache: string | null = null; +export function getClaudeCliVersion(): string { + if (claudeCliVersionCache !== null) return claudeCliVersionCache; + const fromEnv = process.env.GSTACK_CLAUDE_CLI_VERSION; + if (fromEnv) { + claudeCliVersionCache = fromEnv; + return claudeCliVersionCache; + } + try { + const result = spawnSync('claude', ['--version'], { stdio: 'pipe', timeout: 3_000 }); + claudeCliVersionCache = result.stdout?.toString().split('\n')[0].trim() || 'unknown'; + } catch { + claudeCliVersionCache = 'unknown'; + } + return claudeCliVersionCache; +} + export class EvalCollector { private tier: 'e2e' | 'llm-judge'; private tests: EvalTestEntry[] = []; @@ -792,10 +851,23 @@ export class EvalCollector { } addTest(entry: EvalTestEntry): void { - this.tests.push(entry); + // Same-name re-record = the test body ran again = bun retried it (test + // names are unique by convention). Stamp the 1-based attempt so a + // pass-on-attempt-2 stays visible forever — the stream hides it. + const prior = this.tests.filter((t) => t.name === entry.name).length; + this.tests.push({ ...entry, attempt: prior + 1 }); this.savePartial(); } + /** Names recorded more than once this run, with their attempt counts. */ + private flakyRetries(): Array<{ name: string; attempts: number }> { + const counts = new Map(); + for (const t of this.tests) counts.set(t.name, (counts.get(t.name) ?? 0) + 1); + return [...counts.entries()] + .filter(([, n]) => n > 1) + .map(([name, attempts]) => ({ name, attempts })); + } + /** Write incremental results after each test. Atomic write, non-fatal. */ savePartial(): void { try { @@ -812,6 +884,7 @@ export class EvalCollector { git_sha: git.sha, timestamp: new Date().toISOString(), hostname: os.hostname(), + claude_cli_version: getClaudeCliVersion(), tier: this.tier, total_tests: this.tests.length, passed, @@ -842,6 +915,7 @@ export class EvalCollector { const totalDuration = this.tests.reduce((s, t) => s + t.duration_ms, 0); const passed = this.tests.filter(t => t.passed).length; + const flaky = this.flakyRetries(); const result: EvalResult = { schema_version: SCHEMA_VERSION, version, @@ -849,6 +923,7 @@ export class EvalCollector { git_sha: git.sha, timestamp, hostname: os.hostname(), + claude_cli_version: getClaudeCliVersion(), tier: this.tier, total_tests: this.tests.length, passed, @@ -858,6 +933,7 @@ export class EvalCollector { wall_clock_ms: Date.now() - this.createdAt, tests: this.tests, ...(this.shard ? { shard: this.shard } : {}), + ...(flaky.length > 0 ? { flaky_retries: flaky } : {}), }; // Write eval file @@ -920,6 +996,12 @@ export class EvalCollector { const totalCost = `$${result.total_cost_usd.toFixed(2)}`; const totalDur = `${Math.round(result.total_duration_ms / 1000)}s`; lines.push(` Total: ${result.passed}/${result.total_tests} passed${' '.repeat(20)}${totalCost.padStart(6)} ${totalDur}`); + if (result.flaky_retries && result.flaky_retries.length > 0) { + // Loud, never fatal: a flaky pass must not block anyone, but it must + // never be silent either — that invisibility is how flakes calcified. + lines.push(` ⚠ FLAKY: ${result.flaky_retries.length} test(s) recorded multiple attempts this run: ` + + result.flaky_retries.map((f) => `${f.name} (x${f.attempts})`).join(', ')); + } lines.push(`Saved: ${filepath}`); process.stderr.write(lines.join('\n') + '\n'); diff --git a/test/helpers/gemini-session-runner.ts b/test/helpers/gemini-session-runner.ts index 2b080265d..e42f53ade 100644 --- a/test/helpers/gemini-session-runner.ts +++ b/test/helpers/gemini-session-runner.ts @@ -15,7 +15,10 @@ */ import * as path from 'path'; +import { spawn } from 'child_process'; +import { Readable } from 'node:stream'; import { hermeticChildEnv } from './hermetic-env'; +import { killProcessGroup } from '../../scripts/test-strict-output'; // --- Interfaces --- @@ -108,7 +111,7 @@ export async function runGeminiSkill(opts: { const startTime = Date.now(); // Check if gemini binary exists - const whichResult = Bun.spawnSync(['which', 'gemini']); + const whichResult = Bun.spawnSync(['which', 'gemini'], { timeout: 30_000 }); if (whichResult.exitCode !== 0) { return { output: 'SKIP: gemini binary not found', @@ -130,27 +133,40 @@ export async function runGeminiSkill(opts: { // Spawn gemini — uses real HOME for auth (~/.gemini; HOME is allowlisted), // cwd for skill discovery. Hermetic scrub with gemini's auth surface // re-admitted (previously this spawn inherited the full operator env). - const proc = Bun.spawn(['gemini', ...args], { + // node:child_process spawn with `detached` (own process group) — mirrors + // session-runner.ts. A bare kill signalled only gemini itself; tool + // subprocesses survived as orphans holding our pipes open (the same + // blocked-drain hang the claude runner fixed — this copy lacked it). + const proc = spawn('gemini', args, { cwd: cwd || process.cwd(), - stdout: 'pipe', - stderr: 'pipe', + stdio: ['ignore', 'pipe', 'pipe'], + detached: process.platform !== 'win32', env: hermeticChildEnv(undefined, { extraAllow: ['GEMINI_API_KEY', 'GOOGLE_API_KEY', 'GOOGLE_APPLICATION_CREDENTIALS', 'GOOGLE_CLOUD_*', 'GEMINI_*'], }), }); + const stdoutWeb = Readable.toWeb(proc.stdout!) as ReadableStream; + const stderrWeb = Readable.toWeb(proc.stderr!) as ReadableStream; + const procExited: Promise = new Promise((resolve) => { + proc.on('close', (code) => resolve(code ?? 1)); + proc.on('error', () => resolve(1)); + }); // Race against timeout let timedOut = false; const timeoutId = setTimeout(() => { timedOut = true; - proc.kill(); + // Group SIGKILL + reader cancel: kill the whole tree AND unblock the + // read loop even if a stray grandchild survives the group kill. + killProcessGroup(proc, 'SIGKILL'); + reader.cancel().catch(() => { /* stream already closed */ }); }, timeoutMs); // Stream and collect JSONL from stdout const collectedLines: string[] = []; - const stderrPromise = new Response(proc.stderr).text(); + const stderrPromise = new Response(stderrWeb).text(); - const reader = proc.stdout.getReader(); + const reader = stdoutWeb.getReader(); const decoder = new TextDecoder(); let buf = ''; @@ -185,8 +201,18 @@ export async function runGeminiSkill(opts: { collectedLines.push(buf); } - const stderr = await stderrPromise; - const exitCode = await proc.exited; + // Same orphan hazard as stdout: a grandchild holding stderr open would + // block this drain forever. Race against child exit + a short grace window + // (ported from session-runner.ts — the gemini copy lacked it). + const stderr = await Promise.race([ + stderrPromise, + (async () => { + await procExited; + await new Promise((r) => setTimeout(r, 5_000)); + return ''; + })(), + ]); + const exitCode = await procExited; clearTimeout(timeoutId); const durationMs = Date.now() - startTime; diff --git a/test/helpers/session-runner.ts b/test/helpers/session-runner.ts index f433332f5..2bbababea 100644 --- a/test/helpers/session-runner.ts +++ b/test/helpers/session-runner.ts @@ -9,8 +9,11 @@ import * as fs from 'fs'; import * as path from 'path'; import * as os from 'os'; +import { spawn } from 'child_process'; +import { Readable } from 'node:stream'; import { getProjectEvalDir } from './eval-store'; import { hermeticChildEnv, isHermeticEnabled } from './hermetic-env'; +import { killProcessGroup } from '../../scripts/test-strict-output'; const GSTACK_DEV_DIR = path.join(os.homedir(), '.gstack-dev'); const HEARTBEAT_PATH = path.join(GSTACK_DEV_DIR, 'e2e-live.json'); // heartbeat stays global @@ -52,6 +55,14 @@ export interface SkillTestResult { maxInterTurnMs: number; } +/** Local default startup grace: 90s covers observed API queue latency + * (60-90s receipts) without letting a dead API burn a 600s budget. */ +export const STARTUP_GRACE_MS = 90_000; +/** CI floor (TODOS-filed): shared runners queue harder; killing startup + * before 300s in CI converts ordinary queueing into false failures. + * Pinned by test/session-runner-startup-grace.test.ts. */ +export const STARTUP_GRACE_CI_FLOOR_MS = 300_000; + const BROWSE_ERROR_PATTERNS = [ /Unknown command: \w+/, /Unknown snapshot flag: .+/, @@ -131,6 +142,15 @@ export async function runSkillTest(options: { * per-test GSTACK_HOME overrides so the test doesn't have to spell out * env setup in the prompt itself. */ env?: Record; + /** Startup-phase deadline: if NO NDJSON byte arrives within this window, + * the run is killed EARLY with exitReason 'timeout_startup' instead of + * burning the whole work budget waiting on an API that is not answering + * (the recurring '0 turns / $0.00' class — four budget-bump receipts). + * Defaults to min(STARTUP_GRACE_MS, timeout); the CI floor is higher + * because CI queueing is real. Total wall stays <= timeout either way — + * bun-level tier budgets are sized to the runner timeout with no margin, + * so this phase split must never extend the envelope. */ + startupGraceMs?: number; }): Promise { const { prompt, @@ -142,6 +162,15 @@ export async function runSkillTest(options: { runId, env: extraEnv, } = options; + // The CI floor is a FLOOR, not a default: an explicit startupGraceMs below + // 300s in CI would re-open the queueing-becomes-false-red hole the floor + // exists for (review finding — the name promised a clamp the code lacked). + // Local runs honor the caller verbatim; timeout still caps everything. + const requestedGrace = options.startupGraceMs ?? (process.env.CI ? STARTUP_GRACE_CI_FLOOR_MS : STARTUP_GRACE_MS); + const startupGraceMs = Math.min( + process.env.CI ? Math.max(requestedGrace, STARTUP_GRACE_CI_FLOOR_MS) : requestedGrace, + timeout, + ); const model = options.model ?? process.env.EVALS_MODEL ?? 'claude-sonnet-4-6'; const startTime = Date.now(); @@ -174,8 +203,13 @@ export async function runSkillTest(options: { if (isHermeticEnabled()) args.push('--strict-mcp-config'); // Spawn claude directly with array-form args (no shell interpolation). - // Prompt is piped via stdin using a Blob to avoid temp files and shell escaping. - const proc = Bun.spawn(['claude', ...args], { + // node:child_process spawn (not Bun.spawn): `detached` puts the child in + // its OWN process group, so the timeout handler can killpg the whole tree. + // Bun.spawn has no detached option, and its bare proc.kill() signalled only + // claude itself — tool subprocesses claude spawned survived as orphans + // burning shared API rate for the rest of the shard's lifetime. + // Prompt is piped via stdin to avoid temp files and shell escaping. + const proc = spawn('claude', args, { cwd: workingDirectory, // Hermetic by default (see test/helpers/hermetic-env.ts): operator // session context (CONDUCTOR_*, CLAUDECODE, ~/.claude config, ~/.gstack) @@ -185,38 +219,78 @@ export async function runSkillTest(options: { // suite exercising the INTERACTIVE prose-fallback path opts out by passing // `env: { GSTACK_HEADLESS: '' }` — extraEnv wins because it spreads last. env: hermeticChildEnv({ GSTACK_HEADLESS: '1', ...extraEnv }), - stdin: new Blob([prompt]), - stdout: 'pipe', - stderr: 'pipe', + stdio: ['pipe', 'pipe', 'pipe'], + detached: process.platform !== 'win32', + }); + proc.stdin!.on('error', () => { /* child died before reading the prompt — exit handling reports it */ }); + proc.stdin!.write(prompt); + proc.stdin!.end(); + const stdoutWeb = Readable.toWeb(proc.stdout!) as ReadableStream; + const stderrWeb = Readable.toWeb(proc.stderr!) as ReadableStream; + // 'exit' vs 'close' matters here: 'close' waits for stdout/stderr to + // drain, which an orphaned grandchild can hold open long after claude + // itself died with a REAL exit code — labeling must key off 'exit' or an + // auth failure gets triaged as 'timeout_startup' availability noise + // (claude adversarial finding). procExited stays 'close'-based (streams + // complete) for the drain race below. + let childExited = false; + const procExited: Promise = new Promise((resolve) => { + proc.on('exit', () => { childExited = true; }); + proc.on('close', (code) => { childExited = true; resolve(code ?? 1); }); + proc.on('error', () => { childExited = true; resolve(1); }); }); - // Race against timeout + // Two-phase timeout. Phase 1 (startup): no NDJSON byte yet — a shorter + // deadline kills a non-answering API run EARLY and names it, instead of + // the old single timer burning the full work budget to produce an opaque + // '0 turns / $0.00' failure. Phase 2 (work): armed by the read loop when + // the FIRST byte arrives, for the REMAINING budget — total wall is always + // <= timeout (tier envelopes are margin-free by convention). let stderr = ''; let exitReason = 'unknown'; let timedOut = false; + let timedOutInStartup = false; + let phaseTimer: ReturnType; - const timeoutId = setTimeout(() => { - timedOut = true; - proc.kill(); - // proc.kill() signals claude itself (direct spawn, no shell wrapper), - // but tool subprocesses claude spawned can survive as orphans that - // inherited our stdout/stderr pipes, so without cancel() the read loop - // below blocks until the orphan finally exits (observed: a 600s timeout - // stretching past 1400s and tripping bun's per-test timeout instead of - // returning a result). + const killRun = (startupPhase: boolean): void => { + // Labeling and unblocking are SEPARATE concerns: a timer firing after + // the child already exited must not relabel a real exit (auth error, + // crash) as a timeout — but it must STILL group-kill and cancel the + // reader, or an orphan holding the pipes re-creates the exact + // blocked-drain hang this runner fixed (an early `return` here was the + // bug the adversarial pass caught in the first version of this guard). + if (!childExited) { + timedOut = true; + timedOutInStartup = startupPhase; + } + // Group SIGKILL (mirrors runShardChild): claude AND every tool + // subprocess it spawned die together — a bare proc.kill() left orphans + // that inherited our stdout/stderr pipes and kept the API burning + // (observed: a 600s timeout stretching past 1400s while an orphan held + // the pipes open). + killProcessGroup(proc, 'SIGKILL'); + // Belt and braces with the group kill: even if an orphan survives (EPERM + // fallback path), cancel() unblocks the read loop below. reader.cancel().catch(() => { /* stream already closed */ }); - }, timeout); + }; + phaseTimer = setTimeout(() => killRun(true), startupGraceMs); + /** Called once by the read loop on the first NDJSON byte. */ + const armWorkPhase = (elapsedMs: number): void => { + clearTimeout(phaseTimer); + phaseTimer = setTimeout(() => killRun(false), Math.max(0, timeout - elapsedMs)); + }; // Stream NDJSON from stdout for real-time progress const collectedLines: string[] = []; let liveTurnCount = 0; let liveToolCount = 0; let firstResponseMs = 0; + let workPhaseArmed = false; let lastToolTime = 0; let maxInterTurnMs = 0; - const stderrPromise = new Response(proc.stderr).text(); + const stderrPromise = new Response(stderrWeb).text(); - const reader = proc.stdout.getReader(); + const reader = stdoutWeb.getReader(); const decoder = new TextDecoder(); let buf = ''; @@ -232,8 +306,15 @@ export async function runSkillTest(options: { collectedLines.push(line); // Track time to first NDJSON line (measures latency from spawn to first Claude response) - if (firstResponseMs === 0) { + if (!workPhaseArmed) { + // Flag, not `firstResponseMs === 0`: a first line landing in the + // same millisecond as spawn would read as "not yet seen" and leave + // the startup timer live for the whole run (claude adversarial). + workPhaseArmed = true; firstResponseMs = Date.now() - startTime; + // First byte: startup phase over — arm the work phase for the + // REMAINING budget (total wall stays <= timeout). + armWorkPhase(firstResponseMs); } // Real-time progress to stderr + persistent logs @@ -304,16 +385,20 @@ export async function runSkillTest(options: { stderr = await Promise.race([ stderrPromise, (async () => { - await proc.exited; + await procExited; await new Promise((r) => setTimeout(r, 5_000)); return ''; })(), ]); - const exitCode = await proc.exited; - clearTimeout(timeoutId); + const exitCode = await procExited; + clearTimeout(phaseTimer); if (timedOut) { - exitReason = 'timeout'; + // 'timeout_startup' = the API never sent a byte inside the grace — an + // availability problem, not a test failure worth reading transcripts + // for. Distinct so triage (and WS10's inconclusive classification) can + // key off it without receipts archaeology. + exitReason = timedOutInStartup ? 'timeout_startup' : 'timeout'; } else if (exitCode === 0) { exitReason = 'success'; } else { diff --git a/test/helpers/touchfiles-data.ts b/test/helpers/touchfiles-data.ts index c9cc674e1..11a05a4ec 100644 --- a/test/helpers/touchfiles-data.ts +++ b/test/helpers/touchfiles-data.ts @@ -129,12 +129,9 @@ export const E2E_TOUCHFILES: Record = { // Real-PTY E2E batch (#6 new tests on the harness). // Each one tests behavior the SDK harness can't observe (rendered TTY, // numbered-option lists, multi-phase ordering, idempotency state echo). - 'preamble-script-ab': ['bin/gstack-skill-start', 'bin/gstack-skill-end', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-brain-sync-block.ts', 'scripts/resolvers/preamble.ts', 'plan-ceo-review/**', 'test/helpers/auq-sdk-capture.ts', 'test/skill-e2e-preamble-script-ab.test.ts'], 'auq-format-gate': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'test/helpers/llm-judge.ts', 'test/skill-e2e-ask-user-question-format-compliance.test.ts'], - 'auq-repetition-cut-ab': ['scripts/resolvers/preamble/generate-ask-user-format.ts', 'plan-ceo-review/**', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/llm-judge.ts', 'test/fixtures/auq-pre-cut-plan-ceo-review-SKILL.md', 'test/skill-e2e-auq-repetition-cut-ab.test.ts'], 'plan-ceo-mode-routing': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/claude-pty-runner.ts', 'test/skill-e2e-plan-ceo-mode-routing.test.ts'], 'plan-design-with-ui-scope': ['plan-design-review/**', 'test/fixtures/plans/ui-heavy-feature.md', 'test/helpers/claude-pty-runner.ts', 'test/skill-e2e-plan-design-with-ui.test.ts'], - 'budget-regression-pty': ['test/helpers/eval-store.ts', 'test/skill-budget-regression.test.ts'], 'ship-idempotency-pty': ['ship/**', 'bin/gstack-next-version', 'bin/gstack-version-bump', 'scripts/resolvers/sections.ts', 'lib/worktree.ts', 'test/helpers/claude-pty-runner.ts', 'test/skill-e2e-ship-idempotency.test.ts'], 'tpa-present': ['scripts/resolvers/third-party-actions.ts', 'ship/SKILL.md.tmpl', 'ship/sections/apple-release.md.tmpl', 'scripts/gen-skill-docs.ts', 'test/helpers/session-runner.ts', 'test/skill-e2e-third-party-actions.test.ts'], 'tpa-absent-linux': ['scripts/resolvers/third-party-actions.ts', 'ship/SKILL.md.tmpl', 'ship/sections/apple-release.md.tmpl', 'scripts/gen-skill-docs.ts', 'test/helpers/session-runner.ts', 'test/skill-e2e-third-party-actions.test.ts'], @@ -149,7 +146,6 @@ export const E2E_TOUCHFILES: Record = { // skill (D-CODEX A). Touching the registry/helper or sections.ts runs all. 'carve-section-loading': ['design-html/**', 'design-shotgun/**', 'qa/**', 'browse/**', 'retro/**', 'autoplan/**', 'spec/**', 'setup-gbrain/**', 'review/**', 'codex/**', 'land-and-deploy/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'office-hours/**', 'document-release/**', 'design-consultation/**', 'cso/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'test/carve-section-loading.test.ts'], 'autoplan-chain-pty': ['autoplan/**', 'plan-ceo-review/**', 'plan-design-review/**', 'plan-eng-review/**', 'plan-devex-review/**', 'test/fixtures/plans/ui-heavy-feature.md', 'test/helpers/claude-pty-runner.ts', 'test/skill-e2e-autoplan-chain.test.ts'], - 'e2e-harness-audit': ['bin/gstack-skill-start', 'bin/gstack-skill-end', 'plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/claude-pty-runner.ts'], // Per-finding AskUserQuestion count + review-report-at-bottom assertion. // Each test drives its skill end-to-end; touchfiles include preamble + @@ -207,13 +203,6 @@ export const E2E_TOUCHFILES: Record = { 'plan-review-prosons-neutral-neg': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md', 'test/skill-e2e-plan-prosons.test.ts'], // Expanded coverage (CT3) — 6 non-plan-review skills inherit Pros/Cons via preamble - 'ship-prosons-format': ['ship/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'], - 'office-hours-prosons-format': ['office-hours/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'], - 'investigate-prosons-format': ['investigate/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'], - 'qa-prosons-format': ['qa/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'], - 'review-prosons-format': ['review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'], - 'design-review-prosons-format': ['design-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'], - 'document-release-prosons-format': ['document-release/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'], // /plan-tune (v1 observational) 'plan-tune-inspect': ['plan-tune/**', 'scripts/question-registry.ts', 'scripts/psychographic-signals.ts', 'scripts/one-way-doors.ts', 'bin/gstack-question-log', 'bin/gstack-question-preference', 'bin/gstack-developer-profile', 'test/skill-e2e-plan-tune.test.ts'], @@ -240,9 +229,6 @@ export const E2E_TOUCHFILES: Record = { 'retro': ['bin/gstack-retro-metrics', 'retro/**', 'test/skill-e2e-retro.test.ts'], 'retro-base-branch': ['bin/gstack-retro-metrics', 'retro/**', 'test/skill-e2e-retro.test.ts'], - // Global discover - 'global-discover': ['bin/gstack-global-discover.ts', 'test/global-discover.test.ts'], - // CSO 'cso-full-audit': ['cso/**', 'test/skill-e2e-cso.test.ts'], 'cso-diff-mode': ['cso/**', 'test/skill-e2e-cso.test.ts'], @@ -277,14 +263,14 @@ export const E2E_TOUCHFILES: Record = { 'codex-review': ['codex/**', 'test/skill-e2e-workflow.test.ts'], // Codex E2E (tests skills via Codex CLI + worktree) - 'codex-discover-skill': ['codex/**', '.agents/skills/**', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts', 'test/codex-e2e.test.ts'], - 'codex-review-findings': ['review/**', '.agents/skills/gstack-review/**', 'codex/**', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts', 'test/codex-e2e.test.ts'], + 'codex-discover-skill': ['codex/**', 'scripts/gen-skill-docs.ts', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts', 'test/codex-e2e.test.ts'], + 'codex-review-findings': ['review/**', 'scripts/gen-skill-docs.ts', 'codex/**', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts', 'test/codex-e2e.test.ts'], // GPT-5.6 Sol scope-termination E2E (Codex CLI, full generated investigate skill) 'codex-sol-scope-termination': ['model-overlays/gpt-5.6-sol.md', 'scripts/models.ts', 'scripts/resolvers/model-overlay.ts', 'scripts/resolvers/preamble/**', 'investigate/**', 'test/helpers/codex-session-runner.ts', 'test/codex-e2e-sol-scope.test.ts'], // Gemini E2E — smoke test only (Gemini gets lost in worktrees on complex tasks) - 'gemini-smoke': ['.agents/skills/**', 'test/helpers/gemini-session-runner.ts', 'lib/worktree.ts', 'test/gemini-e2e.test.ts'], + 'gemini-smoke': ['scripts/gen-skill-docs.ts', 'test/helpers/gemini-session-runner.ts', 'lib/worktree.ts', 'test/gemini-e2e.test.ts'], // Coverage audit (shared fixture) + triage + gates @@ -309,11 +295,6 @@ export const E2E_TOUCHFILES: Record = { 'test/skill-e2e-docsync-spawned.test.ts', ], - // Plan completion audit + verification - 'ship-plan-completion': ['ship/**', 'scripts/gen-skill-docs.ts'], - 'ship-plan-verification': ['ship/**', 'qa-only/**', 'scripts/gen-skill-docs.ts'], - 'review-plan-completion': ['review/**', 'scripts/gen-skill-docs.ts'], - // Design 'design-consultation-core': ['design-consultation/**', 'scripts/gen-skill-docs.ts', 'test/helpers/llm-judge.ts', 'test/skill-e2e-design.test.ts'], 'design-consultation-existing': ['design-consultation/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-design.test.ts'], @@ -322,11 +303,6 @@ export const E2E_TOUCHFILES: Record = { 'plan-design-review-no-ui-scope': ['plan-design-review/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-design.test.ts'], 'design-review-fix': ['design-review/**', 'browse/src/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-design.test.ts'], - // Design Shotgun - 'design-shotgun-path': ['design-shotgun/**', 'design/src/**', 'scripts/resolvers/design.ts'], - 'design-shotgun-session': ['design-shotgun/**', 'scripts/resolvers/design.ts'], - 'design-shotgun-full': ['design-shotgun/**', 'design/src/**', 'browse/src/**'], - // /diagram (diagram-render bundle consumers). Triplet = deterministic // functional (gate); authoring quality = LLM-judged benchmark (periodic). 'diagram-triplet': ['diagram/**', 'lib/diagram-render/**', 'browse/src/write-commands.ts', 'browse/src/read-commands.ts', 'test/skill-e2e-diagram.test.ts'], @@ -345,7 +321,6 @@ export const E2E_TOUCHFILES: Record = { // Autoplan - 'autoplan-core': ['autoplan/**', 'plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**'], 'autoplan-dual-voice': ['autoplan/**', 'codex/**', 'bin/gstack-codex-probe', 'scripts/resolvers/review.ts', 'scripts/resolvers/design.ts', 'test/skill-e2e-autoplan-dual-voice.test.ts'], // Multi-provider benchmark adapters — live API smoke against real claude/codex/gemini CLIs @@ -538,7 +513,6 @@ export const E2E_TIERS: Record = { 'review-base-branch': 'gate', 'review-design-lite': 'periodic', // 4/7 threshold is subjective 'review-coverage-audit': 'gate', - 'review-plan-completion': 'gate', 'review-dashboard-via': 'gate', // Review Army — gate for core functionality, periodic for multi-specialist @@ -576,7 +550,7 @@ export const E2E_TIERS: Record = { 'plan-ceo-review': 'periodic', 'plan-ceo-review-selective': 'periodic', 'plan-ceo-review-benefits': 'gate', - 'plan-ceo-review-expansion-energy': 'gate', // V1.1 mode-posture regression gate (Opus generator, Sonnet judge) + 'plan-ceo-review-expansion-energy': 'periodic', // Demoted from gate (2026-08 audit): Opus generator + subjective 2-axis >=4/5 LLM-judge threshold in the merge lane — the exact class siblings were demoted for (a +21-line preamble change once flipped the score). CLAUDE.md's own rule: Opus model test -> periodic. 'plan-eng-review': 'periodic', 'plan-eng-review-artifact': 'periodic', 'plan-eng-coverage-audit': 'gate', @@ -598,17 +572,13 @@ export const E2E_TIERS: Record = { 'office-hours-auto-mode': 'gate', 'auto-decide-preserved': 'periodic', 'conductor-prose': 'periodic', - 'e2e-harness-audit': 'gate', // Real-PTY E2E batch — tier classification: // gate: cheap, deterministic, run on every PR // periodic: long-running or expensive (>$3/run), run weekly - 'preamble-script-ab': 'periodic', // Phase 1-3 A/B: script vs inline preamble; demoted post-Phase-3 (OV7) - 'auq-repetition-cut-ab': 'periodic', // AUQ repetition-cut NOT-WORSE gate (passed pre-landing). Periodic runs force EVALS_ALL, so the dep list cannot auto-trigger it — an AUQ format edit carries a MANUAL re-run obligation (bun test test/skill-e2e-auq-repetition-cut-ab.test.ts with EVALS=1 EVALS_TIER=periodic) 'auq-format-gate': 'gate', // ~$0.50/run, SDK capture, single skill probe 'plan-ceo-mode-routing': 'periodic', // ~$3/run, deep navigation through 8-12 prior AskUserQuestions 'plan-design-with-ui-scope': 'gate', // ~$0.80/run - 'budget-regression-pty': 'gate', // free, library-only assertion 'ship-idempotency-pty': 'periodic', // ~$3/run, real /ship in plan mode 'tpa-present': 'gate', // consent/credential safety guardrail; deterministic shims + grep asserts 'tpa-absent-linux': 'gate', // consent/credential safety guardrail; deterministic shims + grep asserts @@ -671,13 +641,6 @@ export const E2E_TIERS: Record = { 'plan-review-prosons-neutral-neg': 'periodic', // CT3 expanded coverage — non-plan-review skills inheriting Pros/Cons (all periodic) - 'ship-prosons-format': 'periodic', - 'office-hours-prosons-format': 'periodic', - 'investigate-prosons-format': 'periodic', - 'qa-prosons-format': 'periodic', - 'review-prosons-format': 'periodic', - 'design-review-prosons-format': 'periodic', - 'document-release-prosons-format': 'periodic', // /plan-tune — gate (core v1 DX promise: plain-English intent routing) 'plan-tune-inspect': 'gate', @@ -718,16 +681,15 @@ export const E2E_TIERS: Record = { 'ship-triage': 'gate', 'ship-docsync': 'gate', 'docsync-spawned': 'gate', // #2733 JSON-contract-through-a-firing-gate proof (deterministic safety) - 'ship-plan-completion': 'gate', - 'ship-plan-verification': 'gate', + // (merge note: main's side also re-added ship-plan-completion / + // ship-plan-verification here — phantom keys with no declaring test, + // deleted by the census-integrity commit; the reverse invariant in + // test/touchfiles.test.ts now fails the suite if they come back.) // Retro — gate for cheap branch detection, periodic for full Opus retro 'retro': 'periodic', 'retro-base-branch': 'gate', - // Global discover - 'global-discover': 'gate', - // CSO — gate for security guardrails, periodic for quality 'cso-full-audit': 'periodic', // D2a demotion 2026-08: 250s/$0.57 full audit; cso targeted tests stay gate 'cso-diff-mode': 'gate', @@ -755,9 +717,6 @@ export const E2E_TIERS: Record = { 'design-consultation-preview': 'periodic', // D2a demotion 2026-08 ($0.89/481s) 'plan-design-review-no-ui-scope': 'gate', 'design-review-fix': 'periodic', - 'design-shotgun-path': 'gate', - 'design-shotgun-session': 'gate', - 'design-shotgun-full': 'periodic', // /diagram — triplet is deterministic functional, judge is a quality benchmark 'diagram-triplet': 'gate', @@ -776,7 +735,6 @@ export const E2E_TIERS: Record = { // Autoplan — periodic (not yet implemented) - 'autoplan-core': 'periodic', 'autoplan-dual-voice': 'periodic', // Multi-provider benchmark — periodic (requires external CLIs + auth, paid) @@ -809,9 +767,14 @@ export const E2E_TIERS: Record = { 'overlay-harness-opus-4-7-fanout-toy': 'periodic', 'overlay-harness-opus-4-7-fanout-realistic': 'periodic', - // /ios-qa daemon + codegen — no-device path runs every PR (no hardware - // dependency, deterministic). with-device path requires GSTACK_HAS_IOS_DEVICE. - 'ios-qa-e2e': 'gate', + // /ios-qa daemon + codegen. Demoted gate -> periodic (2026-08 audit): the + // gate declaration was never executable in CI — the file sits in + // PERIODIC_CI_EXCLUDE ("not a CI runner capability"), but that exclusion + // only applies at tier=periodic, so the gate lane planned a HOLLOW shard + // on every Linux PR. Periodic keeps it in the weekly census on capable + // hosts; re-promote if a macOS runner lands (flagged decision in the + // test-infra overhaul plan). + 'ios-qa-e2e': 'periodic', // Swift toolchain only, no device required, but heavier than TS unit tests. 'ios-qa-swift-build': 'periodic', // Requires a real connected + paired iPhone. Manual-trigger only. @@ -856,10 +819,6 @@ export const LLM_JUDGE_TOUCHFILES: Record = { 'design-review/SKILL.md fix loop': ['design-review/SKILL.md', 'design-review/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts'], 'design-consultation/SKILL.md research': ['design-consultation/SKILL.md', 'design-consultation/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts'], - // Office Hours - 'office-hours/SKILL.md spec review': ['office-hours/SKILL.md', 'office-hours/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'], - 'office-hours/SKILL.md design sketch': ['office-hours/SKILL.md', 'office-hours/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'], - // Deploy skills 'land-and-deploy/SKILL.md workflow': ['land-and-deploy/SKILL.md', 'land-and-deploy/SKILL.md.tmpl', 'land-and-deploy/sections/**', 'test/skill-llm-eval.test.ts'], 'canary/SKILL.md monitoring loop': ['canary/SKILL.md', 'canary/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts'], diff --git a/test/heredoc-pipe-deadlock.test.ts b/test/heredoc-pipe-deadlock.test.ts index 0518fc366..47df8d81c 100644 --- a/test/heredoc-pipe-deadlock.test.ts +++ b/test/heredoc-pipe-deadlock.test.ts @@ -31,7 +31,7 @@ const MAX_BODY = 64 * 1024; const GUARD_RE = /^\s*(?::\s*"\$\{)?BASH_COMPAT(?:[:=]|\}")/m; function trackedShellScripts(): string[] { - const out = execSync('git ls-files', { cwd: ROOT, encoding: 'utf-8', maxBuffer: 32 * 1024 * 1024 }); + const out = execSync('git ls-files', { cwd: ROOT, encoding: 'utf-8', maxBuffer: 32 * 1024 * 1024, timeout: 30_000 }); return out .split('\n') .map((s) => s.trim()) @@ -88,6 +88,7 @@ describe('heredoc pipe-deadlock guard', () => { test('the guard actually moves the body off the pipe', () => { const bash = spawnSync('bash', ['-c', 'echo "${BASH_VERSINFO[0]}.${BASH_VERSINFO[1]}"'], { encoding: 'utf-8', + timeout: 30_000, }); const version = (bash.stdout ?? '').trim(); const [maj, min] = version.split('.').map((n) => parseInt(n, 10)); @@ -102,6 +103,7 @@ describe('heredoc pipe-deadlock guard', () => { // probe would answer OTHER for an unobservable fd. Skip rather than fail. const devStdin = spawnSync('bash', ['-c', '[ -e /dev/stdin ] && echo yes || echo no'], { encoding: 'utf-8', + timeout: 30_000, }); if ((devStdin.stdout ?? '').trim() !== 'yes') return; @@ -114,7 +116,7 @@ $body EOF `; const run = (guard: string) => - (spawnSync('bash', ['-c', probe(guard)], { encoding: 'utf-8' }).stdout ?? '').trim(); + (spawnSync('bash', ['-c', probe(guard)], { encoding: 'utf-8', timeout: 30_000 }).stdout ?? '').trim(); expect(run('')).toBe('PIPE'); expect(run('BASH_COMPAT=50')).toBe('TEMPFILE'); diff --git a/test/host-config.test.ts b/test/host-config.test.ts index cad209557..7cc31ccc8 100644 --- a/test/host-config.test.ts +++ b/test/host-config.test.ts @@ -330,7 +330,7 @@ describe('host-config-export.ts CLI', () => { function run(...args: string[]): { stdout: string; stderr: string; exitCode: number } { const result = Bun.spawnSync(['bun', 'run', EXPORT_SCRIPT, ...args], { - cwd: ROOT, stdout: 'pipe', stderr: 'pipe', + cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 30_000, }); return { stdout: result.stdout.toString().trim(), @@ -445,7 +445,7 @@ describe('golden-file regression', () => { for (const host of ['codex', 'factory']) { const result = Bun.spawnSync( ['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', host, '--out-dir', GOLDEN_OUT], - { cwd: ROOT }, + { cwd: ROOT, timeout: 120_000 }, ); if (result.exitCode !== 0) { throw new Error( diff --git a/test/ios-qa-regen.test.ts b/test/ios-qa-regen.test.ts index 0378869dd..2aa39884c 100644 --- a/test/ios-qa-regen.test.ts +++ b/test/ios-qa-regen.test.ts @@ -105,7 +105,7 @@ describe('gstack-ios-qa-regen', () => { const workDir = mkdtempSync(join(tmpdir(), 'ios-qa-regen-')); workDirs.push(workDir); const { launcher } = copyIntoFakeInstall(workDir); - const result = spawnSync('bash', [launcher, '--app-source', workDir], { encoding: 'utf8' }); + const result = spawnSync('bash', [launcher, '--app-source', workDir], { encoding: 'utf8', timeout: 30_000 }); expect(result.status).toBe(2); expect(result.stderr).toContain('both --app-source and --bridge-dir are required'); @@ -134,6 +134,7 @@ describe('gstack-ios-qa-regen', () => { ], { encoding: 'utf8', env: { ...process.env, PATH: `${fakeBin}:${process.env.PATH ?? ''}` }, + timeout: 30_000, }); expect(result.status).toBe(17); @@ -184,7 +185,7 @@ final class AppState { GEN_ACCESSORS_REV: 'regen-test', }; const args = [launcher, '--app-source', appSource, '--bridge-dir', bridgeDir]; - const first = spawnSync('bash', args, { encoding: 'utf8', env }); + const first = spawnSync('bash', args, { encoding: 'utf8', env, timeout: 30_000 }); expect(first.status).toBe(0); expect(first.stderr).toBe(''); @@ -232,10 +233,11 @@ final class AppState { expect(installedContents).not.toContain('FORBIDDEN-STATE-SENTINEL'); expect(installedContents).not.toContain('OBSOLETE-HARNESS-SENTINEL'); - const swiftAvailable = spawnSync('swift', ['--version'], { encoding: 'utf8' }).status === 0; + const swiftAvailable = spawnSync('swift', ['--version'], { encoding: 'utf8', timeout: 30_000 }).status === 0; if (swiftAvailable) { const dump = spawnSync('swift', ['package', 'dump-package', '--package-path', bridgeDir], { encoding: 'utf8', + timeout: 30_000, }); expect(dump.status).toBe(0); const manifest = JSON.parse(dump.stdout) as { targets: Array<{ name: string }> }; @@ -248,7 +250,7 @@ final class AppState { const firstHash = treeHash(bridgeDir, generatedDir); const firstAccessorHash = accessor.match(/accessorHash: "([a-f0-9]+)"/)?.[1]; - const second = spawnSync('bash', args, { encoding: 'utf8', env }); + const second = spawnSync('bash', args, { encoding: 'utf8', env, timeout: 30_000 }); expect(second.status).toBe(0); expect(second.stderr).toBe(''); expect(second.stdout).toContain('gen-accessors: cache hit'); diff --git a/test/learnings.test.ts b/test/learnings.test.ts index 603fe07f9..e43fe7c04 100644 --- a/test/learnings.test.ts +++ b/test/learnings.test.ts @@ -37,7 +37,7 @@ function runSearch(args: string = ''): string { timeout: 15000, }; try { - return execSync(`${BIN}/gstack-learnings-search ${args}`, execOpts).trim(); + return execSync(`${BIN}/gstack-learnings-search ${args}`, execOpts).trim(); // timeout via execOpts } catch { return ''; } diff --git a/test/memory-cache-injection.test.ts b/test/memory-cache-injection.test.ts index 991a4c019..163eb7286 100644 --- a/test/memory-cache-injection.test.ts +++ b/test/memory-cache-injection.test.ts @@ -53,6 +53,7 @@ function runHook(stdin: object): { stdout: string; stderr: string; status: numbe input: JSON.stringify({ ...stdin, cwd: fixtureCwd }), encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); let parsed: any = null; try { parsed = JSON.parse(res.stdout || '{}'); } catch {} diff --git a/test/memory-ingest-include-gitignored.test.ts b/test/memory-ingest-include-gitignored.test.ts index 51b9d7ae9..901f38a69 100644 --- a/test/memory-ingest-include-gitignored.test.ts +++ b/test/memory-ingest-include-gitignored.test.ts @@ -69,6 +69,7 @@ describe("gstack-memory-ingest: gbrain import must not be filtered by .gitignore try { const git = (args: string[], cwd: string, env?: NodeJS.ProcessEnv) => execFileSync("git", args, { + timeout: 30_000, cwd, encoding: "utf-8", env: { ...process.env, ...env }, @@ -113,7 +114,7 @@ describe("gstack-memory-ingest: gbrain import must not be filtered by .gitignore const dir = mkdtempSync(join(tmpdir(), "gstack-ingest-gitignore-")); try { const git = (...args: string[]) => - execFileSync("git", args, { cwd: dir, encoding: "utf-8" }); + execFileSync("git", args, { timeout: 30_000, cwd: dir, encoding: "utf-8" }); git("init", "-q", "."); const staging = join(dir, ".staging-ingest-12345-1700000000000", "learnings"); diff --git a/test/migrations-v1.27.0.0.test.ts b/test/migrations-v1.27.0.0.test.ts index 0c62a8c22..12c5f24b1 100644 --- a/test/migrations-v1.27.0.0.test.ts +++ b/test/migrations-v1.27.0.0.test.ts @@ -81,6 +81,7 @@ function run(extraEnv: Record = {}, input = ''): { code: number; encoding: 'utf-8', input, cwd: tmpHome, + timeout: 30_000, }); return { code: r.status ?? -1, stdout: r.stdout || '', stderr: r.stderr || '' }; } diff --git a/test/migrations-v1.65.0.0.test.ts b/test/migrations-v1.65.0.0.test.ts index 554120513..c29cb2366 100644 --- a/test/migrations-v1.65.0.0.test.ts +++ b/test/migrations-v1.65.0.0.test.ts @@ -97,6 +97,7 @@ function run(extraEnv: Record = {}): { }, encoding: 'utf-8', cwd: tmpHome, + timeout: 30_000, }); return { code: r.status ?? -1, stdout: r.stdout || '', stderr: r.stderr || '' }; } diff --git a/test/mktemp-portability.test.ts b/test/mktemp-portability.test.ts index e9c2811e3..602a072ea 100644 --- a/test/mktemp-portability.test.ts +++ b/test/mktemp-portability.test.ts @@ -23,6 +23,7 @@ function trackedTmplFiles(): string[] { const out = execFileSync("git", ["ls-files", "*.tmpl", "**/*.tmpl"], { cwd: ROOT, encoding: "utf-8", + timeout: 30_000, }); return out.split("\n").filter(Boolean); } @@ -50,8 +51,9 @@ describe("mktemp portability (#2091)", () => { const tmp = process.env.TMPDIR || "/tmp"; const created = execFileSync("mktemp", [`${tmp.replace(/\/$/, "")}/gstack-portability-XXXXXX`], { encoding: "utf-8", + timeout: 30_000, }).trim(); expect(created.length).toBeGreaterThan(0); - execFileSync("rm", ["-f", created]); + execFileSync("rm", ["-f", created], { timeout: 30_000 }); }); }); diff --git a/test/no-stale-gstack-brain-refs.test.ts b/test/no-stale-gstack-brain-refs.test.ts index 171eb73f7..b2d1da89d 100644 --- a/test/no-stale-gstack-brain-refs.test.ts +++ b/test/no-stale-gstack-brain-refs.test.ts @@ -99,7 +99,7 @@ const SCAN_PATHS = [ function grepRefs(pattern: string): string[] { const args = ['-rn', '--', pattern, ...SCAN_PATHS.map((p) => path.join(ROOT, p))]; - const r = spawnSync('grep', args, { encoding: 'utf-8' }); + const r = spawnSync('grep', args, { encoding: 'utf-8', timeout: 30_000 }); // grep exits 1 when no matches — that's fine for our purposes. const lines = (r.stdout || '').split('\n').filter((l) => l.trim().length > 0); return lines diff --git a/test/paid-orphan-tripwire.test.ts b/test/paid-orphan-tripwire.test.ts index b1f0bb1f6..d34813366 100644 --- a/test/paid-orphan-tripwire.test.ts +++ b/test/paid-orphan-tripwire.test.ts @@ -7,8 +7,9 @@ * execute anywhere — forever, silently. Four files were in that state * (codex-e2e-plan-format, codex-e2e-recommendation-substance, * llm-judge-recommendation, carve-section-loading), and the tripwire built - * for the adjacent class (test/evals-workflow-matrix.test.ts) couldn't see - * them because it filters on isPaidTestFile() FIRST. + * for the adjacent class (the since-retired evals-workflow-matrix test; + * successor: test/evals-workflow-wiring.test.ts) couldn't see them because + * it filtered on isPaidTestFile() FIRST. * * Detection is over source text, so meta-tests and helpers that mention the * gate patterns need reasoned exemptions (same convention as @@ -30,7 +31,7 @@ const SCANNER_EXEMPT = new Map([ // Meta-tests that quote gate-pattern strings to test classification: ['test/helpers/e2e-gate.unit.test.ts', 'free unit test OF the gate predicates (env stubbed)'], ['test/paid-shards.test.ts', 'quotes tier-guard strings as classification fixtures'], - ['test/evals-workflow-matrix.test.ts', 'parses tier guards out of matrix files'], + ['test/evals-workflow-wiring.test.ts', 'pins the sliced-lane yml wiring (successor to the matrix test)'], ['test/e2e-tier-alignment.test.ts', 'parses tier guards to enforce alignment'], ['test/paid-orphan-tripwire.test.ts', 'this scanner'], ]); @@ -46,7 +47,7 @@ const GATE_PATTERNS = [ ]; function trackedTestFiles(): string[] { - const out = spawnSync('git', ['ls-files', '*.test.ts'], { cwd: ROOT, encoding: 'utf-8' }); + const out = spawnSync('git', ['ls-files', '*.test.ts'], { cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); if (out.status !== 0) throw new Error(`git ls-files failed: ${out.stderr}`); return out.stdout.split('\n').filter(Boolean); } diff --git a/test/paid-shards.test.ts b/test/paid-shards.test.ts index 7b6335e58..a73922deb 100644 --- a/test/paid-shards.test.ts +++ b/test/paid-shards.test.ts @@ -22,6 +22,7 @@ import { computePaidDiffSelection, diffSkipDecisionForFile, formatSummary, + isAllSkippedPass, isPaidTestFile, knownTestNamesInSource, partitionShardsByDiffSelection, @@ -159,7 +160,13 @@ describe('shard execution', () => { expect(lines.filter((l) => l.includes(' START ')).length).toBe(3); expect(lines.some((l) => /TIMED-OUT in \d+s/.test(l))).toBe(true); expect(lines.some((l) => /PASSED in \d+s/.test(l))).toBe(true); - }, 30_000); + // 90s, not the default 30s: this test spawns/kills three real children + // (one a busy-loop burning a full core) while 5 sibling shard processes + // compete for 8 vCPUs — observed blowing exactly the 30s ceiling at + // 30009ms under full-suite load while passing in isolation in 1.4s. + // Every assertion above is event-based; the only latency claim is the + // <30s kill-deadline sanity bound, which stays. + }, 90_000); test('exit 0 WITHOUT the terminal summary is FAILED — enforced for injected commands too', async () => { // The invisible-non-execution backstop: previously the paid runner @@ -348,3 +355,60 @@ describe('parent-side diff shard skipping', () => { expect(summaryExitCode(withNeverStarted)).toBe(1); }); }); + +// Green-by-skip census: "Ran N tests" counts skips, so a codex/gemini file +// whose every test self-skipped (binary absent on the runner) exits 0 and +// used to read as coverage in the weekly report. The census label keeps the +// pass (service availability is host state, not a repo regression) but must +// say the shard verified nothing. +describe('all-skipped pass census', () => { + const base = { shard: 1, files: ['test/codex-e2e.test.ts'], exitCode: 0, elapsedMs: 1200, groupPid: 1 }; + + test('isAllSkippedPass: pass with every test skipped → true', () => { + expect(isAllSkippedPass({ ...base, status: 'passed', executedTests: 8, skippedTests: 8 } as ShardOutcome)).toBe(true); + }); + + test('isAllSkippedPass: real work, a failure, or no data → false', () => { + // one test actually ran + expect(isAllSkippedPass({ ...base, status: 'passed', executedTests: 8, skippedTests: 7 } as ShardOutcome)).toBe(false); + // zero tests: that's the hollow-shard guard's territory, not this label's + expect(isAllSkippedPass({ ...base, status: 'passed', executedTests: 0, skippedTests: 0 } as ShardOutcome)).toBe(false); + // non-pass statuses never get the label + expect(isAllSkippedPass({ ...base, status: 'failed', executedTests: 8, skippedTests: 8 } as ShardOutcome)).toBe(false); + // stream gave no counts (crash/timeout) — unknown, not all-skipped + expect(isAllSkippedPass({ ...base, status: 'passed', executedTests: null, skippedTests: null } as ShardOutcome)).toBe(false); + }); + + test('wiring: a real child’s skip recap flows through runPaidShard into skippedTests', async () => { + // End-to-end through the actual spawn/classify path (not hand-built + // outcomes): a fake shard child prints bun’s recap shape with every test + // skipped; the outcome must carry the parsed counts and formatSummary + // must label it. This is the seam the unit tests above skip. + const ALL_SKIP = 'console.log(" 0 pass"); console.log(" 3 skip"); console.log(" 0 fail"); console.log("Ran 3 tests across 1 files. [5ms]")'; + const summary = await runPaidShards([['all-skip']], { + timeoutMs: 30_000, + jobs: 1, + commandFor: () => ({ command: process.execPath, args: ['-e', ALL_SKIP] }), + log: () => {}, + }); + const outcome = summary.outcomes[0]; + expect(outcome.status).toBe('passed'); + expect(outcome.executedTests).toBe(3); + expect(outcome.skippedTests).toBe(3); + expect(isAllSkippedPass(outcome)).toBe(true); + const lines = formatSummary(summary); + expect(lines.find((l) => l.includes('all-skip'))).toContain('all 3 tests SKIPPED'); + }, 30_000); + + test('formatSummary labels an all-skipped pass and leaves real passes alone', () => { + const lines = formatSummary(summarize([ + { ...base, status: 'passed', executedTests: 8, skippedTests: 8 } as ShardOutcome, + { shard: 2, files: ['test/skill-e2e-review.test.ts'], status: 'passed', exitCode: 0, elapsedMs: 900, groupPid: 2, executedTests: 3, skippedTests: 0 } as ShardOutcome, + ])); + const codexLine = lines.find((l) => l.includes('codex-e2e')); + const reviewLine = lines.find((l) => l.includes('skill-e2e-review')); + expect(codexLine).toContain('all 8 tests SKIPPED'); + expect(codexLine).toContain('verified nothing'); + expect(reviewLine).not.toContain('SKIPPED'); + }); +}); diff --git a/test/plan-tune-gates.test.ts b/test/plan-tune-gates.test.ts index faedf1554..93d842ade 100644 --- a/test/plan-tune-gates.test.ts +++ b/test/plan-tune-gates.test.ts @@ -55,7 +55,7 @@ function runBin( } env.GSTACK_STATE_ROOT = stateRoot; delete env.GSTACK_HOME; - const res = spawnSync(bin, args, { env, encoding: 'utf-8', cwd: ROOT }); + const res = spawnSync(bin, args, { env, encoding: 'utf-8', cwd: ROOT, timeout: 30_000 }); return { stdout: res.stdout ?? '', stderr: res.stderr ?? '', diff --git a/test/plan-tune.test.ts b/test/plan-tune.test.ts index 9bb0230aa..4eb9483ba 100644 --- a/test/plan-tune.test.ts +++ b/test/plan-tune.test.ts @@ -568,15 +568,15 @@ describe('end-to-end pipeline (binaries working together)', () => { ts: `2026-04-0${i + 1}T10:00:00Z`, }), ], - { env, cwd: ROOT, encoding: 'utf-8' }, + { env, cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }, ); expect(r.status).toBe(0); } - const derive = spawnSync(devBin, ['--derive'], { env, cwd: ROOT, encoding: 'utf-8' }); + const derive = spawnSync(devBin, ['--derive'], { env, cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); expect(derive.status).toBe(0); - const profileOut = spawnSync(devBin, ['--profile'], { env, cwd: ROOT, encoding: 'utf-8' }); + const profileOut = spawnSync(devBin, ['--profile'], { env, cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); const p = JSON.parse(profileOut.stdout); expect(p.inferred.sample_size).toBe(5); expect(p.inferred.values.scope_appetite).toBeGreaterThan(0.5); @@ -598,13 +598,13 @@ describe('end-to-end pipeline (binaries working together)', () => { '--write', JSON.stringify({ question_id: 'fake-id', preference: 'never-ask', source: 'inline-tool-output' }), ], - { env, cwd: ROOT, encoding: 'utf-8' }, + { env, cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }, ); expect(r.status).toBe(2); expect(r.stderr).toContain('poisoning'); // Verify no preference was written - const read = spawnSync(prefBin, ['--read'], { env, cwd: ROOT, encoding: 'utf-8' }); + const read = spawnSync(prefBin, ['--read'], { env, cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); const prefs = JSON.parse(read.stdout); expect(prefs['fake-id']).toBeUndefined(); } finally { @@ -633,11 +633,11 @@ describe('end-to-end pipeline (binaries working together)', () => { ); // Migrate - const m = spawnSync(devBin, ['--migrate'], { env, cwd: ROOT, encoding: 'utf-8' }); + const m = spawnSync(devBin, ['--migrate'], { env, cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); expect(m.status).toBe(0); // Legacy shim should still return the same KEY: VALUE shape - const shimOut = spawnSync(shimBin, [], { env, cwd: ROOT, encoding: 'utf-8' }); + const shimOut = spawnSync(shimBin, [], { env, cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); expect(shimOut.status).toBe(0); expect(shimOut.stdout).toContain('SESSION_COUNT: 3'); expect(shimOut.stdout).toContain('TIER: welcome_back'); diff --git a/test/pr-title-rewrite.test.ts b/test/pr-title-rewrite.test.ts index cfdef402e..d97f4c921 100644 --- a/test/pr-title-rewrite.test.ts +++ b/test/pr-title-rewrite.test.ts @@ -5,7 +5,7 @@ import * as path from 'path'; const HELPER = path.join(import.meta.dir, '..', 'bin', 'gstack-pr-title-rewrite.sh'); function rewrite(version: string, title: string): { stdout: string; status: number; stderr: string } { - const r = spawnSync(HELPER, [version, title], { encoding: 'utf-8' }); + const r = spawnSync(HELPER, [version, title], { encoding: 'utf-8', timeout: 30_000 }); return { stdout: (r.stdout ?? '').trimEnd(), status: r.status ?? -1, stderr: r.stderr ?? '' }; } @@ -54,7 +54,7 @@ describe('gstack-pr-title-rewrite', () => { }); test('errors on missing args', () => { - const r = spawnSync(HELPER, ['1.2.3.4'], { encoding: 'utf-8' }); + const r = spawnSync(HELPER, ['1.2.3.4'], { encoding: 'utf-8', timeout: 30_000 }); expect(r.status).not.toBe(0); }); diff --git a/test/preamble-first-task-scaffold.test.ts b/test/preamble-first-task-scaffold.test.ts index ac4ffab46..fd641bd73 100644 --- a/test/preamble-first-task-scaffold.test.ts +++ b/test/preamble-first-task-scaffold.test.ts @@ -28,7 +28,7 @@ function detect(cwd: string): string { return execFileSync(DETECT, [], { cwd, encoding: 'utf-8', env: GIT_ENV }).trim(); } function git(cwd: string, args: string) { - execSync(`git ${args}`, { cwd, env: GIT_ENV, stdio: 'ignore' }); + execSync(`git ${args}`, { cwd, env: GIT_ENV, stdio: 'ignore', timeout: 30_000 }); } let tmp: string; diff --git a/test/question-log-hook.test.ts b/test/question-log-hook.test.ts index faa0cfc8b..27b6497b9 100644 --- a/test/question-log-hook.test.ts +++ b/test/question-log-hook.test.ts @@ -48,6 +48,7 @@ function runHook(stdin: object): { stdout: string; stderr: string; status: numbe input: JSON.stringify(stdin), encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); return { stdout: res.stdout ?? '', @@ -345,7 +346,7 @@ describe('PostToolUse hook (crash safety)', () => { } env.GSTACK_STATE_ROOT = stateRoot; env.GSTACK_QUESTION_LOG_NO_DERIVE = '1'; - const res = spawnSync(HOOK, [], { env, input: '', encoding: 'utf-8' }); + const res = spawnSync(HOOK, [], { env, input: '', encoding: 'utf-8', timeout: 30_000 }); expect(res.status).toBe(0); }); @@ -360,6 +361,7 @@ describe('PostToolUse hook (crash safety)', () => { env, input: 'not json', encoding: 'utf-8', + timeout: 30_000, }); expect(res.status).toBe(0); // Error logged to hook-errors.log diff --git a/test/question-preference-hook.test.ts b/test/question-preference-hook.test.ts index 16534da50..000a1aa5d 100644 --- a/test/question-preference-hook.test.ts +++ b/test/question-preference-hook.test.ts @@ -95,6 +95,7 @@ function runHook(stdin: object, cwd?: string, extraEnv?: Record) input: JSON.stringify({ ...stdin, cwd: cwd || fixtureCwd }), encoding: 'utf-8', cwd: ROOT, + timeout: 30_000, }); let parsed: any = null; try { parsed = JSON.parse(res.stdout || '{}'); } catch {} @@ -187,7 +188,7 @@ describe('passes through (no enforcement)', () => { if (v !== undefined) env[k] = v; } env.GSTACK_STATE_ROOT = stateRoot; - const res = spawnSync(HOOK, [], { env, input: '', encoding: 'utf-8' }); + const res = spawnSync(HOOK, [], { env, input: '', encoding: 'utf-8', timeout: 30_000 }); expect(res.status).toBe(0); expect(res.stdout).toBe(''); }); @@ -729,6 +730,7 @@ describe('Conductor spawned deny (#2733)', () => { const scriptKind = spawnSync(BIN, [], { env: { PATH: process.env.PATH ?? '/usr/bin:/bin', ...env }, encoding: 'utf-8', + timeout: 30_000, }).stdout.trim(); expect( spawnedByEnv(env), diff --git a/test/readme-throughput.test.ts b/test/readme-throughput.test.ts index 252dfb836..eba52c157 100644 --- a/test/readme-throughput.test.ts +++ b/test/readme-throughput.test.ts @@ -39,6 +39,7 @@ function runScript(cwd: string): { stdout: string; stderr: string; status: numbe encoding: 'utf-8', cwd, env: { ...process.env }, + timeout: 30_000, }); return { stdout: (res.stdout ?? '').trim(), diff --git a/test/redact-audit-log.test.ts b/test/redact-audit-log.test.ts index ce833954c..115fc07fe 100644 --- a/test/redact-audit-log.test.ts +++ b/test/redact-audit-log.test.ts @@ -91,7 +91,7 @@ describe("CLI", () => { const r = spawnSync( "bun", [LIB, JSON.stringify({ repo_visibility: "public", outcome: "flagged", categories_flagged: ["pii"] }), bodyFile], - { env: { ...process.env, GSTACK_HOME: home }, encoding: "utf8" }, + { env: { ...process.env, GSTACK_HOME: home }, encoding: "utf8", timeout: 30_000 }, ); expect(r.status).toBe(0); const line = JSON.parse(fs.readFileSync(logPath(), "utf8").trim()); diff --git a/test/redact-prepush-hook.test.ts b/test/redact-prepush-hook.test.ts index 16f0f6d4b..3c583a3a9 100644 --- a/test/redact-prepush-hook.test.ts +++ b/test/redact-prepush-hook.test.ts @@ -20,7 +20,7 @@ const REDACT = path.resolve(import.meta.dir, "..", "bin", "gstack-redact"); let repo: string; function git(args: string[], cwd = repo): string { - const r = spawnSync("git", args, { cwd, encoding: "utf8" }); + const r = spawnSync("git", args, { cwd, encoding: "utf8", timeout: 30_000 }); return r.stdout?.trim() ?? ""; } @@ -40,6 +40,7 @@ function runHook( input: Buffer.from(stdinLines), encoding: "utf8", env: { ...process.env, ...env }, + timeout: 30_000, }); return { code: r.status ?? 0, stderr: r.stderr ?? "" }; } @@ -322,7 +323,7 @@ describe("install / chaining", () => { const existing = path.join(hookDir, "pre-push"); fs.writeFileSync(existing, "#!/usr/bin/env bash\necho mine\n", { mode: 0o755 }); - const r = spawnSync("bun", [REDACT, "install-prepush-hook"], { cwd: repo, encoding: "utf8" }); + const r = spawnSync("bun", [REDACT, "install-prepush-hook"], { cwd: repo, encoding: "utf8", timeout: 30_000 }); expect(r.status).toBe(0); const installed = fs.readFileSync(existing, "utf8"); expect(installed).toContain("gstack-redact pre-push (managed)"); @@ -336,7 +337,7 @@ describe("install / chaining", () => { test("chained pre-push.local receives the final ref line (trailing newline preserved)", () => { const hookDir = path.join(repo, ".git", "hooks"); fs.mkdirSync(hookDir, { recursive: true }); - spawnSync("bun", [REDACT, "install-prepush-hook"], { cwd: repo }); + spawnSync("bun", [REDACT, "install-prepush-hook"], { cwd: repo, timeout: 30_000 }); const seen = path.join(repo, "seen.txt"); fs.writeFileSync( @@ -352,6 +353,7 @@ describe("install / chaining", () => { input: Buffer.from(line), encoding: "utf8", env: { ...process.env, GSTACK_REDACT_PREPUSH: "skip" }, + timeout: 30_000, }); expect(r.status).toBe(0); expect(fs.existsSync(seen)).toBe(true); @@ -363,7 +365,7 @@ describe("install / chaining", () => { test("a blocking pre-push.local still short-circuits the push", () => { const hookDir = path.join(repo, ".git", "hooks"); fs.mkdirSync(hookDir, { recursive: true }); - spawnSync("bun", [REDACT, "install-prepush-hook"], { cwd: repo }); + spawnSync("bun", [REDACT, "install-prepush-hook"], { cwd: repo, timeout: 30_000 }); fs.writeFileSync( path.join(hookDir, "pre-push.local"), "#!/usr/bin/env bash\nwhile read -r _a _b _c _d || [ -n \"${_a:-}\" ]; do exit 1; done\nexit 0\n", @@ -374,6 +376,7 @@ describe("install / chaining", () => { input: Buffer.from(`refs/heads/main ${"b".repeat(40)} refs/heads/main ${ZERO}\n`), encoding: "utf8", env: { ...process.env, GSTACK_REDACT_PREPUSH: "skip" }, + timeout: 30_000, }); expect(r.status).toBe(1); }); @@ -384,8 +387,8 @@ describe("install / chaining", () => { fs.writeFileSync(path.join(hookDir, "pre-push"), "#!/usr/bin/env bash\necho mine\n", { mode: 0o755, }); - spawnSync("bun", [REDACT, "install-prepush-hook"], { cwd: repo }); - spawnSync("bun", [REDACT, "uninstall-prepush-hook"], { cwd: repo }); + spawnSync("bun", [REDACT, "install-prepush-hook"], { cwd: repo, timeout: 30_000 }); + spawnSync("bun", [REDACT, "uninstall-prepush-hook"], { cwd: repo, timeout: 30_000 }); const restored = fs.readFileSync(path.join(hookDir, "pre-push"), "utf8"); expect(restored).toContain("echo mine"); expect(restored).not.toContain("managed"); @@ -404,7 +407,7 @@ describe("base resolution when the default branch is neither main nor master", ( // blocks the push having scanned NOTHING — the "scans more, never less" // fallback inverting into "scans nothing". const bare = fs.mkdtempSync(path.join(os.tmpdir(), "prepush-remote-")); - spawnSync("git", ["init", "-q", "--bare", "-b", "trunk", bare]); + spawnSync("git", ["init", "-q", "--bare", "-b", "trunk", bare], { timeout: 30_000 }); git(["branch", "-M", "trunk"]); const old = commit("legacy.txt", FAKE_AWS_KEY + "\n", "secret already on the remote"); diff --git a/test/redact-prepush-rebase-force-push.test.ts b/test/redact-prepush-rebase-force-push.test.ts index d1852b641..b5979ae3f 100644 --- a/test/redact-prepush-rebase-force-push.test.ts +++ b/test/redact-prepush-rebase-force-push.test.ts @@ -38,7 +38,7 @@ let repo: string; let remote: string; function git(args: string[], cwd = repo): string { - const r = spawnSync("git", args, { cwd, encoding: "utf8" }); + const r = spawnSync("git", args, { cwd, encoding: "utf8", timeout: 30_000 }); if (r.status !== 0) throw new Error(`git ${args.join(" ")}\n${r.stderr}`); return r.stdout?.trim() ?? ""; } @@ -56,6 +56,7 @@ function runHook(stdinLines: string): { code: number; stderr: string } { cwd: repo, input: Buffer.from(stdinLines), encoding: "utf8", + timeout: 30_000, env: { ...process.env }, }); return { code: r.status ?? 0, stderr: r.stderr ?? "" }; @@ -115,7 +116,7 @@ describe("rebased force-push does not re-scan upstream commits (#2573)", () => { // The rebased tip exists locally and is NOT an ancestor of HEAD — the // exact condition #2573 identified as the untested third branch. expect(git(["cat-file", "-t", preRebaseTip])).toBe("commit"); - const isAncestor = spawnSync("git", ["merge-base", "--is-ancestor", preRebaseTip, "HEAD"], { cwd: repo }); + const isAncestor = spawnSync("git", ["merge-base", "--is-ancestor", preRebaseTip, "HEAD"], { cwd: repo, timeout: 30_000 }); expect(isAncestor.status).not.toBe(0); // What the OLD range would scan: upstream's published fixture included. const oldDiff = git(["diff", "--unified=0", `${preRebaseTip}..HEAD`]); diff --git a/test/redact-prepush-scan-range.test.ts b/test/redact-prepush-scan-range.test.ts index f44df28c4..2f2d31ec2 100644 --- a/test/redact-prepush-scan-range.test.ts +++ b/test/redact-prepush-scan-range.test.ts @@ -23,7 +23,7 @@ import { dirname, join } from "path"; let dir: string; const run = (args: string[], cwd = dir): string => { - const r = spawnSync("git", args, { cwd, encoding: "utf8" }); + const r = spawnSync("git", args, { cwd, encoding: "utf8", timeout: 30_000 }); if (r.status !== 0) throw new Error(`git ${args.join(" ")}\n${r.stderr}`); return r.stdout ?? ""; }; @@ -146,7 +146,7 @@ describe("narrowing the range does not narrow coverage", () => { run(["push", "-q", "origin", "main"]); run(["fetch", "-q", "origin"]); run(["checkout", "-q", "feature"]); - spawnSync("git", ["merge", "--no-edit", "main"], { cwd: dir, encoding: "utf8" }); // conflicts + spawnSync("git", ["merge", "--no-edit", "main"], { cwd: dir, encoding: "utf8", timeout: 30_000 }); // conflicts writeFileSync(join(dir, "conflict.txt"), `resolved ${FAKE_AWS_RESOLV}\n`); run(["add", "conflict.txt"]); run(["commit", "-q", "--no-edit"]); @@ -178,6 +178,7 @@ describe("S1: exclusion scoped to the push-target remote", () => { input: Buffer.from(stdinLines), encoding: "utf8", env: { ...process.env }, + timeout: 30_000, }); return { code: r.status ?? 0, stderr: r.stderr ?? "" }; } diff --git a/test/regression-issue2091-bsd-mktemp.test.ts b/test/regression-issue2091-bsd-mktemp.test.ts index 0bb65a638..d25e2e93f 100644 --- a/test/regression-issue2091-bsd-mktemp.test.ts +++ b/test/regression-issue2091-bsd-mktemp.test.ts @@ -138,6 +138,7 @@ function tmpRoot(env: Record): string { const result = spawnSync('bash', [PATHS_BIN], { env: { PATH: process.env.PATH, USERPROFILE: '', ...env } as Record, encoding: 'utf-8', + timeout: 30_000, }); if (result.status !== 0) { throw new Error(`gstack-paths failed (status ${result.status}): ${result.stderr}`); diff --git a/test/regression-pr1169-build-app-sed.test.ts b/test/regression-pr1169-build-app-sed.test.ts index 8d2596112..e03e5e65e 100644 --- a/test/regression-pr1169-build-app-sed.test.ts +++ b/test/regression-pr1169-build-app-sed.test.ts @@ -55,7 +55,7 @@ describe("PR #1169 bug #2: build-app.sh sed escape for $APP_NAME", () => { "_", appName, ], - { encoding: "utf-8" } + { encoding: "utf-8", timeout: 30_000 } ); expect(result.status).toBe(0); @@ -149,6 +149,7 @@ describe("PR #1169 bug #3: build-app.sh DMG_TMP mktemp failure guard", () => { { encoding: "utf-8", env: { ...process.env, PATH: `${fakeBin}:${process.env.PATH}` }, + timeout: 30_000, } ); diff --git a/test/resolvers-gbrain-put-rewrite.test.ts b/test/resolvers-gbrain-put-rewrite.test.ts index 75a0d2225..cdfb1d2de 100644 --- a/test/resolvers-gbrain-put-rewrite.test.ts +++ b/test/resolvers-gbrain-put-rewrite.test.ts @@ -31,6 +31,7 @@ function listTrackedSkillMd(): string[] { const out = execFileSync("git", ["ls-files", "*SKILL.md"], { cwd: REPO_ROOT, encoding: "utf-8", + timeout: 30_000, }); return out.split("\n").filter((line) => line.trim().length > 0); } diff --git a/test/review-log.test.ts b/test/review-log.test.ts index a3fc099a8..32c7aa510 100644 --- a/test/review-log.test.ts +++ b/test/review-log.test.ts @@ -19,7 +19,7 @@ function run(input: string, opts: { expectFail?: boolean } = {}): { stdout: stri timeout: 10000, }; try { - const stdout = execSync(`${BIN}/gstack-review-log '${input.replace(/'/g, "'\\''")}'`, execOpts).trim(); + const stdout = execSync(`${BIN}/gstack-review-log '${input.replace(/'/g, "'\\''")}'`, execOpts).trim(); // timeout via execOpts return { stdout, exitCode: 0 }; } catch (e: any) { if (opts.expectFail) { @@ -119,7 +119,7 @@ describe('gstack-review-log', () => { encoding: 'utf-8', timeout: 10000, }; - execSync(`${BIN}/gstack-review-log '{"skill":"review","status":"clean"}'`, execOpts); + execSync(`${BIN}/gstack-review-log '{"skill":"review","status":"clean"}'`, execOpts); // timeout via execOpts // A record landed somewhere under projects/ without a wtree stamp. const found: string[] = []; const walk = (d: string) => { diff --git a/test/routing-probe.test.ts b/test/routing-probe.test.ts index 3d9fb463c..6fb15f131 100644 --- a/test/routing-probe.test.ts +++ b/test/routing-probe.test.ts @@ -78,7 +78,7 @@ describe('routing probe checks AGENTS.md too (#2500)', () => { ); const out = execSync( `bash -c '${probe.replace(/'/g, `'\\''`)}\necho "HAS_ROUTING: $_HAS_ROUTING"'`, - { cwd: dir, encoding: 'utf-8' }, + { cwd: dir, encoding: 'utf-8', timeout: 30_000 }, ); expect(out).toContain('HAS_ROUTING: yes'); } finally { @@ -92,7 +92,7 @@ describe('routing probe checks AGENTS.md too (#2500)', () => { try { const out = execSync( `bash -c '${probe.replace(/'/g, `'\\''`)}\necho "HAS_ROUTING: $_HAS_ROUTING"'`, - { cwd: dir, encoding: 'utf-8' }, + { cwd: dir, encoding: 'utf-8', timeout: 30_000 }, ); expect(out).toContain('HAS_ROUTING: no'); } finally { diff --git a/test/sandbox-doctor-shell.test.ts b/test/sandbox-doctor-shell.test.ts index 54685f43a..5c084afce 100644 --- a/test/sandbox-doctor-shell.test.ts +++ b/test/sandbox-doctor-shell.test.ts @@ -17,7 +17,7 @@ const SCRIPT = path.join(import.meta.dir, '..', 'scripts', 'sandbox-doctor.sh'); describe('sandbox-doctor.sh', () => { test('parses as POSIX sh, fails fast, and guards every mutation for idempotency', () => { // Syntax: `sh -n` parses without executing. - const parse = spawnSync('sh', ['-n', SCRIPT], { encoding: 'utf8' }); + const parse = spawnSync('sh', ['-n', SCRIPT], { encoding: 'utf8', timeout: 30_000 }); expect(parse.status, parse.stderr).toBe(0); const src = fs.readFileSync(SCRIPT, 'utf-8'); diff --git a/test/session-runner-groupkill.test.ts b/test/session-runner-groupkill.test.ts new file mode 100644 index 000000000..17a33d6c3 --- /dev/null +++ b/test/session-runner-groupkill.test.ts @@ -0,0 +1,110 @@ +/** + * Group-kill regression pin for the provider session runners (F7 in the + * test-infra audit): a timed-out `claude -p` used to get a bare proc.kill() + * — the direct child died but tool subprocesses it had spawned survived as + * orphans holding our pipes open and burning shared API rate (observed: a + * 600s timeout stretching past 1400s; a stalled legacy run once burned a + * core for 15 hours). The fix: node:child_process spawn with `detached` + * (child leads its own process group) + killProcessGroup(SIGKILL) in the + * timeout handler, mirroring runShardChild's proven pattern. + * + * The behavioral test drives the REAL runSkillTest against a fake `claude` + * shim (PATH override — hermeticChildEnv allowlists PATH through) that + * spawns a grandchild and never exits: the run must classify as timeout + * within its budget AND leave neither shim nor grandchild alive. + * + * Windows note: the shim is a '/bin/bash' shebang script, which the free + * runner's Windows curation auto-excludes (CreateProcess cannot exec + * shebangs) — this literal mention is what trips the content scan. + */ +import { describe, test, expect } from 'bun:test'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { spawnSync } from 'child_process'; +import { runSkillTest } from './helpers/session-runner'; + +const ROOT = path.resolve(import.meta.dir, '..'); + +function aliveWithArg(marker: string): boolean { + const result = spawnSync('pgrep', ['-f', marker], { stdio: 'pipe', timeout: 5_000 }); + return result.status === 0; +} + +describe('session-runner timeout kills the whole process group', () => { + test('fake claude + its grandchild are both dead after a timeout', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'groupkill-')); + const shimDir = path.join(dir, 'bin'); + fs.mkdirSync(shimDir); + // Markers are unique PER RUN (fractional seconds carry this process's + // pid): sibling Conductor worktrees run free suites concurrently with no + // machine lock, and fixed markers let one run pgrep/pkill the OTHER + // run's shims (review finding — a cross-run flake inside the anti-flake + // tests). GNU sleep accepts decimals, argv stays greppable. + const mark = (n: number) => `${n}.${process.pid}`; + const shim = [ + '#!/bin/bash', + `sleep ${mark(6041)} &`, // the orphan-candidate grandchild + `exec sleep ${mark(6042)}`, // the shim itself, wedged forever, no NDJSON + ].join('\n'); + fs.writeFileSync(path.join(shimDir, 'claude'), `${shim}\n`, { mode: 0o755 }); + + const realPath = process.env.PATH; + process.env.PATH = `${shimDir}:${realPath}`; + try { + const started = Date.now(); + const result = await runSkillTest({ + prompt: 'irrelevant — the shim never reads it', + workingDirectory: dir, + maxTurns: 1, + allowedTools: ['Bash'], + timeout: 3_000, + testName: 'groupkill-probe', + }); + const wall = Date.now() - started; + + // The shim never prints NDJSON, so the two-phase timer kills it in the + // STARTUP phase (grace = min(default, timeout) = 3s here) — the + // distinct reason is the point: no byte ever arrived. + expect(result.exitReason).toBe('timeout_startup'); + // The old bug's signature was the drain blocking long past the budget + // (600s -> 1400s). Generous 10x bound: timeout 3s + the 5s stderr + // grace race must return promptly once the group is dead. + expect(wall).toBeLessThan(30_000); + + // The kill is SIGKILL on the GROUP: give the OS a beat to reap, then + // require both the wedged shim and its grandchild gone. + await new Promise((r) => setTimeout(r, 1_000)); + expect(aliveWithArg(`sleep ${mark(6042)}`), 'the fake claude itself survived the timeout kill').toBe(false); + expect(aliveWithArg(`sleep ${mark(6041)}`), 'the grandchild ORPHANED — group kill regressed to a direct-child kill').toBe(false); + } finally { + process.env.PATH = realPath; + // Belt and braces: never leak the markers into later tests even on + // assertion failure. + spawnSync('pkill', ['-f', `sleep 604[12]\\.${process.pid}`], { stdio: 'ignore', timeout: 5_000 }); + fs.rmSync(dir, { recursive: true, force: true }); + } + }, 60_000); +}); + +describe('all three provider runners carry the group-kill wiring', () => { + // Source pin, not behavior: codex/gemini need their real binaries for a + // behavioral run, but the kill wiring is identical code — a runner that + // drops `detached` or reverts to a bare kill() re-opens the orphan class. + const runners = [ + 'test/helpers/session-runner.ts', + 'test/helpers/codex-session-runner.ts', + 'test/helpers/gemini-session-runner.ts', + ]; + for (const rel of runners) { + test(`${path.basename(rel)}: detached spawn + killProcessGroup, no bare timeout kill`, () => { + const source = fs.readFileSync(path.join(ROOT, rel), 'utf-8'); + expect(source).toContain("detached: process.platform !== 'win32'"); + expect(source).toContain('killProcessGroup(proc'); + expect(source, `${rel} reverted to Bun.spawn for the provider child — detached group-kill is impossible there`) + .not.toMatch(/Bun\.spawn\(\[['"](?:claude|codex|gemini)['"]/); + expect(source, `${rel} has a bare proc.kill() in a timeout handler`) + .not.toMatch(/timedOut = true;\s*\n\s*proc\.kill\(\)/); + }); + } +}); diff --git a/test/session-runner-startup-grace.test.ts b/test/session-runner-startup-grace.test.ts new file mode 100644 index 000000000..9b06cf2c8 --- /dev/null +++ b/test/session-runner-startup-grace.test.ts @@ -0,0 +1,156 @@ +/** + * Two-phase timeout pins for the claude session runner (WS4c). + * + * The old single timer charged API queue latency to the work budget — the + * recurring '0 turns / $0.00 / x3 attempts' failure with four budget-bump + * receipts (180→300s, 240→360s, 300→420s, 90→300s). The split: + * startup phase — no NDJSON byte yet; killed at the grace with the + * DISTINCT reason 'timeout_startup' (availability, not behavior); + * work phase — armed on the first byte for the REMAINING budget, so the + * total wall never exceeds `timeout` (tier envelopes are margin-free: + * tests pass `timeout: CAPTURE_MS` and use the same tier as bun budget). + * + * Also pins the TODOS-filed 300s CI startup floor: shared CI runners queue + * harder, and a floor below 300s converts ordinary queueing into false reds. + */ +import { describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { + runSkillTest, + STARTUP_GRACE_CI_FLOOR_MS, + STARTUP_GRACE_MS, +} from './helpers/session-runner'; + +describe('session-runner two-phase timeout', () => { + test('CI startup-grace floor is 300s and the local default is sane', () => { + expect(STARTUP_GRACE_CI_FLOOR_MS).toBe(300_000); + expect(STARTUP_GRACE_MS).toBeGreaterThanOrEqual(60_000); + expect(STARTUP_GRACE_MS).toBeLessThanOrEqual(STARTUP_GRACE_CI_FLOOR_MS); + }); + + test('a run whose first byte arrives late still gets its work budget honored within the total', async () => { + // Fake claude: silent for 2s (startup latency), then streams NDJSON and + // wedges. startupGraceMs=4s tolerates the latency; work budget then + // kills at ~timeout. exitReason must be plain 'timeout' (work phase), + // NOT 'timeout_startup', and the wall must respect the total envelope. + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'grace-')); + const shimDir = path.join(dir, 'bin'); + fs.mkdirSync(shimDir); + fs.writeFileSync(path.join(shimDir, 'claude'), [ + '#!/bin/bash', + 'sleep 2', + 'echo \'{"type":"system","subtype":"init"}\'', + `exec sleep 6071.${process.pid}`, + ].join('\n') + '\n', { mode: 0o755 }); + + const realPath = process.env.PATH; + const realCI = process.env.CI; + process.env.PATH = `${shimDir}:${realPath}`; + // This probe pins LOCAL grace semantics (caller honored verbatim). In CI + // the runner clamps explicit graces up to the 300s floor by design, so + // the small shim grace would never take effect — clear CI for the call + // and pin the floor itself in its own probe below. + delete process.env.CI; + try { + const started = Date.now(); + const result = await runSkillTest({ + prompt: 'ignored', + workingDirectory: dir, + maxTurns: 1, + timeout: 5_000, + startupGraceMs: 4_000, + testName: 'grace-probe-work-phase', + }); + const wall = Date.now() - started; + expect(result.exitReason).toBe('timeout'); + expect(result.firstResponseMs).toBeGreaterThanOrEqual(1_500); + // Total envelope: startup consumed ~2s, work phase gets the remainder — + // wall ≈ timeout (5s) + stderr grace (5s), never grace+timeout stacked. + expect(wall).toBeLessThan(20_000); + } finally { + process.env.PATH = realPath; + if (realCI !== undefined) process.env.CI = realCI; + Bun.spawnSync(['pkill', '-f', `sleep 6071\\.${process.pid}`], { timeout: 5_000 }); + fs.rmSync(dir, { recursive: true, force: true }); + } + }, 60_000); + + test('a silent API is killed at the grace, early, with the startup reason', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'grace-')); + const shimDir = path.join(dir, 'bin'); + fs.mkdirSync(shimDir); + fs.writeFileSync(path.join(shimDir, 'claude'), `#!/bin/bash\nexec sleep 6072.${process.pid}\n`, { mode: 0o755 }); + + const realPath = process.env.PATH; + const realCI = process.env.CI; + process.env.PATH = `${shimDir}:${realPath}`; + // LOCAL semantics again: in CI the floor clamps this 2s grace to 300s + // (capped by timeout → 30s), which is exactly the false red this test + // shipped with. The floor's own behavior is pinned in the next probe. + delete process.env.CI; + try { + const started = Date.now(); + const result = await runSkillTest({ + prompt: 'ignored', + workingDirectory: dir, + maxTurns: 1, + timeout: 30_000, // generous work budget… + startupGraceMs: 2_000, // …but startup dies fast when nothing answers + testName: 'grace-probe-startup', + }); + const wall = Date.now() - started; + expect(result.exitReason).toBe('timeout_startup'); + // The whole point: ~2s + drain grace, NOT the 30s work budget. + expect(wall).toBeLessThan(15_000); + expect(result.costEstimate.turnsUsed).toBe(0); + } finally { + process.env.PATH = realPath; + if (realCI !== undefined) process.env.CI = realCI; + Bun.spawnSync(['pkill', '-f', `sleep 6072\\.${process.pid}`], { timeout: 5_000 }); + fs.rmSync(dir, { recursive: true, force: true }); + } + }, 60_000); + + test('in CI the floor clamps an explicit low grace (adversarial pin — the clamp is real)', async () => { + // The floor exists because CI queueing converts a low grace into false + // reds; an explicit startupGraceMs must NOT bypass it (review finding: + // "the name promised a clamp the code lacked"). With CI set, a 2s grace + // request against a 6s timeout floors to min(300s, timeout) = 6s — the + // silent shim survives PAST the requested 2s and dies at the cap, still + // in the startup phase. + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'grace-')); + const shimDir = path.join(dir, 'bin'); + fs.mkdirSync(shimDir); + fs.writeFileSync(path.join(shimDir, 'claude'), `#!/bin/bash\nexec sleep 6073.${process.pid}\n`, { mode: 0o755 }); + + const realPath = process.env.PATH; + const realCI = process.env.CI; + process.env.PATH = `${shimDir}:${realPath}`; + process.env.CI = '1'; + try { + const started = Date.now(); + const result = await runSkillTest({ + prompt: 'ignored', + workingDirectory: dir, + maxTurns: 1, + timeout: 6_000, + startupGraceMs: 2_000, // must be clamped up, not honored + testName: 'grace-probe-ci-floor', + }); + const wall = Date.now() - started; + expect(result.exitReason).toBe('timeout_startup'); + // Proof the clamp fired: the kill lands at the 6s timeout cap, not the + // requested 2s (drain grace can only extend, never shorten). + expect(wall).toBeGreaterThanOrEqual(5_500); + expect(wall).toBeLessThan(20_000); + } finally { + process.env.PATH = realPath; + if (realCI !== undefined) process.env.CI = realCI; + else delete process.env.CI; + Bun.spawnSync(['pkill', '-f', `sleep 6073\\.${process.pid}`], { timeout: 5_000 }); + fs.rmSync(dir, { recursive: true, force: true }); + } + }, 60_000); +}); diff --git a/test/session-update-autostash.test.ts b/test/session-update-autostash.test.ts index dc7f89b54..b61b659a2 100644 --- a/test/session-update-autostash.test.ts +++ b/test/session-update-autostash.test.ts @@ -15,7 +15,7 @@ const ROOT = path.resolve(import.meta.dir, '..'); const SCRIPT = path.join(ROOT, 'bin', 'gstack-session-update'); function git(cwd: string, ...args: string[]): string { - return execFileSync('git', args, { cwd, encoding: 'utf8' }).trim(); + return execFileSync('git', args, { cwd, encoding: 'utf8', timeout: 30_000 }).trim(); } function makeFixture() { @@ -205,7 +205,7 @@ describe('gstack-session-update lock identity + TTL (#2613)', () => { test('a dead pid is reclaimed and the run proceeds', async () => { const { base, install, state } = makeFixture(); try { - const dead = spawnSync('true', { encoding: 'utf8' }); // reaped by the time spawnSync returns + const dead = spawnSync('true', { encoding: 'utf8', timeout: 30_000 }); // reaped by the time spawnSync returns const lockDir = path.join(state, '.setup-lock'); fs.mkdirSync(lockDir, { recursive: true }); fs.writeFileSync(path.join(lockDir, 'pid'), String(dead.pid)); diff --git a/test/setup-bun-cmd-and-pipe-bugs.test.ts b/test/setup-bun-cmd-and-pipe-bugs.test.ts index 098597d55..21a73f69b 100644 --- a/test/setup-bun-cmd-and-pipe-bugs.test.ts +++ b/test/setup-bun-cmd-and-pipe-bugs.test.ts @@ -9,7 +9,7 @@ const SETUP_SRC = fs.readFileSync(path.join(ROOT, 'setup'), 'utf-8'); // Run a bash snippet, return {stdout, stderr, status}. function runBash(script: string): { stdout: string; stderr: string; status: number } { - const r = spawnSync('bash', ['-c', script], { encoding: 'utf-8' }); + const r = spawnSync('bash', ['-c', script], { encoding: 'utf-8', timeout: 30_000 }); return { stdout: r.stdout || '', stderr: r.stderr || '', status: r.status ?? -1 }; } diff --git a/test/setup-conductor-worktree.test.ts b/test/setup-conductor-worktree.test.ts index 29609ac8f..2ff48eefa 100644 --- a/test/setup-conductor-worktree.test.ts +++ b/test/setup-conductor-worktree.test.ts @@ -43,7 +43,7 @@ describe('setup: Conductor worktree guard', () => { fs.mkdirSync(source); fs.mkdirSync(dest); // The buggy invocation: target dest is an existing real dir. - const result = spawnSync('ln', ['-snf', source, dest], { encoding: 'utf-8' }); + const result = spawnSync('ln', ['-snf', source, dest], { encoding: 'utf-8', timeout: 30_000 }); expect(result.status).toBe(0); // Child symlink leaked inside dest. const leaked = path.join(dest, path.basename(source)); @@ -85,7 +85,7 @@ describe('setup: Conductor worktree guard', () => { echo "LINKED" fi `; - const result = spawnSync('bash', ['-c', script], { encoding: 'utf-8' }); + const result = spawnSync('bash', ['-c', script], { encoding: 'utf-8', timeout: 30_000 }); expect(result.status).toBe(0); expect(result.stdout.trim()).toBe('SKIP'); // No child symlink leaked. @@ -120,7 +120,7 @@ describe('setup: Conductor worktree guard', () => { echo "LINKED" fi `; - const result = spawnSync('bash', ['-c', script], { encoding: 'utf-8' }); + const result = spawnSync('bash', ['-c', script], { encoding: 'utf-8', timeout: 30_000 }); expect(result.status).toBe(0); expect(result.stdout.trim()).toBe('LINKED'); expect(fs.lstatSync(dest).isSymbolicLink()).toBe(true); @@ -159,7 +159,7 @@ describe('setup: Conductor worktree guard', () => { echo "LINKED" fi `; - const result = spawnSync('bash', ['-c', script], { encoding: 'utf-8' }); + const result = spawnSync('bash', ['-c', script], { encoding: 'utf-8', timeout: 30_000 }); expect(result.status).toBe(0); expect(result.stdout.trim()).toBe('LINKED'); expect(fs.readlinkSync(dest)).toBe(source); @@ -191,7 +191,7 @@ describe('setup: Conductor worktree guard', () => { fi echo "skip=$_SKIP_CLAUDE_REGISTER" `; - const result = spawnSync('bash', ['-c', script], { encoding: 'utf-8' }); + const result = spawnSync('bash', ['-c', script], { encoding: 'utf-8', timeout: 30_000 }); expect(result.status).toBe(0); expect(result.stdout.trim()).toBe('skip=0'); } finally { diff --git a/test/setup-plan-tune-hooks-noninteractive.test.ts b/test/setup-plan-tune-hooks-noninteractive.test.ts index cb4871a41..a9da5d26a 100644 --- a/test/setup-plan-tune-hooks-noninteractive.test.ts +++ b/test/setup-plan-tune-hooks-noninteractive.test.ts @@ -119,7 +119,7 @@ describe('gstack-config: has subcommand (key-presence provenance)', () => { function has(key: string): number { try { - execSync(`${GSTACK_CONFIG} has '${key}'`, { encoding: 'utf-8', env: env2 }); + execSync(`${GSTACK_CONFIG} has '${key}'`, { encoding: 'utf-8', env: env2, timeout: 30_000 }); return 0; } catch (e: any) { return e.status ?? 1; @@ -128,12 +128,12 @@ describe('gstack-config: has subcommand (key-presence provenance)', () => { test('absent key exits nonzero even though get returns the default', () => { expect(has('plan_tune_hooks')).not.toBe(0); - const got = execSync(`${GSTACK_CONFIG} get plan_tune_hooks`, { encoding: 'utf-8', env: env2 }).trim(); + const got = execSync(`${GSTACK_CONFIG} get plan_tune_hooks`, { encoding: 'utf-8', env: env2, timeout: 30_000 }).trim(); expect(got).toBe('prompt'); // default — indistinguishable from a saved value via get }); test('present key exits 0 through the same STATE_DIR resolution as get', () => { - execSync(`${GSTACK_CONFIG} set plan_tune_hooks no`, { encoding: 'utf-8', env: env2 }); + execSync(`${GSTACK_CONFIG} set plan_tune_hooks no`, { encoding: 'utf-8', env: env2, timeout: 30_000 }); expect(has('plan_tune_hooks')).toBe(0); // GSTACK_STATE_ROOT was the writer — a hardcoded ~/.gstack grep would miss it. }); @@ -164,29 +164,30 @@ describe('gstack-config: plan_tune_hooks key', () => { const out = execSync(`${GSTACK_CONFIG} get plan_tune_hooks`, { encoding: 'utf-8', env, + timeout: 30_000, }).trim(); expect(out).toBe('prompt'); }); test('appears in defaults and list output', () => { - const defaults = execSync(`${GSTACK_CONFIG} defaults`, { encoding: 'utf-8', env }); + const defaults = execSync(`${GSTACK_CONFIG} defaults`, { encoding: 'utf-8', env, timeout: 30_000 }); expect(defaults).toContain('plan_tune_hooks'); - const list = execSync(`${GSTACK_CONFIG} list`, { encoding: 'utf-8', env }); + const list = execSync(`${GSTACK_CONFIG} list`, { encoding: 'utf-8', env, timeout: 30_000 }); expect(list).toContain('plan_tune_hooks'); }); test('accepts valid values (round-trips yes/no/prompt)', () => { for (const v of ['yes', 'no', 'prompt']) { - execSync(`${GSTACK_CONFIG} set plan_tune_hooks ${v}`, { encoding: 'utf-8', env }); - const got = execSync(`${GSTACK_CONFIG} get plan_tune_hooks`, { encoding: 'utf-8', env }).trim(); + execSync(`${GSTACK_CONFIG} set plan_tune_hooks ${v}`, { encoding: 'utf-8', env, timeout: 30_000 }); + const got = execSync(`${GSTACK_CONFIG} get plan_tune_hooks`, { encoding: 'utf-8', env, timeout: 30_000 }).trim(); expect(got).toBe(v); } }); test('rejects out-of-domain values (warns + falls back to prompt)', () => { - const res = execSync(`${GSTACK_CONFIG} set plan_tune_hooks maybe 2>&1`, { encoding: 'utf-8', env }); + const res = execSync(`${GSTACK_CONFIG} set plan_tune_hooks maybe 2>&1`, { encoding: 'utf-8', env, timeout: 30_000 }); expect(res.toLowerCase()).toContain('not recognized'); - const got = execSync(`${GSTACK_CONFIG} get plan_tune_hooks`, { encoding: 'utf-8', env }).trim(); + const got = execSync(`${GSTACK_CONFIG} get plan_tune_hooks`, { encoding: 'utf-8', env, timeout: 30_000 }).trim(); expect(got).toBe('prompt'); }); }); diff --git a/test/ship-test-detection-markers.test.ts b/test/ship-test-detection-markers.test.ts index fb0c80f88..e842642a4 100644 --- a/test/ship-test-detection-markers.test.ts +++ b/test/ship-test-detection-markers.test.ts @@ -44,13 +44,13 @@ function detect(files: Record): string { } const script = path.join(dir, '.detect.sh'); fs.writeFileSync(script, detectionScript()); - const git = (...args: string[]) => execFileSync('git', args, { cwd: dir }); + const git = (...args: string[]) => execFileSync('git', args, { timeout: 30_000, cwd: dir }); git('init', '-q', '.'); git('add', '-A'); git('-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qm', 'fixture'); // The block's last line is a `[ -f marker ] && echo`, so a clean project // exits 1 by design — read stdout, don't trust the status. - return execFileSync('bash', [script], { cwd: dir, encoding: 'utf-8' }); + return execFileSync('bash', [script], { timeout: 30_000, cwd: dir, encoding: 'utf-8' }); } catch (err: unknown) { const e = err as { stdout?: string }; if (typeof e.stdout === 'string') return e.stdout; diff --git a/test/ship-version-sync.test.ts b/test/ship-version-sync.test.ts index c657795c5..6504fa665 100644 --- a/test/ship-version-sync.test.ts +++ b/test/ship-version-sync.test.ts @@ -69,7 +69,7 @@ else fi fi`; try { - const stdout = execSync(script, { shell: "/bin/bash", encoding: "utf8" }); + const stdout = execSync(script, { shell: "/bin/bash", encoding: "utf8", timeout: 30_000 }); return { stdout: stdout.trim(), code: 0 }; } catch (e: any) { return { stdout: (e.stdout || "").toString().trim(), code: e.status ?? 1 }; @@ -88,7 +88,7 @@ if [ -f package.json ]; then node -e 'const fs=require("fs"),p=require("./package.json");p.version=process.argv[1];fs.writeFileSync("package.json",JSON.stringify(p,null,2)+"\\n")' "$NEW_VERSION" fi`; try { - execSync(script, { shell: "/bin/bash", stdio: "pipe" }); + execSync(script, { shell: "/bin/bash", stdio: "pipe", timeout: 30_000 }); return { code: 0 }; } catch (e: any) { return { code: e.status ?? 1 }; @@ -104,7 +104,7 @@ if ! printf '%s' "$REPAIR_VERSION" | grep -qE '^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9] fi node -e 'const fs=require("fs"),p=require("./package.json");p.version=process.argv[1];fs.writeFileSync("package.json",JSON.stringify(p,null,2)+"\\n")' "$REPAIR_VERSION"`; try { - execSync(script, { shell: "/bin/bash", stdio: "pipe" }); + execSync(script, { shell: "/bin/bash", stdio: "pipe", timeout: 30_000 }); return { code: 0 }; } catch (e: any) { return { code: e.status ?? 1 }; diff --git a/test/skill-e2e-auq-repetition-cut-ab.test.ts b/test/skill-e2e-auq-repetition-cut-ab.test.ts deleted file mode 100644 index a9bea1bc0..000000000 --- a/test/skill-e2e-auq-repetition-cut-ab.test.ts +++ /dev/null @@ -1,124 +0,0 @@ -/** - * AUQ no-degradation A/B: pre-cut vs post-cut AskUserQuestion Format — periodic, - * paid, SDK capture. - * - * The AskUserQuestion Format preamble section stated several of its rules more - * than once (the completeness rule three times, the auto-decide marker twice, - * the tool-not-prose rule three times). The repetition cut removes the - * duplicate statements while keeping every floor and all 14 format pins - * (Layer 0, auq-format-always-loaded.test.ts, proves presence deterministically). - * - * The risk under test: repetition may be load-bearing for RUNTIME compliance — - * a model may follow rules better because they repeat. This A/B is the gate - * that decision rested on (approved 2026-08-25, option A: "the gate outranks - * the approval"): identical prompt, two renders, and the post-cut AUQ must be - * NOT WORSE than the pre-cut AUQ on format elements and recommendation - * substance. Same harness and bar as skill-e2e-auq-verbose-vs-carved-ab. - * - * - PRE : the pre-cut plan-ceo-review/SKILL.md render, vendored at - * test/fixtures/auq-pre-cut-plan-ceo-review-SKILL.md (captured - * from branch commit 3263fffe, the last commit before the cut — - * vendored because that SHA is branch-local and unreachable from - * fresh clones after the squash-merge), with the current - * sections/ (the cut touched only the preamble skeleton). - * - POST : this worktree's render. - */ -import { test } from 'bun:test'; -import { describeE2ETier } from './helpers/e2e-gate'; -import * as fs from 'node:fs'; -import * as path from 'node:path'; -import { - setupPlanCeoDir, - captureModeSelectionAuq, - scoreAuqFormat, - carvedSkill, -} from './helpers/auq-sdk-capture'; -import { judgeRecommendation } from './helpers/llm-judge'; - -const describeE2E = describeE2ETier('periodic'); -const runId = `auq-cut-ab-${process.env.EVALS_RUN_ID ?? 'local'}`; -const PRE_CUT_FIXTURE = path.join(import.meta.dir, 'fixtures', 'auq-pre-cut-plan-ceo-review-SKILL.md'); - -async function grade(label: string, dir: string) { - const text = await captureModeSelectionAuq({ planDir: dir, testName: `auq-cut-ab-${label}`, runId }); - const fmt = scoreAuqFormat(text); - // null = judge unavailable. Never coerced to 0: a transient judge failure - // on one side must read as INCONCLUSIVE, not as a fabricated degradation - // (POST-side failure) or a masked regression (PRE-side failure) — same - // taxonomy as armJudge's judge_error cells. - let substance: number | null = null; - if (text.trim()) { - try { - const r = await judgeRecommendation(text); - substance = r.reason_substance; - } catch { /* judge unavailable — recorded as null */ } - } - // eslint-disable-next-line no-console - console.log( - `[AUQ-CUT-AB ${label}] captured=${text.length}B format=${fmt.present}/${fmt.total} ` + - `missing=[${fmt.missing.join(',')}] substance=${substance ?? 'inconclusive'}`, - ); - return { text, fmt, substance }; -} - -describeE2E('AUQ no-degradation: repetition cut (periodic)', () => { - test( - 'post-cut AskUserQuestion Format render is not worse than pre-cut on the same prompt', - async () => { - const post = carvedSkill(); - const postDir = setupPlanCeoDir({ - skillMd: post.skillMd, - sectionsFrom: post.sectionsFrom, - tmpPrefix: 'auq-cut-ab-post-', - }); - const preDir = setupPlanCeoDir({ - skillMd: fs.readFileSync(PRE_CUT_FIXTURE, 'utf-8'), - sectionsFrom: post.sectionsFrom, - tmpPrefix: 'auq-cut-ab-pre-', - }); - - let p, q; - try { - q = await grade('POST', postDir); - p = await grade('PRE', preDir); - } finally { - fs.rmSync(postDir, { recursive: true, force: true }); - fs.rmSync(preDir, { recursive: true, force: true }); - } - - const summary = [ - `POST: format ${q.fmt.present}/${q.fmt.total}, substance ${q.substance ?? 'inconclusive'}`, - `PRE : format ${p.fmt.present}/${p.fmt.total}, substance ${p.substance ?? 'inconclusive'}`, - ].join('\n'); - - if (!q.text.trim() || !p.text.trim()) { - throw new Error( - `A/B inconclusive — a side produced no AUQ capture:\n${summary}\n` + - `--- post ---\n${q.text.slice(0, 2000)}\n--- pre ---\n${p.text.slice(0, 2000)}`, - ); - } - - const formatRegressed = q.fmt.present < p.fmt.present; - // Substance compares only when BOTH judge calls succeeded; a null on - // either side logs as inconclusive and the format comparison still gates. - const substanceComparable = q.substance !== null && p.substance !== null; - if (!substanceComparable) { - // eslint-disable-next-line no-console - console.log('[AUQ-CUT-AB] substance inconclusive (judge error on at least one side) — format elements still compared.'); - } - const substanceRegressed = substanceComparable && q.substance! < p.substance! - 1; // 1-pt judge tolerance - if (formatRegressed || substanceRegressed) { - throw new Error( - `AUQ DEGRADATION from the repetition cut — the gate outranks the approval; revert the cut:\n${summary}` + - (formatRegressed ? `\n -> post-cut dropped: [${q.fmt.missing.join(',')}]` : '') + - (substanceRegressed ? `\n -> post-cut substance regressed >1 pt` : '') + - `\n--- post AUQ ---\n${q.text}\n--- pre AUQ ---\n${p.text}`, - ); - } - - // eslint-disable-next-line no-console - console.log('[AUQ-CUT-AB] NO DEGRADATION:\n' + summary); - }, - 600_000, - ); -}); diff --git a/test/skill-e2e-auto-decide-preserved.test.ts b/test/skill-e2e-auto-decide-preserved.test.ts index 2feca71cf..3482f2fe1 100644 --- a/test/skill-e2e-auto-decide-preserved.test.ts +++ b/test/skill-e2e-auto-decide-preserved.test.ts @@ -59,6 +59,7 @@ describeE2E('AUTO_DECIDE opt-in preserved under Conductor flags (periodic)', () const setRes = spawnSync(configBin, ['set', 'question_tuning', 'true'], { env: { ...process.env, GSTACK_HOME: tmpHome }, encoding: 'utf-8', + timeout: 30_000, }); if (setRes.status !== 0) { throw new Error(`gstack-config set failed: ${setRes.stderr || setRes.stdout}`); @@ -73,6 +74,7 @@ describeE2E('AUTO_DECIDE opt-in preserved under Conductor flags (periodic)', () cwd: ROOT, env: { ...process.env, GSTACK_HOME: tmpHome }, encoding: 'utf-8', + timeout: 30_000, }); // gstack-slug emits `eval`-able shell exports like `SLUG=garrytan-gstack`. const slug = (slugRes.stdout.match(/SLUG=([^\s;]+)/)?.[1] ?? 'unknown').replace(/['"]/g, ''); @@ -90,6 +92,7 @@ describeE2E('AUTO_DECIDE opt-in preserved under Conductor flags (periodic)', () { env: { ...process.env, GSTACK_HOME: tmpHome }, encoding: 'utf-8', + timeout: 30_000, }, ); if (writeRes.status !== 0) { diff --git a/test/skill-e2e-design.test.ts b/test/skill-e2e-design.test.ts index 9c97fe665..c0be419ab 100644 --- a/test/skill-e2e-design.test.ts +++ b/test/skill-e2e-design.test.ts @@ -615,7 +615,7 @@ Review the site at ${serverUrl}. Use --quick mode. Skip any AskUserQuestion call // Check if any design fix commits were made const gitLog = spawnSync('git', ['log', '--oneline'], { - cwd: qaDesignDir, stdio: 'pipe', + cwd: qaDesignDir, stdio: 'pipe', timeout: 30_000, }); const commits = gitLog.stdout.toString().trim().split('\n'); const designFixCommits = commits.filter((c: string) => c.includes('style(design)')); diff --git a/test/skill-e2e-first-task-scaffold.test.ts b/test/skill-e2e-first-task-scaffold.test.ts index 619eba1ce..ccb4df40d 100644 --- a/test/skill-e2e-first-task-scaffold.test.ts +++ b/test/skill-e2e-first-task-scaffold.test.ts @@ -78,10 +78,10 @@ describeIfSelected('first-run scaffold detection (E2E)', ['first-task-scaffold'] // greenfield bucket: git repo, zero commits. const greenDir = fs.mkdtempSync(path.join(os.tmpdir(), 'fts-green-')); try { - execSync('git init -q -b main', { cwd: nodeDir, env: GIT_ENV }); + execSync('git init -q -b main', { cwd: nodeDir, env: GIT_ENV, timeout: 30_000 }); fs.writeFileSync(path.join(nodeDir, 'package.json'), '{"name":"x"}'); - execSync('git add -A && git commit -qm init', { cwd: nodeDir, env: GIT_ENV }); - execSync('git init -q -b main', { cwd: greenDir, env: GIT_ENV }); + execSync('git add -A && git commit -qm init', { cwd: nodeDir, env: GIT_ENV, timeout: 30_000 }); + execSync('git init -q -b main', { cwd: greenDir, env: GIT_ENV, timeout: 30_000 }); const nodeOut = await detectVia(nodeDir, 'first-task-scaffold'); expect(nodeOut).toContain('code_node'); diff --git a/test/skill-e2e-ios-device.test.ts b/test/skill-e2e-ios-device.test.ts index 678d80be7..96bd55d2c 100644 --- a/test/skill-e2e-ios-device.test.ts +++ b/test/skill-e2e-ios-device.test.ts @@ -280,7 +280,7 @@ describeIfDevice('ios device path', () => { test('fixture iOS SDK and UIKit compile guards are available', () => { // This is an environment + source-guard preflight. The explicit deployment // test below performs the real signed iOS xcodebuild before installation. - const sdkPath = spawnSync('xcrun', ['--sdk', 'iphoneos', '--show-sdk-path'], { stdio: 'pipe' }); + const sdkPath = spawnSync('xcrun', ['--sdk', 'iphoneos', '--show-sdk-path'], { stdio: 'pipe', timeout: 30_000 }); if (sdkPath.status !== 0) { console.error('iOS SDK not found. Install via Xcode.'); } diff --git a/test/skill-e2e-ios-swift-build.test.ts b/test/skill-e2e-ios-swift-build.test.ts index 8fd126e15..529c6348a 100644 --- a/test/skill-e2e-ios-swift-build.test.ts +++ b/test/skill-e2e-ios-swift-build.test.ts @@ -299,7 +299,7 @@ describe('iOS tap harness regressions', () => { }); function hasSwift(): boolean { - const r = spawnSync('swift', ['--version'], { stdio: 'pipe' }); + const r = spawnSync('swift', ['--version'], { stdio: 'pipe', timeout: 30_000 }); return r.status === 0; } @@ -370,14 +370,14 @@ describeIfSwift('swift build invariants', () => { '-path', '*/release/*', '-name', '*.o', '-path', '*DebugBridge*', - ], { stdio: 'pipe' }); + ], { stdio: 'pipe', timeout: 30_000 }); const files = (oFiles.stdout?.toString() ?? '').trim().split('\n').filter(Boolean); expect(files.length).toBeGreaterThan(0); let foundForbidden = 0; const forbidden = ['StateServer', 'handleRequest', 'sessionAcquire', 'authRotate', 'snapshotGet']; for (const f of files) { - const nm = spawnSync('nm', ['-j', f], { stdio: 'pipe' }); + const nm = spawnSync('nm', ['-j', f], { stdio: 'pipe', timeout: 30_000 }); const syms = nm.stdout?.toString() ?? ''; for (const tok of forbidden) { if (syms.includes(tok)) { diff --git a/test/skill-e2e-opus-47.test.ts b/test/skill-e2e-opus-47.test.ts index 7177b2898..70c669c5d 100644 --- a/test/skill-e2e-opus-47.test.ts +++ b/test/skill-e2e-opus-47.test.ts @@ -1,17 +1,13 @@ /** * Opus 4.7 behavior evals. * - * Two cases, both pinned to claude-opus-4-7: + * One case, pinned to claude-opus-4-7: * - * 1. Fanout rate — the "Fan out explicitly" overlay nudge should make 4.7 - * spawn parallel tool calls when the prompt has independent sub-problems. - * A/B: SKILL.md regenerated with `--model opus-4-7` (overlay ON) vs - * default `--model claude` (overlay OFF). Assert A ≥ B on parallel-call - * count in the first assistant turn. - * - * 2. Routing precision — the new "when in doubt, invoke the skill" policy - * should route ambiguous dev prompts to the right skill WITHOUT routing - * casual/non-dev prompts. A handful of positive and negative controls. + * Routing precision — the "when in doubt, invoke the skill" policy should + * route ambiguous dev prompts to the right skill WITHOUT routing + * casual/non-dev prompts. A handful of positive and negative controls. + * (The fanout A/B retired 2026-08 — single-run parallel-call comparison was + * a coin flip; the SDK overlay-harness is the maintained instrument.) * * Both cases require a running Anthropic API key. Gated behind EVALS=1. * Classify as `periodic` in touchfiles — behavior measurement, not gate. @@ -61,15 +57,18 @@ const INSTALLED_SKILLS = [ function mkEvalRoot(suffix: string, includeOverlay: boolean): string { const tmp = fs.mkdtempSync(path.join(os.tmpdir(), `opus47-${suffix}-`)); - // Regenerate at opus-4-7 so the per-skill SKILL.md files reflect that - // model's overlay. If includeOverlay is false we'll re-regen at default - // later just for the root SKILL.md copy. For individual skills, opus-4-7 - // content doesn't matter for the routing test (we only need discovery). + // Render at opus-4-7 INTO A MKDTEMP via --out-dir — never the live tree. + // The previous cwd=ROOT regeneration rewrote every in-repo SKILL.md + // mid-run while concurrent paid shards copyFileSync those same files in + // their beforeAll (EVALS_JOBS>=4 locally, 2 per CI slice): a sibling could + // capture a half-regenerated or opus-rendered SKILL.md, and a timeout + // before afterAll left the whole tree rendered at the wrong model for + // every later shard. --out-dir mirrors the repo layout (// + // SKILL.md), which is all this fixture reads. + const renderDir = fs.mkdtempSync(path.join(os.tmpdir(), `opus47-render-${suffix}-`)); const result = spawnSync( 'bun', - ['run', 'scripts/gen-skill-docs.ts', '--model', includeOverlay ? 'opus-4-7' : 'claude'], - // LIVE-REPO CWD: gen-skill-docs reads .tmpl sources and regenerates the - // in-repo SKILL.md files (restored to default in afterAll below). + ['run', 'scripts/gen-skill-docs.ts', '--model', includeOverlay ? 'opus-4-7' : 'claude', '--out-dir', renderDir], { cwd: ROOT, stdio: 'pipe', encoding: 'utf-8', timeout: 60_000 }, ); if (result.status !== 0) { @@ -82,12 +81,13 @@ function mkEvalRoot(suffix: string, includeOverlay: boolean): string { // (CLAUDE.md: "E2E test fixtures: extract, don't copy"). const skillsDir = path.join(tmp, '.claude', 'skills'); for (const skill of INSTALLED_SKILLS) { - const src = path.join(ROOT, skill, 'SKILL.md'); + const src = path.join(renderDir, skill, 'SKILL.md'); if (!fs.existsSync(src)) continue; const destDir = path.join(skillsDir, skill); fs.mkdirSync(destDir, { recursive: true }); fs.writeFileSync(path.join(destDir, 'SKILL.md'), extractSkillHead(src)); } + fs.rmSync(renderDir, { recursive: true, force: true }); // Extract the opus-4-7 model-overlay content from the checked-in file // so we can inline it into CLAUDE.md when includeOverlay is true. @@ -167,104 +167,18 @@ const ROUTING_CASES: RoutingCase[] = [ describeE2E('Opus 4.7 overlay behavior evals', () => { afterAll(() => { evalCollector?.finalize(); - // Restore working tree: mkEvalRoot runs `gen-skill-docs` with various - // --model flags, leaving the in-repo SKILL.md files generated at - // whichever model ran last. Reset to the default (claude) so the tree - // matches what would be checked in. - spawnSync('bun', ['run', 'scripts/gen-skill-docs.ts'], { - // LIVE-REPO CWD: restores the in-repo SKILL.md files to the default - // model render after mkEvalRoot's --model regens. - cwd: ROOT, - stdio: 'pipe', - timeout: 60_000, - }); + // No tree restore needed: mkEvalRoot renders into a mkdtemp via + // --out-dir, so the live repo's SKILL.md files are never touched — a + // timeout mid-run can no longer strand the tree at the wrong model for + // concurrent shards. }); - test( - 'fanout: overlay ON emits >= parallel calls vs overlay OFF on 3-file investigate task', - async () => { - const armA = mkEvalRoot('on', true); - const armB = mkEvalRoot('off', false); + // (fanout A/B retired, 2026-08 audit: it compared parallel-call counts of + // two SINGLE stochastic runs — parA >= parB is a coin flip with near-zero + // remaining information; the overlay-fanout question is answered and the + // SDK overlay-harness (test/skill-e2e-overlay-harness.test.ts) is the + // maintained instrument for the next experiment.) - // Populate three tiny independent files in each arm. The prompt asks - // the agent to read all three and report. Opus 4.7 (without nudge) - // tends to serialize; with the nudge it should parallelize. - for (const dir of [armA, armB]) { - fs.writeFileSync(path.join(dir, 'alpha.txt'), 'alpha content: 1\n'); - fs.writeFileSync(path.join(dir, 'beta.txt'), 'beta content: 2\n'); - fs.writeFileSync(path.join(dir, 'gamma.txt'), 'gamma content: 3\n'); - } - - const prompt = - "Read alpha.txt, beta.txt, and gamma.txt in this directory and report what's inside each. These three reads are independent."; - - try { - const [resA, resB] = await Promise.all([ - runSkillTest({ - prompt, - workingDirectory: armA, - maxTurns: 5, - allowedTools: ['Read', 'Bash', 'Glob', 'Grep'], - timeout: JUDGE_MS, - testName: 'fanout-arm-overlay-on', - runId, - model: OPUS_47, - }), - runSkillTest({ - prompt, - workingDirectory: armB, - maxTurns: 5, - allowedTools: ['Read', 'Bash', 'Glob', 'Grep'], - timeout: JUDGE_MS, - testName: 'fanout-arm-overlay-off', - runId, - model: OPUS_47, - }), - ]); - - const parA = firstTurnParallelism(resA.transcript); - const parB = firstTurnParallelism(resB.transcript); - - console.log( - `[opus-4-7 fanout] arm A (overlay ON): ${parA} parallel tool calls in first turn; ` + - `arm B (overlay OFF): ${parB}`, - ); - console.log(` cost A=$${resA.costEstimate.estimatedCost.toFixed(2)} B=$${resB.costEstimate.estimatedCost.toFixed(2)}`); - - evalCollector?.addTest({ - name: 'fanout-arm-overlay-on', - suite: 'Opus 4.7 overlay', - tier: 'e2e', - passed: parA >= parB, - duration_ms: resA.duration, - cost_usd: resA.costEstimate.estimatedCost, - transcript: resA.transcript, - output: `parallel=${parA}`, - turns_used: resA.costEstimate.turnsUsed, - exit_reason: resA.exitReason, - }); - evalCollector?.addTest({ - name: 'fanout-arm-overlay-off', - suite: 'Opus 4.7 overlay', - tier: 'e2e', - passed: true, // baseline arm, recorded for comparison - duration_ms: resB.duration, - cost_usd: resB.costEstimate.estimatedCost, - transcript: resB.transcript, - output: `parallel=${parB}`, - turns_used: resB.costEstimate.turnsUsed, - exit_reason: resB.exitReason, - }); - - // Main assertion: overlay arm is at least as parallel as baseline. - expect(parA, `overlay arm emitted ${parA} parallel calls, baseline ${parB}`).toBeGreaterThanOrEqual(parB); - } finally { - fs.rmSync(armA, { recursive: true, force: true }); - fs.rmSync(armB, { recursive: true, force: true }); - } - }, - CAPTURE_MS, - ); test( 'routing precision: positives route, negatives do not', diff --git a/test/skill-e2e-plan-tune-cathedral.test.ts b/test/skill-e2e-plan-tune-cathedral.test.ts index 85a455036..bd5d8fc3c 100644 --- a/test/skill-e2e-plan-tune-cathedral.test.ts +++ b/test/skill-e2e-plan-tune-cathedral.test.ts @@ -45,12 +45,12 @@ function scaffoldFixture(prefix: string): { workDir: string; stateRoot: string; fs.mkdirSync(stateRoot, { recursive: true }); // git init so gstack-slug resolves a deterministic slug. - spawnSync('git', ['init', '-b', 'main'], { cwd: workDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 't@t.com'], { cwd: workDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'T'], { cwd: workDir, stdio: 'pipe' }); + spawnSync('git', ['init', '-b', 'main'], { cwd: workDir, stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['config', 'user.email', 't@t.com'], { cwd: workDir, stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['config', 'user.name', 'T'], { cwd: workDir, stdio: 'pipe', timeout: 30_000 }); fs.writeFileSync(path.join(workDir, 'README.md'), '# cathedral fixture\n'); - spawnSync('git', ['add', '.'], { cwd: workDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: workDir, stdio: 'pipe' }); + spawnSync('git', ['add', '.'], { cwd: workDir, stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['commit', '-m', 'init'], { cwd: workDir, stdio: 'pipe', timeout: 30_000 }); // Copy bins. const binDir = path.join(workDir, 'bin'); @@ -146,6 +146,7 @@ describeIfSelected('PlanTune cathedral E2E: hook capture', ['plan-tune-hook-capt }, input: JSON.stringify(payload), encoding: 'utf-8', + timeout: 30_000, }); expect(res.status).toBe(0); const logPath = path.join(fixture.stateRoot, 'projects', fixture.slug, 'question-log.jsonl'); @@ -209,6 +210,7 @@ describeIfSelected('PlanTune cathedral E2E: enforcement', ['plan-tune-enforcemen }, input: JSON.stringify(payload), encoding: 'utf-8', + timeout: 30_000, }); expect(res.status).toBe(0); const parsed = JSON.parse(res.stdout || '{}'); @@ -293,6 +295,7 @@ describeIfSelected('PlanTune cathedral E2E: annotation', ['plan-tune-annotation' }, input: JSON.stringify(payload), encoding: 'utf-8', + timeout: 30_000, }); expect(res.status).toBe(0); const parsed = JSON.parse(res.stdout || '{}'); @@ -352,6 +355,7 @@ describeIfSelected('PlanTune cathedral E2E: codex import', ['plan-tune-codex-imp }, encoding: 'utf-8', cwd: fixture.workDir, + timeout: 30_000, }); expect(res.status).toBe(0); expect(res.stdout).toContain('IMPORTED: 1'); @@ -412,6 +416,7 @@ describeIfSelected('PlanTune cathedral E2E: dream cycle', ['plan-tune-dream-cycl env: { ...process.env, GSTACK_STATE_ROOT: fixture.stateRoot }, encoding: 'utf-8', cwd: fixture.workDir, + timeout: 30_000, }); expect(applyRes.status).toBe(0); @@ -453,6 +458,7 @@ describeIfSelected('PlanTune cathedral E2E: dream cycle', ['plan-tune-dream-cycl }, input: JSON.stringify(payload), encoding: 'utf-8', + timeout: 30_000, }); expect(hookRes.status).toBe(0); const parsed = JSON.parse(hookRes.stdout || '{}'); diff --git a/test/skill-e2e-preamble-script-ab.test.ts b/test/skill-e2e-preamble-script-ab.test.ts deleted file mode 100644 index 2a2603c8a..000000000 --- a/test/skill-e2e-preamble-script-ab.test.ts +++ /dev/null @@ -1,108 +0,0 @@ -/** - * Preamble script-consolidation A/B: inline-bash render vs script render — - * gate during token-reduction Phases 1-3 (demote to periodic after Phase 3, - * plan OV7), paid, SDK capture. - * - * Phase 1 moved ~13KB of inline preamble bash per skill into - * bin/gstack-skill-start. Layer 0 (test/gstack-skill-start.test.ts) proves the - * script emits the same STATUS lines deterministically; THIS proves the model - * driven by the slim render still runs the preamble and produces an - * equal-quality decision brief on the same prompt. - * - * Arms (precedent: skill-e2e-auq-verbose-vs-carved-ab.test.ts): - * - INLINE : pre-Phase-1 plan-ceo-review/SKILL.md read from git - * (29785978 = the v1.69.1.0 bump, the last inline-bash render). - * - SCRIPT : this worktree's render, with the fence's install-root bin path - * rewritten to THIS WORKTREE's bin/ (plan EOV2: hermetic evals - * resolve $HOME/.claude/skills/gstack/bin to the operator - * install, which would silently exercise the degraded path; - * the rewrite makes the branch's script the subject under test). - * - * Both arms pin GSTACK_HOME to the fixture dir (EOV7: onboarding state is - * hermetic now that the script honors GSTACK_HOME). - */ -import { test, expect } from 'bun:test'; -import { describeE2ETier } from './helpers/e2e-gate'; -import * as fs from 'node:fs'; -import * as path from 'node:path'; -import { execSync } from 'node:child_process'; -import { - setupPlanCeoDir, - captureModeSelectionAuq, - scoreAuqFormat, - carvedSkill, -} from './helpers/auq-sdk-capture'; - -const describeE2E = describeE2ETier('periodic'); -const runId = `preamble-ab-${process.env.EVALS_RUN_ID ?? 'local'}`; -const ROOT = path.resolve(import.meta.dir, '..'); -const INLINE_REF = '29785978'; // last pre-Phase-1 commit (v1.69.1.0 bump) - -function inlineSkill(): string { - return execSync(`git show ${INLINE_REF}:plan-ceo-review/SKILL.md`, { - // LIVE-REPO CWD: git show needs this repo's history to read the - // pre-Phase-1 SKILL.md render at INLINE_REF. - cwd: ROOT, - encoding: 'utf-8', - maxBuffer: 8 * 1024 * 1024, - }); -} - -/** EOV2 redirection: point the fence at the worktree's bin. */ -function scriptSkillWorktreeBin(): string { - const current = carvedSkill(); - const rewritten = current.skillMd.replaceAll( - '$HOME/.claude/skills/gstack/bin/gstack-skill-start', - path.join(ROOT, 'bin', 'gstack-skill-start'), - ); - if (!rewritten.includes(path.join(ROOT, 'bin', 'gstack-skill-start'))) { - throw new Error('binDir rewrite matched nothing — fence shape changed; update the A/B redirection'); - } - return rewritten; -} - -async function grade(label: string, dir: string) { - const text = await captureModeSelectionAuq({ planDir: dir, testName: `preamble-ab-${label}`, runId }); - const fmt = scoreAuqFormat(text); - // eslint-disable-next-line no-console - console.log(`[PREAMBLE-AB ${label}] captured=${text.length}B format=${fmt.present}/${fmt.total} missing=[${fmt.missing.join(',')}]`); - return { text, fmt }; -} - -describeE2E('Preamble consolidation no-degradation: inline bash vs script (gate)', () => { - test( - 'script-render plan-ceo-review AUQ is not worse than the inline-bash render on the same prompt', - async () => { - const sections = carvedSkill().sectionsFrom; - const scriptDir = setupPlanCeoDir({ - skillMd: scriptSkillWorktreeBin(), - sectionsFrom: sections, - tmpPrefix: 'preamble-ab-script-', - }); - const inlineDir = setupPlanCeoDir({ - skillMd: inlineSkill(), - sectionsFrom: sections, - tmpPrefix: 'preamble-ab-inline-', - }); - - let s, i; - try { - s = await grade('SCRIPT', scriptDir); - i = await grade('INLINE', inlineDir); - } finally { - fs.rmSync(scriptDir, { recursive: true, force: true }); - fs.rmSync(inlineDir, { recursive: true, force: true }); - } - - // Both arms must produce a capture at all (an empty script-arm capture - // means the preamble derailed the workflow — exactly the regression this - // guards against). - expect(s.text.length).toBeGreaterThan(100); - expect(i.text.length).toBeGreaterThan(100); - // Relative parity: the script render is NOT WORSE on decision-brief - // format elements (absolute compliance is auq-format-gate's job). - expect(s.fmt.present).toBeGreaterThanOrEqual(i.fmt.present); - }, - 20 * 60 * 1000, - ); -}); diff --git a/test/skill-e2e-qa-workflow.test.ts b/test/skill-e2e-qa-workflow.test.ts index d2c028d46..a07315d7f 100644 --- a/test/skill-e2e-qa-workflow.test.ts +++ b/test/skill-e2e-qa-workflow.test.ts @@ -151,7 +151,7 @@ Write your report to ${qaOnlyDir}/qa-reports/qa-only-report.md`, // Verify git working tree is still clean (no source modifications) const gitStatus = spawnSync('git', ['status', '--porcelain'], { - cwd: qaOnlyDir, stdio: 'pipe', + cwd: qaOnlyDir, stdio: 'pipe', timeout: 30_000, }); const statusLines = gitStatus.stdout.toString().trim().split('\n').filter( (l: string) => l.trim() && !l.includes('.prompt-tmp') && !l.includes('.gstack/') && !l.includes('qa-reports/'), @@ -263,7 +263,7 @@ This is a test+fix loop: find bugs, fix them in the source code, commit each fix // Verify at least one fix commit was made beyond the initial commit const gitLog = spawnSync('git', ['log', '--oneline'], { - cwd: qaFixDir, stdio: 'pipe', + cwd: qaFixDir, stdio: 'pipe', timeout: 30_000, }); const commits = gitLog.stdout.toString().trim().split('\n'); console.log(`/qa fix loop: ${commits.length} commits total (1 initial + ${commits.length - 1} fixes)`); diff --git a/test/skill-e2e-review-attribution.test.ts b/test/skill-e2e-review-attribution.test.ts index 6a60f1d1f..5098eb889 100644 --- a/test/skill-e2e-review-attribution.test.ts +++ b/test/skill-e2e-review-attribution.test.ts @@ -184,7 +184,7 @@ describeIfSelected('Review Dashboard Via Attribution', ['review-dashboard-via'], run('git', ['commit', '-m', 'feat: update']); // Get HEAD commit for review entries - const headResult = spawnSync('git', ['rev-parse', '--short', 'HEAD'], { cwd: dashDir, stdio: 'pipe' }); + const headResult = spawnSync('git', ['rev-parse', '--short', 'HEAD'], { cwd: dashDir, stdio: 'pipe', timeout: 30_000 }); const commit = headResult.stdout.toString().trim(); // Pre-populate review log with autoplan-sourced entries diff --git a/test/skill-e2e-session-intelligence.test.ts b/test/skill-e2e-session-intelligence.test.ts index ae8bb50f8..90fdbe149 100644 --- a/test/skill-e2e-session-intelligence.test.ts +++ b/test/skill-e2e-session-intelligence.test.ts @@ -78,14 +78,14 @@ describeIfSelected('Session Intelligence E2E', [ spawnSync(logBin, [JSON.stringify({ skill: 'review', event: 'started', branch: 'main', session: 'test-1', - })], opts); + })], opts); // timeout via opts spawnSync(logBin, [JSON.stringify({ skill: 'review', event: 'completed', branch: 'main', outcome: 'success', duration_s: 120, session: 'test-1', - })], opts); + })], opts); // timeout via opts // Read via gstack-timeline-read - const readResult = spawnSync(readBin, ['--branch', 'main'], opts); + const readResult = spawnSync(readBin, ['--branch', 'main'], opts); // timeout via opts const readOutput = readResult.stdout?.toString() || ''; // Verify timeline.jsonl exists and has content diff --git a/test/skill-e2e-ship-idempotency.test.ts b/test/skill-e2e-ship-idempotency.test.ts index a20020dfa..29ef83827 100644 --- a/test/skill-e2e-ship-idempotency.test.ts +++ b/test/skill-e2e-ship-idempotency.test.ts @@ -137,12 +137,12 @@ function snapshotFixture(workTree: string): FixtureSnapshot { const changelog = fs.readFileSync(path.join(workTree, 'CHANGELOG.md'), 'utf-8'); // Count `## [0.0.2]` headings — should stay at 1 across re-runs. const changelogEntryCount = (changelog.match(/^##\s*\[0\.0\.2\]/gm) ?? []).length; - const head = spawnSync('git', ['rev-parse', 'HEAD'], { cwd: workTree, stdio: 'pipe' }); + const head = spawnSync('git', ['rev-parse', 'HEAD'], { cwd: workTree, stdio: 'pipe', timeout: 30_000 }); const branchHead = head.stdout?.toString().trim() ?? ''; // Count "chore: bump version" commits on this branch since main. const log = spawnSync( 'git', ['log', '--format=%s', 'main..HEAD'], - { cwd: workTree, stdio: 'pipe' }, + { cwd: workTree, stdio: 'pipe', timeout: 30_000 }, ); const subjects = log.stdout?.toString() ?? ''; const bumpCommitCount = subjects.split('\n').filter(s => /chore:\s*bump\s+version/i.test(s)).length; diff --git a/test/skill-e2e-triage.test.ts b/test/skill-e2e-triage.test.ts index e971c5e23..89f75db18 100644 --- a/test/skill-e2e-triage.test.ts +++ b/test/skill-e2e-triage.test.ts @@ -101,6 +101,7 @@ const path = require('path'); let failures = 0; for (const f of ['math.test.js', 'string.test.js']) { try { + // tripwire-exempt: line below is CONTENT of the generated run.js fixture (child-executed), not a test-process call execSync('node ' + path.join(__dirname, f), { stdio: 'inherit' }); } catch (e) { failures++; diff --git a/test/skill-e2e-workflow.test.ts b/test/skill-e2e-workflow.test.ts index 055974e9c..5db37f089 100644 --- a/test/skill-e2e-workflow.test.ts +++ b/test/skill-e2e-workflow.test.ts @@ -183,7 +183,7 @@ describeIfSelected('Ship workflow E2E', ['ship-local-workflow'], () => { logCost('/ship local workflow', result); // Check push succeeded — verify the feature branch exists on the bare remote - const branchCheck = spawnSync('git', ['branch', '--list', 'feature/ship-test'], { cwd: shipRemoteDir, stdio: 'pipe' }); + const branchCheck = spawnSync('git', ['branch', '--list', 'feature/ship-test'], { cwd: shipRemoteDir, stdio: 'pipe', timeout: 30_000 }); const branchExists = branchCheck.stdout.toString().trim().length > 0; // Check VERSION was bumped locally (even if push failed, this shows the LLM did the work) diff --git a/test/skill-routing-e2e.test.ts b/test/skill-routing-e2e.test.ts index 7e86b03f6..cdf91463b 100644 --- a/test/skill-routing-e2e.test.ts +++ b/test/skill-routing-e2e.test.ts @@ -90,10 +90,14 @@ function installSkills(tmpDir: string) { fs.writeFileSync(path.join(destDir, 'SKILL.md'), extractSkillHead(srcPath)); } - // Write a CLAUDE.md with explicit routing instructions. - // The skill descriptions in system-reminder aren't strong enough to override - // Claude's default behavior of answering directly. A CLAUDE.md instruction - // puts routing rules in project context which Claude weighs more heavily. + // Write a CLAUDE.md with a GENERIC invoke-skills nudge — deliberately NO + // per-skill routing table. These journey tests exist to catch skill + // DESCRIPTION regressions (their touchfiles key on */SKILL.md.tmpl), and + // the old fixture shipped an explicit prompt→skill answer key: with the + // lookup table in context, a badly regressed frontmatter description + // still routed correctly and the tests could not fail on the regression + // class they select for (2026-08 audit). The generic nudge keeps Claude's + // reach-for-a-skill posture; the FRONTMATTER carries the routing load. fs.writeFileSync(path.join(tmpDir, 'CLAUDE.md'), `# Project Instructions ## Skill routing @@ -101,18 +105,7 @@ function installSkills(tmpDir: string) { When the user's request matches an available skill, ALWAYS invoke it using the Skill tool as your FIRST action. Do NOT answer directly, do NOT use other tools first. The skill has specialized workflows that produce better results than ad-hoc answers. - -Key routing rules: -- Product ideas, "is this worth building", brainstorming → invoke office-hours -- Bugs, errors, "why is this broken", 500 errors → invoke investigate -- Ship, deploy, push, create PR → invoke ship -- QA, test the site, find bugs → invoke qa -- Code review, check my diff → invoke review -- Update docs after shipping → invoke document-release -- Weekly retro → invoke retro -- Design system, brand → invoke design-consultation -- Visual audit, design polish → invoke design-review -- Architecture review → invoke plan-eng-review +Choose the skill by matching the request against each skill's description. `); } @@ -196,8 +189,12 @@ describeE2E('Skill Routing E2E — Developer Journey', () => { const result = await runSkillTest({ prompt: "I've been thinking about building a waitlist management tool for restaurants. The existing solutions are expensive and overcomplicated. I want something simple — a tablet app where hosts can add parties, see wait times, and text customers when their table is ready. Help me think through whether this is worth building and what the key design decisions are.", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -246,8 +243,12 @@ describeE2E('Skill Routing E2E — Developer Journey', () => { const result = await runSkillTest({ prompt: "I wrote up a plan for the waitlist app in plan.md. Can you take a look at the architecture and make sure I'm not missing any edge cases or failure modes before I start coding?", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -308,8 +309,12 @@ export default app; const result = await runSkillTest({ prompt: "The GET /api/waitlist endpoint was working fine yesterday but now it's returning 500 errors. The tests are passing locally but the endpoint fails when I hit it with curl. Can you figure out what's going on?", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -344,8 +349,12 @@ export default app; const result = await runSkillTest({ prompt: "I think the app is mostly working now. Can you go through the site and test everything — find any bugs and fix them?", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -385,8 +394,12 @@ export default app; const result = await runSkillTest({ prompt: "I'm about to merge this into main. Can you look over my changes and flag anything risky before I land it?", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -424,8 +437,12 @@ export default app; const result = await runSkillTest({ prompt: "This looks good. Let's get it deployed — push the code up and create a PR.", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -461,8 +478,12 @@ export default app; const result = await runSkillTest({ prompt: "We just shipped the waitlist feature. Can you go through the README and any other docs and make sure they match what we actually built?", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -504,8 +525,12 @@ export default app; const result = await runSkillTest({ prompt: "It's Friday. What did we ship this week? I want to do a quick retrospective on what the team accomplished.", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -533,8 +558,12 @@ export default app; const result = await runSkillTest({ prompt: "Before we build the UI, I want to establish a design system — typography, colors, spacing, the whole thing. Can you put together brand guidelines for this project?", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, @@ -584,8 +613,12 @@ body { font-family: sans-serif; } const result = await runSkillTest({ prompt: "Something looks off on the site. The spacing between sections is inconsistent and the font sizes don't feel right. Can you audit the visual design and fix anything that doesn't look polished?", workingDirectory: tmpDir, - maxTurns: 5, - allowedTools: ['Skill', 'Read', 'Bash', 'Glob', 'Grep'], + // Turn/tool cap (2026-08 audit): only the FIRST Skill call is + // asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — the + // session ends at the routing decision, roughly halving each + // journey's cost. + maxTurns: 2, + allowedTools: ['Skill', 'Read'], timeout: JUDGE_MS, testName, runId, diff --git a/test/skill-size-budget.test.ts b/test/skill-size-budget.test.ts index 3c61638f0..aa8104723 100644 --- a/test/skill-size-budget.test.ts +++ b/test/skill-size-budget.test.ts @@ -239,7 +239,7 @@ describe('SKILL.md size budget regression (gate, free)', () => { // estimate was a moving target: 4177 solo, 8356 and 8041 in two parallel // runs. A repo-budget ratchet measures the catalog that ships; CI always // checks the PR's committed tree anyway. - const trackedPaths = execSync('git ls-files -- "*/SKILL.md"', { cwd: REPO_ROOT, encoding: 'utf-8' }) + const trackedPaths = execSync('git ls-files -- "*/SKILL.md"', { cwd: REPO_ROOT, encoding: 'utf-8', timeout: 30_000 }) .split('\n') .filter(Boolean) .filter((p) => p.split('/').length === 2); @@ -249,6 +249,7 @@ describe('SKILL.md size budget regression (gate, free)', () => { cwd: REPO_ROOT, encoding: 'utf-8', maxBuffer: 8 * 1024 * 1024, + timeout: 30_000, }); descriptionBytes += Buffer.byteLength(extractDescription(committed), 'utf-8'); } diff --git a/test/skill-validation.test.ts b/test/skill-validation.test.ts index 0c1013f29..a0aa8690c 100644 --- a/test/skill-validation.test.ts +++ b/test/skill-validation.test.ts @@ -19,7 +19,7 @@ const CODEX_OUT = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-skillval-codex-' { const render = Bun.spawnSync( ['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--out-dir', CODEX_OUT], - { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }, + { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 }, ); if (render.exitCode !== 0) { throw new Error( @@ -383,7 +383,7 @@ describe('Update check preamble', () => { const result = Bun.spawnSync(['bash', '-c', '_sanitize() { sed "s/GSTACK_INSTRUCTION/GSTACK-INSTRUCTION-(stripped)/g"; }; ' + '_UPD=$(echo "" || true); [ -n "$_UPD" ] && printf "%s\\n" "$_UPD" | _sanitize || true' - ], { stdout: 'pipe', stderr: 'pipe' }); + ], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.exitCode).toBe(0); }); @@ -391,7 +391,7 @@ describe('Update check preamble', () => { const result = Bun.spawnSync(['bash', '-c', '_sanitize() { sed "s/GSTACK_INSTRUCTION/GSTACK-INSTRUCTION-(stripped)/g"; }; ' + '_UPD=$(echo "UPGRADE_AVAILABLE 0.3.3 0.4.0" || true); [ -n "$_UPD" ] && printf "%s\\n" "$_UPD" | _sanitize || true' - ], { stdout: 'pipe', stderr: 'pipe' }); + ], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.exitCode).toBe(0); expect(result.stdout.toString().trim()).toBe('UPGRADE_AVAILABLE 0.3.3 0.4.0'); }); @@ -1074,7 +1074,7 @@ describe('gstack-slug', () => { }); test('outputs SLUG and BRANCH lines in a git repo', () => { - const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }); + const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); expect(result.exitCode).toBe(0); const output = result.stdout.toString(); expect(output).toContain('SLUG='); @@ -1082,21 +1082,21 @@ describe('gstack-slug', () => { }); test('SLUG does not contain forward slashes', () => { - const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }); + const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const slug = result.stdout.toString().match(/SLUG=(.*)/)?.[1] ?? ''; expect(slug).not.toContain('/'); expect(slug.length).toBeGreaterThan(0); }); test('BRANCH does not contain forward slashes', () => { - const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }); + const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const branch = result.stdout.toString().match(/BRANCH=(.*)/)?.[1] ?? ''; expect(branch).not.toContain('/'); expect(branch.length).toBeGreaterThan(0); }); test('output is eval-compatible (KEY=VALUE format)', () => { - const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }); + const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const lines = result.stdout.toString().trim().split('\n'); expect(lines.length).toBe(2); expect(lines[0]).toMatch(/^SLUG=.+/); @@ -1104,7 +1104,7 @@ describe('gstack-slug', () => { }); test('output values contain only safe characters (no shell metacharacters)', () => { - const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }); + const result = Bun.spawnSync([SLUG_BIN], { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 }); const slug = result.stdout.toString().match(/SLUG=(.*)/)?.[1] ?? ''; const branch = result.stdout.toString().match(/BRANCH=(.*)/)?.[1] ?? ''; // Only alphanumeric, dot, dash, underscore are allowed (#133) @@ -1114,7 +1114,7 @@ describe('gstack-slug', () => { test('eval sets variables under bash with set -euo pipefail', () => { const result = Bun.spawnSync( ['bash', '-c', 'set -euo pipefail; eval "$(./bin/gstack-slug 2>/dev/null)"; echo "SLUG=$SLUG"; echo "BRANCH=$BRANCH"'], - { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' } + { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 } ); expect(result.exitCode).toBe(0); const output = result.stdout.toString(); @@ -1125,7 +1125,7 @@ describe('gstack-slug', () => { test('no templates or bin scripts use source process substitution for gstack-slug', () => { const result = Bun.spawnSync( ['grep', '-r', 'source <(.*gstack-slug', '--include=*.tmpl', '--include=gstack-review-*', '.'], - { cwd: ROOT, stdout: 'pipe', stderr: 'pipe' } + { cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 30_000 } ); // grep returns exit code 1 when no matches found — that's what we want expect(result.stdout.toString().trim()).toBe(''); @@ -1994,7 +1994,7 @@ describe('no compiled binaries in git', () => { // Tracked files enumerated once and reused by both assertions. git ls-files -z // + split is ~ms; the previous xargs-per-file shell loops blew past 5s on CI. const trackedFiles: string[] = require('child_process') - .execSync('git ls-files -z', { cwd: ROOT, encoding: 'utf-8' }) + .execSync('git ls-files -z', { cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }) .split('\0') .filter(Boolean); @@ -2004,6 +2004,7 @@ describe('no compiled binaries in git', () => { const lsOut: string = require('child_process').execSync('git ls-files -s', { cwd: ROOT, encoding: 'utf-8', + timeout: 30_000, }); const executableFiles = lsOut .split('\n') @@ -2022,6 +2023,7 @@ describe('no compiled binaries in git', () => { .execSync(`file --mime-type -- ${executableFiles.map((f: string) => `'${f.replace(/'/g, "'\\''")}'`).join(' ')}`, { cwd: ROOT, encoding: 'utf-8', + timeout: 30_000, }) .trim(); diff --git a/test/spawnsync-timeout-tripwire.test.ts b/test/spawnsync-timeout-tripwire.test.ts new file mode 100644 index 000000000..569a017ab --- /dev/null +++ b/test/spawnsync-timeout-tripwire.test.ts @@ -0,0 +1,132 @@ +/** + * Synchronous-spawn timeout tripwire — the enforcement for the shard-wedge + * class that reached main. + * + * Mechanism: spawnSync / execSync / Bun.spawnSync BLOCK the main thread, so + * bun's in-process per-test timeout can never fire while one waits. A child + * that hangs (stdin read, network probe, dead daemon) wedges the whole shard + * until the runner's external wall-clock SIGKILL — observed live on main: + * test/gstack-memory-ingest.test.ts (normally 2.3s) held shard 2 at the + * 360s wall on free-tests run 33262077256 while its five siblings finished + * in ~65s. The 2026-08 sweep added a `timeout` to ~400 call sites across + * ~130 test files; this tripwire keeps the class extinct. + * + * Rule: every sync-spawn call site in the test trees must carry a `timeout` + * within its options window (WINDOW_LINES below), or the line above / the + * call line must carry an explicit exemption marker with a reason: + * + * // tripwire-exempt: + * + * Exemptions are counted and ratcheted (EXEMPT_CEILING can only go down). + */ +import { describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +const ROOT = path.resolve(import.meta.dir, '..'); + +// Mirrors the free runner's TEST_ROOTS plus test/helpers (helpers spawn on +// behalf of tests and wedge shards just the same). +const SCAN_ROOTS = [ + 'test', + 'browse/test', + 'design/test', + 'make-pdf/test', + 'ios-qa/daemon/test', + 'ios-qa/scripts', + 'browser-skills', +]; + +const SYNC_SPAWN = /\b(?:spawnSync|execSync|execFileSync|Bun\.spawnSync)\s*\(/; +/** Generous on purpose: multi-line arg arrays push the options object far + * below the call line (observed: +13 lines in codex-model-probe). A wide + * window trades a sliver of false-negative risk for zero rename churn. */ +const WINDOW_LINES = 30; +const EXEMPT_MARKER = /tripwire-exempt:/; +/** Comment lines legitimately NAME the calls (doc headers, grep-needle + * tables in sibling tripwires) without being call sites. */ +const COMMENT_LINE = /^\s*(?:\/\/|\*|\/\*)/; + +/** Shrink-only: lower it when exemptions burn down; never raise it. */ +const EXEMPT_CEILING = 8; + +const SELF = path.join('test', 'spawnsync-timeout-tripwire.test.ts'); + +function walk(dir: string, out: string[] = []): string[] { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) walk(full, out); + else if (/\.(?:[cm]?[jt]s|tsx)$/.test(entry.name)) out.push(full); + } + return out; +} + +interface Violation { file: string; line: number; text: string } + +function scan(): { violations: Violation[]; exempt: Violation[] } { + const violations: Violation[] = []; + const exempt: Violation[] = []; + for (const root of SCAN_ROOTS) { + const abs = path.join(ROOT, root); + if (!fs.existsSync(abs)) continue; + for (const file of walk(abs)) { + const rel = path.relative(ROOT, file); + if (rel === SELF) continue; + const lines = fs.readFileSync(file, 'utf-8').split('\n'); + for (let i = 0; i < lines.length; i += 1) { + if (!SYNC_SPAWN.test(lines[i])) continue; + if (COMMENT_LINE.test(lines[i])) continue; + const record = { file: rel, line: i + 1, text: lines[i].trim().slice(0, 120) }; + if (EXEMPT_MARKER.test(lines[i]) || (i > 0 && EXEMPT_MARKER.test(lines[i - 1]))) { + exempt.push(record); + continue; + } + const window = lines.slice(i, i + WINDOW_LINES).join('\n'); + if (!/\btimeout\b/.test(window)) violations.push(record); + } + } + } + return { violations, exempt }; +} + +describe('sync-spawn timeout tripwire', () => { + const { violations, exempt } = scan(); + + test('every sync spawn in the test trees carries a timeout (or a reasoned exemption)', () => { + const detail = violations + .map((v) => ` ${v.file}:${v.line} ${v.text}`) + .join('\n'); + expect( + violations, + `sync-spawn call site(s) without a timeout — each one can wedge a whole shard to its wall-clock kill (bun's per-test timeout cannot interrupt a blocking spawn):\n${detail}\n` + + 'Fix: add `timeout: 30_000` (120_000 for genuinely slow ops) to the options, or fix the wrapping helper once. ' + + 'A call that must legitimately block unbounded gets `// tripwire-exempt: ` on or above the line.', + ).toEqual([]); + }); + + test('exemptions only shrink', () => { + expect( + exempt.length, + `tripwire-exempt count ${exempt.length} exceeds the ceiling ${EXEMPT_CEILING}. ` + + 'New unbounded sync spawns are not allowed — add a timeout instead. If an exemption was ' + + `removed, lower EXEMPT_CEILING in ${SELF} in the same commit:\n` + + exempt.map((v) => ` ${v.file}:${v.line}`).join('\n'), + ).toBeLessThanOrEqual(EXEMPT_CEILING); + }); + + test('scan sanity: the pattern still matches real code (must not rot to vacuous green)', () => { + // The test trees legitimately contain hundreds of sync spawns WITH + // timeouts; if the scanner suddenly sees none at all, the regex or the + // roots rotted and the tripwire is scanning nothing. + let total = 0; + for (const root of SCAN_ROOTS) { + const abs = path.join(ROOT, root); + if (!fs.existsSync(abs)) continue; + for (const file of walk(abs)) { + const src = fs.readFileSync(file, 'utf-8'); + for (const line of src.split('\n')) if (SYNC_SPAWN.test(line)) total += 1; + } + } + expect(total).toBeGreaterThan(100); + }); +}); diff --git a/test/strict-output.test.ts b/test/strict-output.test.ts index 832283081..a059dd9d5 100644 --- a/test/strict-output.test.ts +++ b/test/strict-output.test.ts @@ -89,6 +89,38 @@ describe('BunTestOutputClassifier', () => { expect(summary.terminalFileCounts).toEqual([2]); expect(strictTestExitCode(0, summary, 2)).toBe(0); }); + + // "Ran N tests" COUNTS skipped tests, so the skip/pass count lines are the + // only stream evidence separating verified work from green-by-skip (a + // codex/gemini file whose every test self-skips on a binary-less runner). + it('parses the skip and pass count lines from bun’s recap block', () => { + const c = new BunTestOutputClassifier(); + c.write(' 1 pass\n 2 skip\n 0 fail\nRan 3 tests across 1 file. [7.00ms]\n'); + const summary = c.end(); + expect(summary.passedTests).toBe(1); + expect(summary.skippedTests).toBe(2); + expect(summary.terminalTestCounts).toEqual([3]); + // all-skipped is still exit-0 at the classifier layer — the census + // labeling happens in the paid runner, not here + expect(strictTestExitCode(0, summary, 1)).toBe(0); + }); + + it('skip/pass counts survive ANSI color and chunk shears', () => { + const c = new BunTestOutputClassifier(); + c.write(' 4 pa'); + c.write('ss\n 9 skip\n'); + const summary = c.end(); + expect(summary.passedTests).toBe(4); + expect(summary.skippedTests).toBe(9); + }); + + it('prose mentioning skip counts does not pollute the tally', () => { + const c = new BunTestOutputClassifier(); + c.write('console.log said: 7 skip is what we expect later\n'); + c.write('(fail) 3 skip handling [1.00ms]\n'); + const summary = c.end(); + expect(summary.skippedTests).toBe(0); + }); }); describe('installChildSignalForwarding — cancellation terminates the RUN', () => { diff --git a/test/tasks-section-jq.test.ts b/test/tasks-section-jq.test.ts index 78f8de9cf..a3f61b593 100644 --- a/test/tasks-section-jq.test.ts +++ b/test/tasks-section-jq.test.ts @@ -41,7 +41,7 @@ function runJq(program: string, inputLines: string[], branch: string, commits: s const out = execFileSync( "jq", ["-c", "--arg", "branch", branch, "--arg", "commits", commits, program], - { input: inputLines.join("\n"), encoding: "utf-8" }, + { input: inputLines.join("\n"), encoding: "utf-8", timeout: 30_000 }, ); return out.split("\n").filter(Boolean); } diff --git a/test/taste-engine.test.ts b/test/taste-engine.test.ts index e92a69da7..5a4c6c835 100644 --- a/test/taste-engine.test.ts +++ b/test/taste-engine.test.ts @@ -40,7 +40,7 @@ beforeEach(() => { stateDir = fs.mkdtempSync(path.join(os.tmpdir(), 'taste-state-')); workdir = fs.mkdtempSync(path.join(os.tmpdir(), 'taste-work-')); // Initialize a git repo so gstack-taste-update's getSlug() finds a toplevel - spawnSync('git', ['init', '-b', 'main'], { cwd: workdir, stdio: 'pipe' }); + spawnSync('git', ['init', '-b', 'main'], { cwd: workdir, stdio: 'pipe', timeout: 30_000 }); }); afterEach(() => { diff --git a/test/team-mode.test.ts b/test/team-mode.test.ts index ce8c1d610..87187f038 100644 --- a/test/team-mode.test.ts +++ b/test/team-mode.test.ts @@ -133,8 +133,8 @@ describe('gstack-session-update', () => { fs.mkdirSync(stateDir, { recursive: true }); // Init a git repo to pass the .git guard - execSync('git init', { cwd: gstackDir }); - execSync('git commit --allow-empty -m "init"', { cwd: gstackDir }); + execSync('git init', { cwd: gstackDir, timeout: 30_000 }); + execSync('git commit --allow-empty -m "init"', { cwd: gstackDir, timeout: 30_000 }); fs.writeFileSync(path.join(gstackDir, 'VERSION'), '0.1.0'); // Create a minimal gstack-config that returns auto_upgrade=true @@ -191,8 +191,8 @@ describe('gstack-team-init', () => { beforeEach(() => { tmpDir = mkTmpDir(); - execSync('git init', { cwd: tmpDir }); - execSync('git commit --allow-empty -m "init"', { cwd: tmpDir }); + execSync('git init', { cwd: tmpDir, timeout: 30_000 }); + execSync('git commit --allow-empty -m "init"', { cwd: tmpDir, timeout: 30_000 }); }); afterEach(() => { @@ -265,8 +265,8 @@ describe('gstack-team-init', () => { fs.writeFileSync(path.join(vendoredDir, 'VERSION'), '0.14.0.0'); fs.writeFileSync(path.join(vendoredDir, 'README.md'), 'vendored'); // Track it in git - execSync('git add .claude/skills/gstack/', { cwd: tmpDir }); - execSync('git commit -m "add vendored gstack"', { cwd: tmpDir }); + execSync('git add .claude/skills/gstack/', { cwd: tmpDir, timeout: 30_000 }); + execSync('git commit -m "add vendored gstack"', { cwd: tmpDir, timeout: 30_000 }); const result = run(`${TEAM_INIT} optional`, { cwd: tmpDir }); expect(result.exitCode).toBe(0); @@ -306,8 +306,8 @@ describe('gstack-team-init', () => { const vendoredDir = path.join(tmpDir, '.claude', 'skills', 'gstack'); fs.mkdirSync(vendoredDir, { recursive: true }); fs.writeFileSync(path.join(vendoredDir, 'VERSION'), '0.14.0.0'); - execSync('git add .claude/skills/gstack/', { cwd: tmpDir }); - execSync('git commit -m "add vendored"', { cwd: tmpDir }); + execSync('git add .claude/skills/gstack/', { cwd: tmpDir, timeout: 30_000 }); + execSync('git commit -m "add vendored"', { cwd: tmpDir, timeout: 30_000 }); run(`${TEAM_INIT} optional`, { cwd: tmpDir }); diff --git a/test/telemetry-repo-strip.test.ts b/test/telemetry-repo-strip.test.ts index 201cbc966..0d6bc2417 100644 --- a/test/telemetry-repo-strip.test.ts +++ b/test/telemetry-repo-strip.test.ts @@ -166,6 +166,7 @@ describe('telemetry no-repo-identity-egress invariant', () => { } const out = spawnSync(['sed', ...sedArgs], { stdin: Buffer.from(sample), + timeout: 30_000, }); const cleaned = out.stdout.toString(); @@ -198,7 +199,7 @@ describe('telemetry no-repo-identity-egress invariant', () => { expect(anonymous).toBeTruthy(); const runJq = (filter: string, input: string) => { - const out = spawnSync(['jq', '-c', filter], { stdin: Buffer.from(input) }); + const out = spawnSync(['jq', '-c', filter], { stdin: Buffer.from(input), timeout: 30_000 }); return { exitCode: out.exitCode, stdout: out.stdout.toString().trim() }; }; diff --git a/test/test-free-shards.test.ts b/test/test-free-shards.test.ts index 0e7daa157..c807c9b0e 100644 --- a/test/test-free-shards.test.ts +++ b/test/test-free-shards.test.ts @@ -83,6 +83,7 @@ describe('test-free-shards: Windows curation', () => { }); test('detects spawn("sh", ...)', () => { + // tripwire-exempt: string fixture fed to detectWindowsFragility, not a call withTempFile(`spawnSync('sh', ['-c', 'command -v claude']);`, (f) => { expect(detectWindowsFragility(f)?.reason).toBe('spawn("sh", ...)'); }); @@ -95,6 +96,7 @@ describe('test-free-shards: Windows curation', () => { }); test('detects which claude shell command', () => { + // tripwire-exempt: string fixture fed to detectWindowsFragility, not a call withTempFile(`execSync('which claude').trim();`, (f) => { expect(detectWindowsFragility(f)?.reason).toBe('which claude (use Bun.which)'); }); diff --git a/test/timeline.test.ts b/test/timeline.test.ts index 6d30a939e..a7cfa3f13 100644 --- a/test/timeline.test.ts +++ b/test/timeline.test.ts @@ -50,7 +50,7 @@ function runReadArgs(args: string[] = []): string { timeout: 15000, }; try { - return execFileSync(path.join(BIN, 'gstack-timeline-read'), args, execOpts).trim(); + return execFileSync(path.join(BIN, 'gstack-timeline-read'), args, execOpts).trim(); // timeout via execOpts } catch { return ''; } diff --git a/test/touchfiles.test.ts b/test/touchfiles.test.ts index 6bf5fb25c..b920bd889 100644 --- a/test/touchfiles.test.ts +++ b/test/touchfiles.test.ts @@ -81,14 +81,12 @@ describe('selectTests', () => { expect(result.selected).toContain('plan-ceo-review-selective'); expect(result.selected).toContain('plan-ceo-review-benefits'); expect(result.selected).toContain('plan-ceo-review-expansion-energy'); - expect(result.selected).toContain('autoplan-core'); expect(result.selected).toContain('codex-offered-ceo-review'); expect(result.selected).toContain('plan-ceo-review-format-mode'); expect(result.selected).toContain('plan-ceo-review-format-approach'); // v1.10.2.0 plan-mode handshake entries also depend on plan-ceo-review/** expect(result.selected).toContain('plan-ceo-review-plan-mode'); expect(result.selected).toContain('plan-mode-no-op'); - expect(result.selected).toContain('e2e-harness-audit'); expect(result.selected).toContain('plan-ceo-review-prosons-cadence'); expect(result.selected).toContain('plan-review-prosons-format'); expect(result.selected).toContain('plan-review-prosons-hardstop-neg'); @@ -111,12 +109,8 @@ describe('selectTests', () => { expect(result.selected).toContain('plan-ceo-split-overflow'); // v2 plan Phase B carve: the section-loading E2E depends on plan-ceo-review/**. expect(result.selected).toContain('plan-ceo-section-loading'); - // Token-reduction Phase 1: the preamble script A/B also keys on plan-ceo-review/**. - expect(result.selected).toContain('preamble-script-ab'); - // AUQ repetition-cut NOT-WORSE gate drives plan-ceo-review, so it keys on it too. - expect(result.selected).toContain('auq-repetition-cut-ab'); - expect(result.selected.length).toBe(25); - expect(result.skipped.length).toBe(Object.keys(E2E_TOUCHFILES).length - 25); + expect(result.selected.length).toBe(21); + expect(result.skipped.length).toBe(Object.keys(E2E_TOUCHFILES).length - 21); }); test('global touchfile triggers ALL tests', () => { @@ -388,3 +382,65 @@ describe('touchfile dependency paths exist', () => { } }); }); + +// --- Reverse invariant: every selection key names a LIVING test --- +// The forward invariants above catch stale dep PATHS; nothing caught stale +// KEYS. The 2026-08 audit found 15 phantom E2E keys (6 gate-tier) selecting +// tests that existed nowhere — the merge-blocking census counted work that +// could not run. A key earns its place by appearing as a quoted testName in +// a living paid test file; constructed names get a reasoned exception. + +describe('reverse invariant — keys must name living paid tests', () => { + const { isPaidTestFile } = require('./helpers/paid-test-set') as typeof import('./helpers/paid-test-set'); + + /** Keys whose testNames are CONSTRUCTED at runtime (template literals), so + * a quoted-occurrence scan cannot see them. Each entry needs the file that + * constructs it. Shrink-only: prefer literal names in new tests. */ + const CONSTRUCTED_NAME_EXCEPTIONS: Record = {}; + + const paidSources: string[] = []; + for (const name of fs.readdirSync(path.join(ROOT, 'test'))) { + const rel = `test/${name}`; + if (!isPaidTestFile(rel)) continue; + paidSources.push(fs.readFileSync(path.join(ROOT, rel), 'utf-8')); + } + + const quotedSomewhere = (key: string): boolean => + paidSources.some((src) => + src.includes(`'${key}'`) || src.includes(`"${key}"`) || src.includes('`' + key + '`')); + + /** Clause (b) of liveness: constructed testNames (template literals) bind + * through SELF-REGISTRATION — the 2026-08 dep-list sweep put each test + * FILE into its key's dep list, and the parent mapper keeps a shard on + * that registration union. So a key is alive when its name is quoted in a + * paid file OR its dep list names an existing paid test file. */ + const registeredToLivingFile = (key: string): boolean => + (E2E_TOUCHFILES[key] ?? []).some((dep) => + /\.test\.ts$/.test(dep) && isPaidTestFile(dep) && fs.existsSync(path.join(ROOT, dep))); + + test('every E2E_TOUCHFILES key is declared in a living paid test file', () => { + expect(paidSources.length).toBeGreaterThan(50); // scan-rot guard + const phantoms = Object.keys(E2E_TOUCHFILES) + .filter((key) => !(key in CONSTRUCTED_NAME_EXCEPTIONS)) + .filter((key) => !quotedSomewhere(key) && !registeredToLivingFile(key)); + expect( + phantoms, + `E2E_TOUCHFILES key(s) with NO declaring paid test — the census counts tests that cannot run. ` + + `Delete the key (both maps) or implement the test:\n ${phantoms.join('\n ')}`, + ).toEqual([]); + }); + + test('every LLM_JUDGE_TOUCHFILES key is declared in a living paid test file', () => { + const phantoms = Object.keys(LLM_JUDGE_TOUCHFILES).filter((key) => !quotedSomewhere(key)); + expect( + phantoms, + `LLM_JUDGE_TOUCHFILES key(s) with NO declaring test:\n ${phantoms.join('\n ')}`, + ).toEqual([]); + }); + + test('constructed-name exceptions stay live (files exist and construct them)', () => { + const stale = Object.entries(CONSTRUCTED_NAME_EXCEPTIONS) + .filter(([, file]) => !fs.existsSync(path.join(ROOT, file))); + expect(stale.map(([k]) => k), 'exception points at a deleted file — remove the entry').toEqual([]); + }); +}); diff --git a/test/tracker-guard-wiring.test.ts b/test/tracker-guard-wiring.test.ts index b4daaab5c..ccf8bad55 100644 --- a/test/tracker-guard-wiring.test.ts +++ b/test/tracker-guard-wiring.test.ts @@ -61,7 +61,7 @@ const SCANNER_EXEMPT: { file: string; pattern: string; reason: string }[] = [ ]; function trackedFiles(): string[] { - const out = execSync('git ls-files', { cwd: ROOT, encoding: 'utf-8', maxBuffer: 32 * 1024 * 1024 }); + const out = execSync('git ls-files', { cwd: ROOT, encoding: 'utf-8', maxBuffer: 32 * 1024 * 1024, timeout: 30_000 }); return out .split('\n') .map((s) => s.trim()) diff --git a/test/uninstall.test.ts b/test/uninstall.test.ts index 96e2e689e..1590c4894 100644 --- a/test/uninstall.test.ts +++ b/test/uninstall.test.ts @@ -9,12 +9,12 @@ const UNINSTALL = path.join(ROOT, 'bin', 'gstack-uninstall'); describe('gstack-uninstall', () => { test('syntax check passes', () => { - const result = spawnSync('bash', ['-n', UNINSTALL], { stdio: 'pipe' }); + const result = spawnSync('bash', ['-n', UNINSTALL], { stdio: 'pipe', timeout: 30_000 }); expect(result.status).toBe(0); }); test('--help prints usage and exits 0', () => { - const result = spawnSync('bash', [UNINSTALL, '--help'], { stdio: 'pipe' }); + const result = spawnSync('bash', [UNINSTALL, '--help'], { stdio: 'pipe', timeout: 30_000 }); expect(result.status).toBe(0); const output = result.stdout.toString(); expect(output).toContain('gstack-uninstall'); @@ -25,6 +25,7 @@ describe('gstack-uninstall', () => { test('unknown flag exits with error', () => { const result = spawnSync('bash', [UNINSTALL, '--bogus'], { stdio: 'pipe', + timeout: 30_000, env: { ...process.env, HOME: '/nonexistent' }, }); expect(result.status).toBe(1); @@ -58,7 +59,7 @@ describe('gstack-uninstall', () => { // Create mock git repo fs.mkdirSync(mockGitRoot, { recursive: true }); - spawnSync('git', ['init', '-b', 'main'], { cwd: mockGitRoot, stdio: 'pipe' }); + spawnSync('git', ['init', '-b', 'main'], { cwd: mockGitRoot, stdio: 'pipe', timeout: 30_000 }); }); afterEach(() => { @@ -68,6 +69,7 @@ describe('gstack-uninstall', () => { test('--force removes global Claude skills and state', () => { const result = spawnSync('bash', [UNINSTALL, '--force'], { stdio: 'pipe', + timeout: 30_000, env: { ...process.env, HOME: mockHome, @@ -98,6 +100,7 @@ describe('gstack-uninstall', () => { test('--keep-state preserves state directory', () => { const result = spawnSync('bash', [UNINSTALL, '--force', '--keep-state'], { stdio: 'pipe', + timeout: 30_000, env: { ...process.env, HOME: mockHome, @@ -129,6 +132,7 @@ describe('gstack-uninstall', () => { GSTACK_DIR: path.join(cleanHome, 'nonexistent'), GSTACK_STATE_DIR: path.join(cleanHome, '.gstack'), }, + timeout: 30_000, cwd: mockGitRoot, }); @@ -143,6 +147,7 @@ describe('gstack-uninstall', () => { const result = spawnSync('bash', [UNINSTALL, '--force'], { stdio: 'pipe', + timeout: 30_000, env: { ...process.env, HOME: mockHome, @@ -180,6 +185,7 @@ describe('gstack-uninstall', () => { const result = spawnSync('bash', [UNINSTALL, '--force'], { stdio: 'pipe', + timeout: 30_000, env: { ...process.env, HOME: mockHome, @@ -212,6 +218,7 @@ describe('gstack-uninstall', () => { const result = spawnSync('bash', [UNINSTALL, '--force'], { stdio: 'pipe', + timeout: 30_000, env: { ...process.env, HOME: mockHome, @@ -275,6 +282,7 @@ describe('hook cleanup runs before the install root is deleted', () => { const result = spawnSync('bash', [path.join(installBin, 'gstack-uninstall'), '--force', '--keep-state'], { stdio: 'pipe', + timeout: 30_000, env: { ...process.env, HOME: mockHome, @@ -331,6 +339,7 @@ describe('hook cleanup under lock contention is loud, never silent (review-army) const result = spawnSync('bash', [path.join(installBin, 'gstack-uninstall'), '--force', '--keep-state'], { stdio: 'pipe', + timeout: 30_000, env: { ...process.env, HOME: mockHome, diff --git a/test/upgrade-migration-v1.test.ts b/test/upgrade-migration-v1.test.ts index 09fdaf2c2..c135ceaf4 100644 --- a/test/upgrade-migration-v1.test.ts +++ b/test/upgrade-migration-v1.test.ts @@ -33,6 +33,7 @@ function run(): { stdout: string; stderr: string; status: number } { const res = spawnSync('bash', [MIGRATION], { encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: tmpHome, HOME: tmpHome }, + timeout: 30_000, }); return { stdout: (res.stdout ?? '').trim(), diff --git a/test/user-render-out-dir-install.test.ts b/test/user-render-out-dir-install.test.ts index 4c0980d23..02e20d79a 100644 --- a/test/user-render-out-dir-install.test.ts +++ b/test/user-render-out-dir-install.test.ts @@ -193,7 +193,7 @@ describe('link_claude_skill_dirs prefers rendered SKILL.md (behavior)', () => { describe('migration v1.67.0.0 — legacy in-place render cleanup (F12)', () => { function git(cwd: string, ...args: string[]): void { - const r = spawnSync('git', args, { cwd, encoding: 'utf-8' }); + const r = spawnSync('git', args, { cwd, encoding: 'utf-8', timeout: 30_000 }); if (r.status !== 0) throw new Error(`git ${args.join(' ')} failed: ${r.stderr}`); } diff --git a/test/worktree.test.ts b/test/worktree.test.ts index 47a58d236..aabe2f458 100644 --- a/test/worktree.test.ts +++ b/test/worktree.test.ts @@ -16,9 +16,9 @@ import * as os from 'os'; /** Create a minimal git repo in a tmpdir for testing. */ function createTestRepo(): string { const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'worktree-test-')); - spawnSync('git', ['init'], { cwd: dir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: dir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: dir, stdio: 'pipe' }); + spawnSync('git', ['init'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['config', 'user.name', 'Test'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); // Create initial commit so HEAD exists fs.writeFileSync(path.join(dir, 'README.md'), '# Test repo\n'); @@ -31,8 +31,8 @@ function createTestRepo(): string { fs.mkdirSync(path.join(dir, 'browse', 'dist'), { recursive: true }); fs.writeFileSync(path.join(dir, 'browse', 'dist', 'browse'), '#!/bin/sh\necho browse\n'); - spawnSync('git', ['add', 'README.md', '.gitignore'], { cwd: dir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'Initial commit'], { cwd: dir, stdio: 'pipe' }); + spawnSync('git', ['add', 'README.md', '.gitignore'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['commit', '-m', 'Initial commit'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); return dir; } @@ -40,7 +40,7 @@ function createTestRepo(): string { /** Clean up a test repo. */ function cleanupRepo(dir: string): void { // Prune worktrees first to avoid git lock issues - spawnSync('git', ['worktree', 'prune'], { cwd: dir, stdio: 'pipe' }); + spawnSync('git', ['worktree', 'prune'], { cwd: dir, stdio: 'pipe', timeout: 30_000 }); fs.rmSync(dir, { recursive: true, force: true }); } @@ -94,7 +94,7 @@ describe('WorktreeManager', () => { repos.push(repo); const mgr = new WorktreeManager(repo); - const expectedSha = spawnSync('git', ['rev-parse', 'HEAD'], { cwd: repo, stdio: 'pipe' }) + const expectedSha = spawnSync('git', ['rev-parse', 'HEAD'], { cwd: repo, stdio: 'pipe', timeout: 30_000 }) .stdout.toString().trim(); mgr.create('test-sha'); @@ -154,8 +154,8 @@ describe('WorktreeManager', () => { // Make a commit in the worktree (simulating agent running git commit) fs.writeFileSync(path.join(worktreePath, 'committed.txt'), 'Agent committed this\n'); - spawnSync('git', ['add', 'committed.txt'], { cwd: worktreePath, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'Agent commit'], { cwd: worktreePath, stdio: 'pipe' }); + spawnSync('git', ['add', 'committed.txt'], { cwd: worktreePath, stdio: 'pipe', timeout: 30_000 }); + spawnSync('git', ['commit', '-m', 'Agent commit'], { cwd: worktreePath, stdio: 'pipe', timeout: 30_000 }); const result = mgr.harvest('test-harvest-commit'); @@ -228,7 +228,7 @@ describe('WorktreeManager', () => { expect(fs.existsSync(oldPath)).toBe(true); // Remove via git but leave directory (simulating a crash) - spawnSync('git', ['worktree', 'remove', '--force', oldPath], { cwd: repo, stdio: 'pipe' }); + spawnSync('git', ['worktree', 'remove', '--force', oldPath], { cwd: repo, stdio: 'pipe', timeout: 30_000 }); // Recreate the directory to simulate orphaned state fs.mkdirSync(oldPath, { recursive: true }); // Backdate mtime to simulate a stale worktree (> 1 hour old)