CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-09 06:28:59 +02:00
Code Issues Packages Projects Releases Wiki Activity

v1.77.0.0 feat: test-infrastructure overhaul wave 1 — matrix deletion, flake telemetry, sync-spawn wedge class extinct (#2746)

Browse Source 
* fix: pin the claude CLI to an exact version in the CI image + tripwire

The image installed @anthropic-ai/claude-code UNPINNED and rebuilt weekly
'to pick up CLI updates' — while bun sat carefully pinned at 1.3.13 two RUN
lines above. The PTY harness screen-scrapes this CLI's TUI, and that drift
broke it three separate times (welcome-screen wedge on 2.1.233, skillify
HOME discovery on 2.1.237, guard/freeze hooks on 2.1.162), each debugged as
a flake first. Pin 2.1.251 (current latest), bump deliberately via a PR
that runs the PTY gate, and enforce with test/ci-image-cli-pin.test.ts:
any global npm install in Dockerfile.ci without an exact @X.Y.Z pin fails
the free suite. The weekly ci-image cron stays as a cheap tag self-heal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: stamp the claude CLI version into every eval-store run record

Three harness breakages were traced to claude-CLI TUI drift only after long
flake hunts, because no run record said which CLI it actually exercised.
EvalCollector now stamps claude_cli_version (claude --version, cached once
per process, 'unknown' when the binary is absent) into both partial and
finalized records — schema-additive optional field, no SCHEMA_VERSION bump.
Correlating a flake wave with a CLI release becomes a grep over
~/.gstack/projects/<slug>/evals/ instead of archaeology.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: give the spinning-shard kill test load headroom (30s -> 90s)

The test spawns and group-kills three real children (one a busy-loop
burning a full core) while five sibling shard processes compete for eight
vCPUs. Under full-suite load it blew bun's default 30s per-test ceiling at
30,009ms — while passing in isolation in 1.4s — and red the only required
lane. Every assertion in it is event-based (statuses, group-kill proof,
heartbeat lines); the sole latency claim is the <30s kill-deadline sanity
bound, which stays. Explicit 90s headroom, not a weakened oracle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: green-by-skip census — skip counts in the classifier, all-skipped labeling in the paid runner

bun's 'Ran N tests' line COUNTS skipped tests, so a codex/gemini shard
whose every test self-skipped (binary absent on the runner — true of every
CI runner today) exits 0, dodges the hollow-shard guard, and reads as
coverage in the weekly census. The classifier now parses bun's ' N skip' /
' N pass' recap lines; ShardOutcome carries skippedTests; formatSummary and
the fail-closed slices report label an all-skipped pass explicitly:
'all N tests SKIPPED — verified nothing'. Status stays 'passed' (external
service availability is host state, not a repo regression) but the census
can no longer mistake absence for coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: extract composite actions for eval-lane setup; surviving lanes gain the fail-fast registry verification

'Fix bun temp' x3, 'Restore deps' x5, 'Seed claude interactive config' x3,
and 'Register gstack skills' x3 were byte-near-identical copies across the
legacy matrix, the sliced lane, and the periodic lane — and only the MATRIX
copy of register-skills carried the 19-line dangling-symlink + frontmatter
fail-fast loop written after a silent 'Unknown command' + 35-min-timeout
incident. Extract all four into .github/actions/ composites; the register
composite carries the verification loop (generalized over the skill list),
so the sliced and periodic lanes — the lanes that SURVIVE the matrix
deletion — now inherit the check they had silently dropped. Matrix-job
inline copies are left untouched: that job is deleted next.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: delete the legacy 17-row eval matrix — the sliced lane is the only paid lane

Every PR paid twice: the hand-enumerated matrix (18 test files, 22.6 min,
~$21 API measured on run 33263204465) ran serialized AHEAD of the strictly
superior sliced lane via 'needs: evals' — 35.5 min wall and ~2x paid spend
for the same diff. 14 of 17 rows carried no tier:, so periodic Opus
benchmarks leaked into every PR (the e2e-plan row alone: 12/12 tests,
21.7 min, $7.28 — the wall-clock bound of ALL of CI).

Parity receipt (static, pre-deletion): the sliced lane's gate census (49
files, derived from the runner itself) strictly contains all 18 matrix test
files, plus 31 files the matrix never ran. Pure deletion — one revert
restores it. The PR comment moved into slices-report (same '## E2E Evals'
upsert marker, now sourced from slice artifacts + carrying the fail-closed
reconciliation verdict). plan-slices loses the needs edge; the dead
workflow-level EVALS_TIER env goes with it.

test/evals-workflow-matrix.test.ts (and its KNOWN_MATRIX_GAPS /
KNOWN_TIER_UNSET burn-down ratchets — retired: the sliced census makes
'every gate file runs' true by construction) is rewritten as
test/evals-workflow-wiring.test.ts: matrix stays deleted, planner/executor/
report tier + slice-count agreement, both surviving lanes on the shared
register-skills composite with its fail-fast verification loop, PR comment
survival. Expected: PR eval wall 35.5 -> ~13 min, per-PR paid spend ~halved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: provider-runner timeouts kill the whole process GROUP; codex/gemini inherit the orphan-drain hardening

All three provider runners (claude/codex/gemini) killed only the direct
child on timeout: tool subprocesses the CLI spawned survived as orphans
holding our pipes open and burning shared API rate (observed: a 600s
timeout stretching past 1400s; a stalled run once burned a core for 15
hours). gstack-detach's watchdog had the same shape one level up — killpg
SIGTERM, 5s grace, then a direct-child proc.kill() that orphaned
grandchildren.

Fix: spawn provider children via node:child_process with detached (own
process group) and killProcessGroup(SIGKILL) in the timeout handler —
runShardChild's proven pattern, EPERM/ESRCH fallbacks included. The codex
and gemini copies also gain the reader.cancel() + stderr Promise.race
hardening only the claude copy had (they still carried the blocked-drain
hang it fixed). gstack-detach's watchdog now group-SIGKILLs after the
grace.

Regression net: test/session-runner-groupkill.test.ts drives the REAL
runSkillTest against a fake claude shim (PATH override) that spawns a
grandchild and wedges — the run must classify timeout within budget and
leave neither shim nor grandchild alive — plus source pins on all three
runners (detached + killProcessGroup, no bare timeout kill, no Bun.spawn
reversion).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: skill-e2e-opus-47 renders SKILL.md fixtures into a mkdtemp — never the live tree

mkEvalRoot ran gen-skill-docs with cwd=ROOT, regenerating every in-repo
SKILL.md mid-run while concurrent paid shards copyFileSync those same files
in their beforeAll (EVALS_JOBS>=4 locally, 2 per CI slice) — a sibling
could capture a half-regenerated or opus-rendered SKILL.md, and a timeout
before afterAll stranded the whole tree at the wrong model for every later
shard. A cross-shard race that could flake ANY concurrent paid test.

Render via the --out-dir flag gen-skill-docs grew for exactly this reason
(mirrors the repo layout, which is all the fixture reads), read the skill
heads from the render dir, delete it, and drop the afterAll restore-regen
entirely.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: claude CLI version resolves in the runner parent, never on a test thread

Eng-review finding: getClaudeCliVersion's fallback is a SYNCHRONOUS
spawnSync on the same thread that polls concurrent PTY/session tests — the
judgePtyState blocking class this overhaul kills elsewhere. The paid runner
parent now resolves it once (cached) and stamps GSTACK_CLAUDE_CLI_VERSION
into every shard's env; eval-store short-circuits on the env var, and the
fallback spawn's budget tightens 10s -> 3s (bounded one-time stall, records
'unknown' on a slow CLI).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: wire skippedTests end-to-end through runPaidShard

The census unit tests hand-built outcomes and the classifier tests parsed
strings; nothing proved a real child's ' N skip' recap flows into
outcome.skippedTests and the formatSummary label. A commandFor fake now
prints the recap shape and the test asserts the parsed counts, the
all-skipped predicate, and the 'verified nothing' label.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: make the setup composites rerun-safe (codex diff-review hardenings)

restore-deps: 'cp -r SRC node_modules' with an existing node_modules NESTS
the copy and leaves stale deps active — rm first. register-gstack-skills:
'ln -snf' hard-errors under set -eu when a REAL directory occupies the
gstack slot — clear a non-symlink leftover first. CI workspaces are fresh
today; a reusable composite must survive dirty reruns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: sweep — every sync spawn in the test trees carries a timeout (436 sites, 157 files)

spawnSync/execSync/Bun.spawnSync BLOCK the main thread, so bun's in-process
per-test timeout can never fire while one waits — a hung child (stdin read,
network probe, dead daemon) wedges the whole shard until the runner's
external wall-clock SIGKILL. This exact class reached main: free-tests run
33262077256, test/gstack-memory-ingest.test.ts (normally 2.3s) held shard 2
at the 360s wall while its five siblings finished in ~65s.

Mechanical sweep in two waves (12 + 4 fan-out agents, every edit verified
against its call site): default timeout: 30_000 (matches the free runner's
per-test budget), 120_000 for genuinely slow ops (installs, builds,
playwright, provider CLIs), helper wrappers fixed ONCE where call sites
route through them. Sites that only LOOK like calls (string fixtures, grep
needles, comments) were skipped with reasons — the enforcement commit that
follows marks them exempt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: sync-spawn timeout tripwire — the wedge class stays extinct

Free scanner over all test trees (test/, browse/test/, design/test/,
make-pdf/test/, ios-qa, browser-skills): every spawnSync/execSync/
Bun.spawnSync call site must carry a timeout within a 30-line options
window, or an explicit '// tripwire-exempt: <reason>' marker. Comment
lines are skipped; exemptions are counted and ratcheted shrink-only
(ceiling 6 = the 6 string-fixture/grep-needle sites where the pattern is
CONTENT, not a call — marked in this commit). A scan-sanity test pins that
the scanner still sees >100 real call sites so it can never rot to a
vacuous green. Companion to the 436-site sweep in the previous commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: paid-lane flake telemetry — record-level attempts, flaky_retries, report surfacing

bun --retry leaves a retried pass INVISIBLE in its output: a fail-then-pass
prints the error detail but no (fail) result line and recaps as a clean
pass (probed live on 1.3.10). So attempts are recorded where they cannot
lie: EvalCollector.addTest stamps a 1-based attempt on same-name re-records
(a retried test runs its body again and re-records), finalized runs carry
flaky_retries, printSummary warns loudly, and the fail-closed slices report
lists every passed-only-on-retry test — recorded and ranked, never blocking
and never silent. Cross-model confirmed (codex reached the same don't-parse
-the-stream conclusion independently).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: free-lane flake ledger — retry ON in CI, flaky-passes recorded and uploaded

The runner's attribution-gated flaky-retry pass (cap 5, truncation veto)
was OFF in the required lane and its FLAKY-PASS evidence was console-only —
so a single timing flake red the merge gate while repeat offenders stayed
unenumerable. free-tests.yml now sets GSTACK_FREE_RETRY_FLAKY=1 and points
GSTACK_FLAKE_LEDGER at runner.temp; every flaky-pass appends a JSONL entry
(SINGLE writer: the parent runner — no concurrent-append hazard by
construction; fail-open with a loud warning so a broken ledger can never
red the lane) and the artifact uploads UNCONDITIONALLY — a flaky-pass run
is green, which is exactly when the evidence matters. Wiring pinned by
free-tests-workflow-wiring; ledger behavior unit-tested incl. the fail-open
path. Matches 2026 industry practice (retry for data, quarantine out of
merge-blocking but never out of logging) with the repo's own receipts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: eval:flake-rank — the flake-telemetry dial

Aggregates per-test series across every finalized eval-store run (shard
dirs included) plus the free flake ledger: runs, fails, RETRIED PASSES
(the flake signature), avg duration — ranked retries-first. This is the
readable dial behind two policies: a flaky pass never blocks a merge but
is always ranked here, and the WS16 required-check promotion needs weeks
of clean flake-rank, not vibes. --json for machines, --dir for downloaded
CI artifacts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: two-phase session timeout — silent APIs die at the startup grace, named

The single spawn-armed timer charged API queue latency to the work budget:
the recurring '0 turns / $0.00 / x3 attempts' failure with four budget-bump
receipts (180->300s, 240->360s, 300->420s, 90->300s). Split: startup phase
(no NDJSON byte yet) kills EARLY at min(grace, timeout) with the distinct
exitReason 'timeout_startup' — an availability verdict, not transcript
archaeology — and the work phase arms on the first byte for the REMAINING
budget, so total wall never exceeds the timeout (tier envelopes are
margin-free: tests pass timeout: CAPTURE_MS and bun-budget the same tier).
Local grace 90s (observed queue latency 60-90s), CI floor 300s (TODOS-filed;
shared runners queue harder), both pinned by the new grace tests with fake
-claude shims covering the late-first-byte and silent-API paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: census integrity — 17 phantom selection keys deleted, reverse invariant added, gitignored dep patterns replaced, local map forks derived

The merge-blocking gate census counted tests that could not run. Deleted
(critic-verified against both quoted-occurrence and dep-registration
liveness): 7 *-prosons-format keys with no declaring test, ship-plan-
completion/-verification, review-plan-completion, design-shotgun-path/
session/full, autoplan-core (dead ~10 months), e2e-harness-audit (its
namesake is a FREE-suite file), plus 2 dead LLM-judge keys and 2 free-file
keys (budget-regression-pty, global-discover) misplaced in the PAID maps.
Census: 191 -> 174 keys, gate 86 -> 78 honest.

The new reverse invariant in touchfiles.test.ts makes the class structurally
impossible: every key must be quoted in a living paid test file OR
registered to an existing paid test file via its dep list (the constructed-
name binding the 2026-08 self-registration sweep established) — zero
exceptions needed today, with a live-file check on any future exception.

Also: '.agents/skills/**' dep patterns replaced with the generator
(scripts/gen-skill-docs.ts) — .agents/ is gitignored, so those patterns
could NEVER match a git diff and review-template edits silently stopped
selecting codex/gemini tests; the codex/gemini local touchfile maps are now
DERIVED from the canonical map (loud throw if a key vanishes) instead of
hand-forked copies that had already drifted. ios-qa-e2e demoted gate ->
periodic: its gate declaration was never executable in CI (hardware
exclusion only applies at tier=periodic), so every Linux PR planned a
hollow shard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: routing journeys lose their answer key and end at the routing decision

The journey tests exist to catch skill-DESCRIPTION regressions (touchfiles:
*/SKILL.md.tmpl), but the fixture CLAUDE.md shipped an explicit
prompt->skill lookup table — with the answer key in context, a badly
regressed frontmatter description still routed correctly, so the tests
could not fail on the exact class they select for. The fixture now carries
only the generic invoke-skills nudge; the frontmatter carries the routing
load. Also capped all 10 journeys at maxTurns 2 / tools [Skill, Read]:
only the FIRST Skill call is asserted, so 5 turns of Read/Bash/Glob/Grep
was pure spend — roughly halves each journey's cost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: retire decided A/B experiments; vendor the pre-cut fixture; ban raw-SHA fixtures

Three one-shot decision experiments kept re-running weekly as N=1
stochastic comparisons — flaky by construction with near-zero remaining
information: skill-e2e-auq-repetition-cut-ab (its own header: gate "passed
pre-landing, approved 2026-08-25"), skill-e2e-preamble-script-ab ("demoted
post-Phase-3"), and opus-47's fanout arm-vs-arm (parA >= parB across two
SINGLE stochastic runs — a coin flip). Deleted, with their selection keys;
the SDK overlay-harness stays as the maintained instrument for the next
experiment, and opus-47 keeps its routing-precision cases.

verboseSkill() now reads the VENDORED test/fixtures/auq-pre-cut-...-SKILL.md
instead of `git show ab66193e^:...` — a branch-local ref that dies on
branch prune and already failed on shallow clones. New free tripwire
(test/git-ref-fixture-tripwire.test.ts) bans the raw-SHA fixture class
outright: quoted SHA:path rev-specs and gitRef-style hex defaults in the
test trees fail the suite with the vendor-instead instruction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: demote plan-ceo-review-expansion-energy to periodic

Opus generator + a subjective 2-axis >=4/5 LLM-judge threshold sat in the
MERGE-BLOCKING gate — the exact class its sibling posture tests were
demoted for, with a receipt (a +21-line preamble change once flipped the
score). CLAUDE.md's own tiering rule: Opus model test -> periodic. The
weekly lane keeps the regression signal; merges stop paying a judge-
temperament tax.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: paid shards get per-shard TMPDIR + CHROMIUM_PROFILE isolation and a kill-path cleanup backstop

The free runner treats this isolation as MANDATORY (two concurrent shards
on one Chromium profile kill each other's browser; shared tmp
cross-contaminates) — the paid lane had none of it. Doubly load-bearing
here: a shard that hits its 30-min wall is group-SIGKILLed, so per-test
afterAll cleanup never runs; the rmSync backstop is the only thing keeping
wedged runs from accumulating full git-repo workspaces in the shared
tmpdir forever. This is the DAG prerequisite for raising EVALS_JOBS (next
commit) — more concurrency on shared state amplifies exactly the
shared-tree race class opus-47 exhibited.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: paid-runner defaults 4x4 -> 8x2 — halve the local gate worst case

39 of 75 skill-e2e files hold exactly ONE test, so within-shard
concurrency was dead weight for most shards: 4 jobs x 4 concurrency
yielded only ~4-6 real in-flight sessions and a 13-wave local gate worst
case (~6.5h). 8 jobs x 2 gives ~10-13 in-flight — under the
documented-safe ~15 — and ~7 waves (~3.3h worst case). CI lanes keep
their explicit EVALS_JOBS env (2 per slice; 4 for gate-census); this
changes local defaults. Rollback trigger: sustained 429 storms in the WS1
telemetry across 2 PR cycles. test/eval-detach-timeout-floor.test.ts
recomputed green (the raise LOWERS the worst-case floor).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: SHA-pin every action in the secrets-bearing eval lanes

evals.yml and evals-periodic.yml execute PR-authored code with three
provider API keys in env, yet rode mutable action tags (@v7/@v8/@v2/@v4)
— while quality-gate.yml, osv-scanner.yml, and dependency-review.yml
already model the SHA-pin pattern. All 30 uses sites across both lanes now
pin the exact commit (tag noted in a trailing comment); dependabot's
github-actions ecosystem keeps them fresh via PRs instead of silent tag
moves. Pulled forward from the plan's endgame on the CEO-review + outside-
voice agreement: supply-chain pins on secret lanes go first, not last.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: sweep wave 3 — the execFileSync family gets timeouts (90 sites, 17 files)

The tripwire's regex covered spawnSync/execSync/Bun.spawnSync but not
execFileSync — an entire blocking sync-spawn API family that could
reintroduce the shard-wedge class undetected (ship review army). Same
mechanical recipe as waves 1-2: timeout: 30_000 default, 120_000 for slow
ops, shared wrappers fixed once, string-needle sites skipped with reasons.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: review-army + adversarial test hardening

- Tripwire scans execFileSync too (ceiling 8: two more grep-needle string
  exemptions); merge-introduced timeout-less spawnSync in
  question-preference-hook fixed — the tripwire caught a site that landed
  on main AFTER the sweep, on its first day.
- gstack-detach gains TWO watchdog kill regression tests: TERM-immune
  grandchild (the killpg-after-grace escalation) and the leader-dies
  variant (the pgid-at-spawn fix — the case the first test cannot see).
- eval-flake-rank gets its unit suite (final-attempt accounting, artifact
  exclusion, shard recursion, recency bound).
- Groupkill/startup-grace shim markers are per-run unique (pid-suffixed
  sleep durations): sibling Conductor worktrees run free suites with no
  machine lock, and fixed markers let one run pgrep/pkill the other's
  shims — a cross-run flake inside the anti-flake tests.
- flake-ledger test pins the project-scoped local default; stale empty
  section headers in touchfiles-data deleted (they invited entries under
  deliberately retired categories).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: adversarial-review runtime fixes across the telemetry + kill paths

- session-runner: exit-labeling keys off 'exit', not 'close' — an orphan
  holding the pipes could relabel a REAL exit (auth failure) as
  'timeout_startup' availability noise; the kill path still always
  group-kills and cancels the reader (labeling and unblocking are separate
  concerns). Work phase arms on a flag, not firstResponseMs===0 (a same-ms
  first byte left the startup timer live all run). The CI startup grace is
  now a real FLOOR (Math.max), matching its name and pinning test.
- gstack-detach: pgid captured AT SPAWN (== child pid under
  start_new_session) — resolving it after the grace raised ESRCH once the
  leader died on SIGTERM, orphaning TERM-immune grandchildren forever.
- test-free-shards: ledger entries carry branch + git_sha (rev-parse split:
  '--abbrev-ref HEAD HEAD' printed the branch twice and recorded it as the
  sha); local ledger default is per-PROJECT, not the machine-global tmpdir.
- eval-flake-rank: per-LINE ledger parse (one torn JSONL line vanished the
  whole series), 60-day recency bound (transcript-bearing files are MBs),
  shared isFinalizedEvalResultFile predicate (the artifact-taxonomy rule
  lived in three places); eval-store exports the predicate and finalize
  stops computing flakyRetries twice; paid-shards cleanup uses async rm
  (a SIGKILLed shard's git-workspace teardown blocked every sibling's
  stream classification on the parent event loop).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: CI trust-boundary + fail-closed repairs (adversarial findings)

- Token/exec separation restored: slices-report (runs PR-authored code:
  bun install + the reconcile runner) drops to contents:read; the PR
  comment moves to a NEW slices-comment job holding the write token with
  ZERO repo code — no checkout, no bun, only downloaded artifacts + jq/gh.
  $GITHUB_ENV/BASH_ENV persistence is job-scoped, so the split is the
  boundary. The matrix-era report job had this property; the consolidation
  had regressed it. Pinned by the wiring test.
- Reconcile exit captured via PIPESTATUS[0] in BOTH lanes: GitHub's default
  run-step shell has no pipefail, so `$?` after `| tee` was tee's exit —
  the fail-closed gate was silently fail-open. Wiring test pins it.
- PR comment: final-attempt accounting restored the dropped COST
  accumulation (the dial read $0 forever), flaky passes render as the
  warning they are (never as failures), and a malformed tests[] artifact
  skips that file instead of aborting the whole comment under bash -e.
- Remaining mutable action tags pinned (free-tests upload-artifact,
  ci-image checkout/docker trio — the image publisher holds packages:write
  and feeds the secret-bearing lanes). restore-deps fallback installs
  --frozen-lockfile; register-gstack-skills validates skill names before
  its rm -rf.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v1.77.0.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v1.77.0.0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: cross-model doc-review fixes — flake-ledger env knobs, CI retry-on note, stale version comment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: correct CHANGELOG receipt numbers to measured values

Gate census keys: 78 -> 77 (bun-imported E2E_TIERS count). Sweep receipt:
586 sites/176 files -> 499 sites/146 files, measured by running this
branch's spawnsync-timeout-tripwire against origin/main (exit 1, 499
violations across 146 unique files; green on this branch).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: slices-comment creates the PR comment via REST — the write-token job has no git context

The token/exec split gives slices-comment NO checkout by design, and gh's
pr-comment subcommand resolves the repo FROM git — it died with 'not a git
repository' on PR #2746's first run (the update-existing PATCH path was
already explicit-repo REST and worked). Create now posts through
gh api repos/.../issues/N/comments, and the wiring test pins that no
git-context-requiring comment call can creep back into the job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: startup-grace probes clear CI for local semantics; new probe pins the floor clamp

The two shim probes pass explicit 2s/4s graces, but in CI the runner clamps
any explicit grace up to the 300s floor (deliberate adversarial-review fix),
so 'silent API killed at the grace' died at the 30s work cap instead of 2s —
a deterministic red on every CI run, green locally. The probes now pin LOCAL
semantics with CI cleared (same save/restore pattern as their PATH shim),
and a fourth probe pins the clamp itself: CI=1 + 2s grace + 6s timeout must
kill at the 6s cap, still in the startup phase — proof an explicit low grace
cannot bypass the floor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-31 08:55:30 -07:00
committed by GitHub
co-authored by Claude Fable 5
parent 253d1dfe26
commit e76f65a8da
200 changed files with 3003 additions and 1753 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/actions/fix-bun-temp/action.yml
+18
View File
@@ -0,0 +1,18 @@
name: Fix bun temp
description: >
Redirect bun's cache/temp to a runner-writable dir. Bun creates root-owned
temp dirs during the Docker image build; GH Actions runs container jobs as
the `runner` user with HOME=/github/home, so without this redirect bun hits
EACCES on its default temp paths. Shared by every eval lane — extracted from
three byte-identical copies (legacy matrix, sliced lane, periodic).
runs:
using: composite
steps:
- shell: bash
run: |
mkdir -p /home/runner/.cache/bun
{
echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun"
echo "BUN_TMPDIR=/home/runner/.cache/bun"
echo "TMPDIR=/home/runner/.cache"
} >> "$GITHUB_ENV"
.github/actions/register-gstack-skills/action.yml
+108
View File
@@ -0,0 +1,108 @@
name: Register gstack skills for PTY tests
description: >
Register the skills PTY smokes invoke, in every place claude looks:
user-scoped ($HOME/.claude/skills — a gstack root symlink for the
preamble's absolute runtime paths, plus per-skill REAL-FILE copies because
claude's interactive-TUI skill scanner does not follow the
/github/home -> /__w cross-mount symlink), and project-scoped
(<repo>/.claude/skills, which the TUI reads for /slash commands and which
is gitignored so absent on a fresh CI checkout). Also pre-seeds every
one-time preamble marker so no PTY child takes a first-run branch mid-test
(feature discovery under ~/.claude/skills/gstack trips Claude Code's
sensitive-file permission prompt — the documented intermittent
scope-gate-question-NOT-observed failure).
Ends with the fail-fast verification loop the legacy matrix copy grew
after a silent "Unknown command" + 35-min-timeout incident: a dangling
symlink or renamed committed target fails HERE, in seconds, with a named
path — never as a wedged PTY session at the shard wall. Every consuming
lane inherits the loop by construction (it previously existed only in the
matrix copy; the sliced + periodic copies had silently dropped it).
inputs:
skills:
description: Space-separated skill dirs (each must have SKILL.md + sections/).
required: false
default: office-hours plan-ceo-review plan-eng-review plan-design-review
runs:
using: composite
steps:
- shell: bash
env:
SKILLS: ${{ inputs.skills }}
run: |
set -eu
SKILLS_DIR="$HOME/.claude/skills"
REPO="$GITHUB_WORKSPACE"
mkdir -p "$SKILLS_DIR"
# The gstack root stays a symlink — the preamble's runtime bash resolves
# ~/.claude/skills/gstack/bin/* and ~/.claude/skills/gstack/<skill>/sections/*
# through it, and bash follows cross-mount symlinks fine.
# `ln -snf` replaces a stale SYMLINK but hard-errors on a REAL
# directory under set -eu — clear a real-dir leftover first so the
# composite is rerun-safe (codex diff review). CI HOMEs are fresh;
# this only fires on a dirty rerun.
if [ -d "$SKILLS_DIR/gstack" ] && [ ! -L "$SKILLS_DIR/gstack" ]; then
rm -rf "${SKILLS_DIR:?}/gstack"
fi
ln -snf "$REPO" "$SKILLS_DIR/gstack"
for s in $SKILLS; do
# Input validation before rm -rf: a future caller passing '..' or a
# path segment would delete $HOME/.claude (claude adversarial).
case "$s" in */*|.|..|'') echo "ERROR: invalid skill name: '$s'" >&2; exit 1;; esac
rm -rf "${SKILLS_DIR:?}/$s"
mkdir -p "$SKILLS_DIR/$s"
cp "$REPO/$s/SKILL.md" "$SKILLS_DIR/$s/SKILL.md"
cp -R "$REPO/$s/sections" "$SKILLS_DIR/$s/sections"
done
PROJ_SKILLS="$REPO/.claude/skills"
mkdir -p "$PROJ_SKILLS"
for s in $SKILLS; do
rm -rf "${PROJ_SKILLS:?}/$s"
mkdir -p "$PROJ_SKILLS/$s"
cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md"
cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections"
done
# Pre-seed every ONE-TIME preamble marker so no PTY child ever takes a
# first-run branch mid-test.
mkdir -p "$HOME/.gstack"
touch "$HOME/.gstack/.activated" \
"$HOME/.gstack/.first-loop-tip-shown" \
"$HOME/.gstack/.telemetry-prompted" \
"$HOME/.gstack/.proactive-prompted" \
"$HOME/.gstack/.completeness-intro-seen" \
"$HOME/.gstack/.plan-tune-nudge-shown"
# These two resolve through the gstack root symlink into $REPO —
# untracked scratch in the CI checkout, exactly where the preamble looks.
touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \
"$SKILLS_DIR/gstack/.feature-prompted-model-overlay"
echo "--- registry under $SKILLS_DIR ---"
ls -la "$SKILLS_DIR/gstack"
# ── Fail-fast verification ──────────────────────────────────────────
# A dangling symlink or moved/renamed committed target must fail here
# with a named path, not resurface as a silent "Unknown command" and a
# PTY session wedged to its wall timeout.
for f in \
"$SKILLS_DIR/gstack/bin/gstack-update-check" \
"$SKILLS_DIR/gstack/scripts/gen-skill-docs.ts"; do
if [ ! -e "$f" ]; then
echo "ERROR: gstack root symlink dangles or target moved: $f" >&2
exit 1
fi
done
for s in $SKILLS; do
if [ ! -e "$SKILLS_DIR/$s/SKILL.md" ]; then
echo "ERROR: skill-registry target missing: $SKILLS_DIR/$s/SKILL.md" >&2
exit 1
fi
grep -m1 "^name: $s\$" "$SKILLS_DIR/$s/SKILL.md" >/dev/null \
|| { echo "ERROR: $s SKILL.md missing 'name: $s' frontmatter" >&2; exit 1; }
# Sections must exist BOTH as the copied real files (TUI reads) and
# through the gstack root symlink (the preamble's runtime paths).
for d in "$SKILLS_DIR/$s/sections" "$SKILLS_DIR/gstack/$s/sections"; do
if [ ! -d "$d" ] || [ -z "$(ls -A "$d")" ]; then
echo "ERROR: skill sections missing or empty: $d" >&2
exit 1
fi
done
done
echo "skill registry OK"
.github/actions/restore-deps/action.yml
+27
View File
@@ -0,0 +1,27 @@
name: Restore deps
description: >
Restore the CI image's pre-installed node_modules via recursive copy, or
fall back to bun install when the lockfile changed. Symlinking breaks bun's
realpath-based module resolution (realpath escapes the workspace and
sibling deps stop resolving); hardlink copy fails across overlay-fs layers
("Invalid cross-device link"). Recursive copy costs ~5s for ~200 packages —
still far cheaper than a network install. Extracted from five byte-similar
copies across the eval lanes.
runs:
using: composite
steps:
- shell: bash
run: |
if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then
# rm first: `cp -r SRC node_modules` with an existing node_modules
# NESTS the copy (node_modules/node_modules_cache) and leaves stale
# deps active. CI workspaces are fresh today, but a reusable
# composite must survive a rerun/dirty workspace (codex diff review).
rm -rf node_modules
cp -r /opt/node_modules_cache node_modules
else
# Frozen: this composite is canonical for lanes that run PR code
# with provider keys in env — a drifted lockfile must fail loudly,
# never silently re-resolve versions (claude adversarial).
bun install --frozen-lockfile
fi
.github/actions/seed-claude-config/action.yml
+30
View File
@@ -0,0 +1,30 @@
name: Seed claude interactive config
description: >
Seed ~/.claude.json with onboarding-complete plus the API-key approval so
PTY sessions never wedge on the fresh-container onboarding / "use detected
ANTHROPIC_API_KEY?" dialog. Mirrors what the hermetic E2E child env seeds.
Idempotent — safe to run unconditionally in sliced lanes where any slice
can host a PTY test. Only the key's last 20 chars are persisted (the
approval-hash form claude itself writes), never the whole key.
inputs:
anthropic-api-key:
description: API key whose suffix gets pre-approved. Pass from secrets at the call site.
required: false
default: ''
runs:
using: composite
steps:
- shell: bash
env:
ANTHROPIC_API_KEY: ${{ inputs.anthropic-api-key }}
run: |
node -e '
const fs = require("fs"), os = require("os"), path = require("path");
const p = path.join(os.homedir(), ".claude.json");
const seed = fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, "utf8")) : {};
seed.hasCompletedOnboarding = true;
const key = process.env.ANTHROPIC_API_KEY || "";
if (key) seed.customApiKeyResponses = { approved: [key.slice(-20)], rejected: [] };
fs.writeFileSync(p, JSON.stringify(seed, null, 2));
console.log("seeded", p);
'
.github/docker/Dockerfile.ci
+9 -2
View File
@@ -74,8 +74,15 @@ ENV BUN_INSTALL="/usr/local"
RUN curl --retry 5 --retry-delay 5 --retry-connrefused -fsSL https://bun.sh/install \
| bash -s "bun-v1.3.13"
# Claude CLI
RUN npm i -g @anthropic-ai/claude-code
# Claude CLI — pinned to an EXACT version, same discipline as the bun pin
# above. The PTY harness (test/helpers/claude-pty-runner.ts) screen-scrapes
# this CLI's TUI (trust dialog, input prompt, spinner glyphs); an unpinned
# install rebuilt weekly rode the TUI wherever it drifted, and that drift
# broke the harness three separate times (welcome-screen wedge on 2.1.233,
# skillify HOME discovery on 2.1.237, guard/freeze hooks on 2.1.162).
# Bump deliberately, via a PR that runs the PTY gate against the new TUI.
# test/ci-image-cli-pin.test.ts fails the free suite if this pin is removed.
RUN npm i -g @anthropic-ai/claude-code@2.1.251
# Playwright system deps (Chromium) — needed for browse E2E tests
RUN npx playwright install-deps chromium
.github/workflows/ci-image.yml
+11 -7
View File
@@ -1,8 +1,12 @@
name: Build CI Image
on:
# Rebuild weekly (Monday 4am UTC) to pick up CLI updates — deliberately 2h
# BEFORE evals-periodic's 6am cron so the weekly eval run finds a fresh
# image instead of racing a half-pushed tag or duplicating the build.
# Weekly self-heal (Monday 4am UTC) — deliberately 2h BEFORE
# evals-periodic's 6am cron so the weekly eval run finds the image instead
# of racing a half-pushed tag. With the claude CLI pinned in Dockerfile.ci
# (v1.77+), this cron no longer pulls CLI updates: when the content-hash
# tag already exists it's a ~30s no-op, and it only rebuilds if the tag
# was somehow lost. CLI bumps happen by editing the Dockerfile pin in a PR
# that runs the PTY gate against the new TUI.
schedule:
- cron: '0 4 * * 1'
# Rebuild on Dockerfile or lockfile changes. package.json is deliberately
@@ -32,7 +36,7 @@ jobs:
contents: read
packages: write
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
# Copy lockfile + package.json into Docker build context
- run: cp package.json bun.lock .github/docker/ && cp -R patches .github/docker/patches
@@ -44,7 +48,7 @@ jobs:
- id: meta
run: echo "tag=ghcr.io/${{ github.repository }}/ci:${{ hashFiles('.github/docker/Dockerfile.ci', 'bun.lock', 'patches/**') }}" >> "$GITHUB_OUTPUT"
- uses: docker/login-action@v4
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -65,10 +69,10 @@ jobs:
# Registry cache export needs a docker-container builder — the default
# `docker` driver hard-errors on cache-to.
- if: steps.check.outputs.exists == 'false'
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
- if: steps.check.outputs.exists == 'false'
uses: docker/build-push-action@v7
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
with:
context: .github/docker
file: .github/docker/Dockerfile.ci
.github/workflows/evals-periodic.yml
+34 -92
View File
@@ -32,7 +32,7 @@ jobs:
outputs:
image-tag: ${{ steps.meta.outputs.tag }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- id: meta
# Keep in sync with evals.yml — key on Dockerfile + lockfile only
@@ -40,7 +40,7 @@ jobs:
# Byte-identity pinned by test/ci-image-tag-binding.test.ts.
run: echo "tag=${{ env.IMAGE }}:${{ hashFiles('.github/docker/Dockerfile.ci', 'bun.lock', 'patches/**') }}" >> "$GITHUB_OUTPUT"
- uses: docker/login-action@v4
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -61,10 +61,10 @@ jobs:
# Registry cache export needs a docker-container builder — the default
# `docker` driver hard-errors on cache-to.
- if: steps.check.outputs.exists == 'false'
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
- if: steps.check.outputs.exists == 'false'
uses: docker/build-push-action@v7
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
with:
context: .github/docker
file: .github/docker/Dockerfile.ci
@@ -90,24 +90,19 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}
options: --user runner
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- name: Restore deps
run: |
if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then
cp -r /opt/node_modules_cache node_modules
else
bun install
fi
uses: ./.github/actions/restore-deps
- name: Emit run manifest (ALL periodic tests minus reasoned excludes)
env:
EVALS_ALL: "1"
run: EVALS_TIER=periodic bun run scripts/test-paid-shards.ts --tier periodic --emit-plan /tmp/paid-plan/manifest.json --slices 6
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: paid-plan
path: /tmp/paid-plan/manifest.json
@@ -133,7 +128,7 @@ jobs:
matrix:
slice: [1, 2, 3, 4, 5, 6]
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# Full history: files with SELF-derived selection (the LLM-judge
# map, routing) walk git at module load, and selection is
@@ -144,73 +139,27 @@ jobs:
persist-credentials: false
- name: Fix bun temp
run: |
mkdir -p /home/runner/.cache/bun
{
echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun"
echo "BUN_TMPDIR=/home/runner/.cache/bun"
echo "TMPDIR=/home/runner/.cache"
} >> "$GITHUB_ENV"
uses: ./.github/actions/fix-bun-temp
- name: Restore deps
run: |
if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then
cp -r /opt/node_modules_cache node_modules
else
bun install
fi
uses: ./.github/actions/restore-deps
- run: bun run build
# Any slice can host a PTY test — seed + registration run
# unconditionally (idempotent; mirrors evals.yml's sliced lane).
# unconditionally (idempotent; mirrors evals.yml's sliced lane). The
# register composite carries the fail-fast dangling-symlink/frontmatter
# verification loop — this lane previously LACKED it, so a moved skill
# target surfaced as a silent "Unknown command" + wedged PTY session.
- name: Seed claude interactive config
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
node -e '
const fs = require("fs"), os = require("os"), path = require("path");
const p = path.join(os.homedir(), ".claude.json");
const seed = fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, "utf8")) : {};
seed.hasCompletedOnboarding = true;
const key = process.env.ANTHROPIC_API_KEY || "";
if (key) seed.customApiKeyResponses = { approved: [key.slice(-20)], rejected: [] };
fs.writeFileSync(p, JSON.stringify(seed, null, 2));
console.log("seeded", p);
'
uses: ./.github/actions/seed-claude-config
with:
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
- name: Register gstack skills for PTY tests
run: |
set -eu
SKILLS_DIR="$HOME/.claude/skills"
REPO="$GITHUB_WORKSPACE"
mkdir -p "$SKILLS_DIR"
ln -snf "$REPO" "$SKILLS_DIR/gstack"
for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do
rm -rf "${SKILLS_DIR:?}/$s"
mkdir -p "$SKILLS_DIR/$s"
cp "$REPO/$s/SKILL.md" "$SKILLS_DIR/$s/SKILL.md"
cp -R "$REPO/$s/sections" "$SKILLS_DIR/$s/sections"
done
PROJ_SKILLS="$REPO/.claude/skills"
mkdir -p "$PROJ_SKILLS"
for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do
rm -rf "${PROJ_SKILLS:?}/$s"
mkdir -p "$PROJ_SKILLS/$s"
cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md"
cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections"
done
mkdir -p "$HOME/.gstack"
touch "$HOME/.gstack/.activated" \
"$HOME/.gstack/.first-loop-tip-shown" \
"$HOME/.gstack/.telemetry-prompted" \
"$HOME/.gstack/.proactive-prompted" \
"$HOME/.gstack/.completeness-intro-seen" \
"$HOME/.gstack/.plan-tune-nudge-shown"
touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \
"$SKILLS_DIR/gstack/.feature-prompted-model-overlay"
uses: ./.github/actions/register-gstack-skills
- uses: actions/download-artifact@v8
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: paid-plan
path: /tmp/paid-plan
@@ -228,7 +177,7 @@ jobs:
- name: Upload slice results
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: paid-slice-${{ matrix.slice }}
path: /tmp/paid-slice-results
@@ -236,7 +185,7 @@ jobs:
- name: Upload shard logs on failure
if: failure()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: paid-slice-${{ matrix.slice }}-logs
# The Fix-bun-temp step points TMPDIR at /home/runner/.cache, so the
@@ -266,7 +215,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}
options: --user runner
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# Full history: files with SELF-derived selection (the LLM-judge
# map, routing) walk git at module load, and selection is
@@ -277,21 +226,10 @@ jobs:
persist-credentials: false
- name: Fix bun temp
run: |
mkdir -p /home/runner/.cache/bun
{
echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun"
echo "BUN_TMPDIR=/home/runner/.cache/bun"
echo "TMPDIR=/home/runner/.cache"
} >> "$GITHUB_ENV"
uses: ./.github/actions/fix-bun-temp
- name: Restore deps
run: |
if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then
cp -r /opt/node_modules_cache node_modules
else
bun install
fi
uses: ./.github/actions/restore-deps
- run: bun run build
@@ -309,7 +247,7 @@ jobs:
- name: Upload census results
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: gate-census
path: /tmp/gate-census-results
@@ -328,22 +266,22 @@ jobs:
# `gh api /issues` — gated by the issues permission.
issues: write
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- uses: oven-sh/setup-bun@v2
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install --frozen-lockfile
- uses: actions/download-artifact@v8
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: paid-plan
path: /tmp/paid-report
- uses: actions/download-artifact@v8
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: paid-slice-[0-9]*
path: /tmp/paid-report
@@ -354,7 +292,11 @@ jobs:
run: |
set +e
EVALS_TIER=periodic bun run scripts/test-paid-shards.ts --tier periodic --report /tmp/paid-report | tee /tmp/report.txt
echo "exit=$?" >> "$GITHUB_OUTPUT"
# PIPESTATUS[0], NOT $?: GitHub's default run-step shell is
# `bash -e {0}` with NO pipefail, so $? after the pipe is tee's
# exit (always 0) — the fail-closed gate was silently fail-open
# (caught by the ship review army; the wiring test now pins this).
echo "exit=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
# A red weekly lane nobody must action is waste — upsert ONE tracking
# issue (never a new issue per week) with the reconciliation output, so
.github/workflows/evals.yml
+192 -496
View File
@@ -15,7 +15,6 @@ concurrency:
env:
IMAGE: ghcr.io/${{ github.repository }}/ci
EVALS_TIER: gate
jobs:
# Build Docker image with pre-baked toolchain (cached — only rebuilds on Dockerfile/lockfile change)
@@ -34,7 +33,7 @@ jobs:
outputs:
image-tag: ${{ steps.meta.outputs.tag }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- id: meta
# Key on Dockerfile + lockfile only. package.json is deliberately NOT
@@ -44,7 +43,7 @@ jobs:
# overwrites /workspace and node_modules comes from the lockfile.
run: echo "tag=${{ env.IMAGE }}:${{ hashFiles('.github/docker/Dockerfile.ci', 'bun.lock', 'patches/**') }}" >> "$GITHUB_OUTPUT"
- uses: docker/login-action@v4
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -69,10 +68,10 @@ jobs:
# Registry cache export needs a docker-container builder — the default
# `docker` driver hard-errors on cache-to (first live run of the trio).
- if: steps.check.outputs.exists == 'false'
uses: docker/setup-buildx-action@v4
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
- if: steps.check.outputs.exists == 'false'
uses: docker/build-push-action@v7
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
with:
context: .github/docker
file: .github/docker/Dockerfile.ci
@@ -85,430 +84,27 @@ jobs:
${{ steps.meta.outputs.tag }}
${{ env.IMAGE }}:latest
# Fork PRs never receive repository secrets (ANTHROPIC_API_KEY et al), so every
# API-calling eval fails at SDK auth before a model runs. Skip deterministically
# rather than leaving the outcome to Docker-cache luck: a warm cache let these
# run and fail, a cold one made build-image fail its push and the shards skip.
# Same-repo PRs, pushes, and workflow_dispatch keep full coverage. Fork work
# gets real coverage via a trusted base-repo branch.
evals:
runs-on: ${{ matrix.suite.runner || 'ubicloud-standard-8' }}
needs: build-image
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
# Least privilege for the job that executes PR-authored code with three
# provider API keys in env: read-only contents, packages:read for the
# container-image pull below. Without this block the job ran on the
# repo-default token grant.
permissions:
contents: read
packages: read
container:
image: ${{ needs.build-image.outputs.image-tag }}
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
options: --user runner
timeout-minutes: ${{ matrix.suite.timeout || 25 }}
strategy:
fail-fast: false
matrix:
suite:
- name: llm-judge
file: test/skill-llm-eval.test.ts
- name: e2e-browse
file: test/skill-e2e-bws.test.ts
runner: ubicloud-standard-8
- name: e2e-plan
file: test/skill-e2e-plan.test.ts
- name: e2e-deploy
file: test/skill-e2e-deploy.test.ts
- name: e2e-design
file: test/skill-e2e-design.test.ts
- name: e2e-qa-bugs
file: test/skill-e2e-qa-bugs.test.ts
- name: e2e-qa-workflow
file: test/skill-e2e-qa-workflow.test.ts
- name: e2e-review
file: test/skill-e2e-review.test.ts
- name: e2e-retro
file: test/skill-e2e-retro.test.ts
- name: e2e-review-attribution
file: test/skill-e2e-review-attribution.test.ts
- name: e2e-workflow
file: test/skill-e2e-workflow.test.ts
# Earned its extra attempt with receipts: document-release is a
# long multi-step E2E that timed out on attempt 2 under in-shard
# concurrency (PR #2593 round 4) while passing other rounds.
retries: 2
# Rehomed from the deleted pre-split monolith (its filename never
# matched the skill-e2e-* glob, so these gate tests silently never
# ran). Both files hold gate-tier tests: review/plan-eng coverage
# audits and the /ship failure-ownership triage.
- name: e2e-coverage-audit
file: test/skill-e2e-coverage-audit.test.ts
- name: e2e-triage
file: test/skill-e2e-triage.test.ts
# ship-docsync is whole-file tier-gated (describeE2ETier('gate') keeps
# it out of the periodic shard census), so its row MUST set tier: gate
# — without it the self-gate skips every test and the job reports a
# hollow green (the same silent-skip class as the rehomed monolith
# above, one layer deeper). The Run step exports EVALS_TIER from this
# property; rows without it keep EVALS_TIER empty (= unset: every
# reader is `=== '<tier>'` or truthiness). Enforced by
# test/evals-workflow-matrix.test.ts.
- name: e2e-ship-docsync
file: test/skill-e2e-ship-docsync.test.ts
tier: gate
# #2733 behavioral proof: the document-release JSON contract survives
# a firing AUQ gate inside a spawned-marked subagent. Whole-file
# tier-gated like ship-docsync, so the row carries tier: gate.
- name: e2e-docsync-spawned
file: test/skill-e2e-docsync-spawned.test.ts
tier: gate
# Consent-gate guardrail for the Third-Party Web Actions contract
# (Aside recommended-driver rewrite): hermetic shims, deterministic
# grep assertions, gate tier in E2E_TIERS.
- name: e2e-third-party-actions
file: test/skill-e2e-third-party-actions.test.ts
tier: gate
- name: e2e-routing
file: test/skill-routing-e2e.test.ts
# (e2e-codex / e2e-gemini rows deleted: both files are whole-file
# periodic-tier, so with no row tier: they ran ZERO tests and
# reported green on every PR — ~2 min of runner per PR of pure
# false confidence. The periodic lane owns these suites.)
# Real-PTY plan-mode smokes. Only the deterministically-reliable ones
# are CI-gated: office-hours (asks its mode question first, caught by
# the collapsed/bullet prose-AUQ detector) and plan-mode-no-op (no
# ask-first dependency). The plan-eng/plan-design plan-mode + floor
# smokes are periodic (stochastic ask-first — see touchfiles E2E_TIERS).
# Needs the interactive-config seed step below; PTY sessions otherwise
# wedge on the fresh-container onboarding/API-key dialog.
- name: e2e-pty-plan-smoke
file: test/skill-e2e-office-hours-auto-mode.test.ts test/skill-e2e-plan-mode-no-op.test.ts
# Both files are whole-file describeE2ETier('gate') — without this
# row tier: the job burned ~7 min of setup then skipped every
# describe (hollow-green since the files adopted the self-gate).
tier: gate
timeout: 35
# The documented contention-heavy PTY family: ROTATING members
# failed attempt 2 in consecutive PR #2593 rounds
# (plan-design-review, then plan-eng-review) while the family
# passes on branches still running three attempts. Every other
# row keeps --retry 1.
retries: 2
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
# Don't write the token into .git/config — this job runs
# PR-authored code; nothing in it pushes.
persist-credentials: false
# Bun creates root-owned temp dirs during Docker build. GH Actions runs as
# runner user with HOME=/github/home. Redirect bun's cache to a writable dir.
- name: Fix bun temp
run: |
mkdir -p /home/runner/.cache/bun
{
echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun"
echo "BUN_TMPDIR=/home/runner/.cache/bun"
echo "TMPDIR=/home/runner/.cache"
} >> "$GITHUB_ENV"
# Restore pre-installed node_modules from Docker image via recursive
# copy. Symlink (`ln -s`) breaks bun's module resolution because bun
# resolves a file's realpath when walking up to find node_modules/<dep>;
# from a symlinked path, realpath escapes the workspace and sibling
# deps no longer resolve. Hardlink copy (`cp -al`) fails because /opt
# and /workspace are on different overlay-fs layers ("Invalid
# cross-device link"). Recursive copy works on every layout. Cost:
# ~5s for ~200 packages of small JS files vs ~0s for symlink — still
# vastly cheaper than rerunning `bun install` (network + resolution).
- name: Restore deps
run: |
if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then
cp -r /opt/node_modules_cache node_modules
else
bun install
fi
- run: bun run build
# Verify Playwright can launch Chromium (fails fast if sandbox/deps are broken)
- name: Verify Chromium
if: matrix.suite.name == 'e2e-browse'
run: |
echo "whoami=$(whoami) HOME=$HOME TMPDIR=${TMPDIR:-unset}"
touch /tmp/.bun-test && rm /tmp/.bun-test && echo "/tmp writable"
bun -e "import {chromium} from 'playwright';const b=await chromium.launch({args:['--no-sandbox']});console.log('Chromium OK');await b.close()"
# PTY smokes spawn the interactive `claude` TUI. A fresh container has no
# ~/.claude.json, so claude wedges on the onboarding + "use detected
# ANTHROPIC_API_KEY?" dialog and the spawned session never reaches the
# skill. Seed onboarding-complete + the key approval (mirrors what the
# hermetic E2E child env seeds). Scoped to this suite; needs its OWN key
# env (the secrets block below is on the Run step only).
- name: Seed claude interactive config
if: matrix.suite.name == 'e2e-pty-plan-smoke'
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
node -e '
const fs = require("fs"), os = require("os"), path = require("path");
const p = path.join(os.homedir(), ".claude.json");
const seed = fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, "utf8")) : {};
seed.hasCompletedOnboarding = true;
const key = process.env.ANTHROPIC_API_KEY || "";
if (key) seed.customApiKeyResponses = { approved: [key.slice(-20)], rejected: [] };
fs.writeFileSync(p, JSON.stringify(seed, null, 2));
console.log("seeded", p);
'
# PTY smokes drive the interactive `claude` TUI and send /office-hours,
# /plan-ceo-review, /plan-eng-review, and /plan-design-review. Claude Code
# discovers user-scoped skills from $HOME/.claude/skills/<name>/SKILL.md,
# but .claude/skills is gitignored, so a fresh CI checkout has NO registry
# — claude prints "Unknown command: /plan-ceo-review". Mirror setup's
# --no-prefix registry minimally: a gstack root symlink (resolves the
# preamble's absolute ~/.claude/skills/gstack/bin/* and
# ~/.claude/skills/gstack/<skill>/sections/* paths) plus a per-skill
# top-level dir holding SKILL.md (+ sections) symlinks for the four skills
# these tests invoke. No ./setup (it builds binaries, launches Chromium,
# installs fonts, reads a /dev/tty prompt) and no binary build (SKILL.md +
# bin/ + sections/ are committed). $HOME is /github/home here; the spawned
# claude inherits it (this runner adds no HOME/CLAUDE_CONFIG_DIR override,
# no hermetic mode) and the Seed step already proved claude reads $HOME.
#
# KEEP THIS STEP even though seedSkills/hermeticSkillsConfigDir() now
# registers skills for hermetic PTY children: that registry is SYMLINKS
# into the repo checkout, and this container's cross-mount symlinks
# defeat the TUI skill scanner (see the note inside the step below) —
# the real-file copies here are what the TUI actually reads. HOME is
# also not hermeticized, so the absolute ~/.claude/skills/gstack/...
# preamble paths resolve through the gstack root symlink this step makes.
- name: Register gstack skills for PTY smoke
if: matrix.suite.name == 'e2e-pty-plan-smoke'
run: |
set -eu
SKILLS_DIR="$HOME/.claude/skills"
REPO="$GITHUB_WORKSPACE" # /__w/gstack/gstack
mkdir -p "$SKILLS_DIR"
# The gstack root stays a symlink — the preamble's runtime bash resolves
# ~/.claude/skills/gstack/bin/* and ~/.claude/skills/gstack/<skill>/sections/*
# through it, and bash follows cross-mount symlinks fine.
ln -snf "$REPO" "$SKILLS_DIR/gstack"
# But the per-skill SKILL.md the TUI DISCOVERS must be a REAL file on the
# same mount as $HOME. claude 2.1.187's interactive-TUI skill scanner does
# not follow the /github/home -> /__w cross-mount symlink (proven: `claude
# -p` discovered the skill — READY — while the TUI rejected /office-hours
# as "Unknown command"; a local macOS repro with the identical symlinked
# registry recognized it, isolating the failure to the container's
# cross-mount symlink). Copy SKILL.md + sections as real files so the TUI
# reads them directly.
for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do
rm -rf "${SKILLS_DIR:?}/$s"
mkdir -p "$SKILLS_DIR/$s"
cp "$REPO/$s/SKILL.md" "$SKILLS_DIR/$s/SKILL.md"
cp -R "$REPO/$s/sections" "$SKILLS_DIR/$s/sections"
done
# Also register PROJECT-scoped (cwd) skills. claude's interactive TUI
# surfaces /slash commands from <cwd>/.claude/skills, and the smokes run
# with cwd=$REPO whose .claude/skills is gitignored (absent on a fresh CI
# checkout) — the user-dir registration above feeds `claude -p` but the
# TUI looks here. No gstack symlink in the project dir: it would point at
# its own parent ($REPO). Runtime preamble paths use the user-dir
# ~/.claude/skills/gstack symlink above.
PROJ_SKILLS="$REPO/.claude/skills"
mkdir -p "$PROJ_SKILLS"
for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do
rm -rf "${PROJ_SKILLS:?}/$s"
mkdir -p "$PROJ_SKILLS/$s"
cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md"
cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections"
done
# Pre-seed every ONE-TIME preamble marker so no PTY child ever takes a
# first-run branch mid-test. On a fresh runner these are all missing, so
# each smoke's preamble fires feature discovery / telemetry / lake-intro
# prompts before the behavior under test — and touching the
# feature-discovery marker under ~/.claude/skills/gstack/ trips Claude
# Code's sensitive-file permission prompt, stalling the run before the
# scope gate renders (the documented intermittent
# scope-gate-question-NOT-observed failure: outcome=asked was the
# permission dialog, not the gate). Dev machines never hit this because
# the operator's markers already exist; CI must seed them explicitly.
mkdir -p "$HOME/.gstack"
touch "$HOME/.gstack/.activated" \
"$HOME/.gstack/.first-loop-tip-shown" \
"$HOME/.gstack/.telemetry-prompted" \
"$HOME/.gstack/.proactive-prompted" \
"$HOME/.gstack/.completeness-intro-seen" \
"$HOME/.gstack/.plan-tune-nudge-shown"
# These two resolve through the gstack root symlink into $REPO —
# untracked scratch in the CI checkout, exactly where the preamble looks.
touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \
"$SKILLS_DIR/gstack/.feature-prompted-model-overlay"
echo "--- registry under $SKILLS_DIR ---"
ls -la "$SKILLS_DIR/gstack" "$SKILLS_DIR/office-hours" "$SKILLS_DIR/plan-ceo-review"
# Fail fast if any committed target moved/renamed — a dangling symlink
# would otherwise resurface as a silent "Unknown command" + 35-min timeout.
for f in \
"$SKILLS_DIR/office-hours/SKILL.md" \
"$SKILLS_DIR/plan-ceo-review/SKILL.md" \
"$SKILLS_DIR/plan-eng-review/SKILL.md" \
"$SKILLS_DIR/plan-design-review/SKILL.md" \
"$SKILLS_DIR/gstack/bin/gstack-update-check" \
"$SKILLS_DIR/gstack/office-hours/sections/design-and-handoff.md" \
"$SKILLS_DIR/gstack/plan-ceo-review/sections/review-sections.md" \
"$SKILLS_DIR/gstack/plan-eng-review/sections/review-sections.md" \
"$SKILLS_DIR/gstack/plan-design-review/sections/review-sections.md"; do
if [ ! -e "$f" ]; then
echo "ERROR: skill-registry target missing (symlink dangles): $f" >&2
exit 1
fi
done
for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do
grep -m1 "^name: $s\$" "$SKILLS_DIR/$s/SKILL.md" >/dev/null \
|| { echo "ERROR: $s SKILL.md missing 'name: $s' frontmatter" >&2; exit 1; }
done
echo "skill registry OK"
- name: Run ${{ matrix.suite.name }}
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
EVALS_CONCURRENCY: "40"
PLAYWRIGHT_BROWSERS_PATH: /opt/playwright-browsers
# Per-row tier activation for whole-file-gated suites. Empty when the
# row declares no tier — every EVALS_TIER reader treats empty as unset
# (`=== '<tier>'` comparisons and the truthiness check in
# test/helpers/e2e-helpers.ts:70), so untiered rows are byte-for-byte
# unaffected.
EVALS_TIER: ${{ matrix.suite.tier || '' }}
run: EVALS=1 bun test --retry ${{ matrix.suite.retries || 1 }} --concurrent --max-concurrency 40 ${{ matrix.suite.file }}
- name: Upload eval results
if: always()
uses: actions/upload-artifact@v7
with:
name: eval-${{ matrix.suite.name }}
path: ~/.gstack-dev/evals/*.json
retention-days: 90
report:
runs-on: ubicloud-standard-2
needs: evals
if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
timeout-minutes: 5
permissions:
contents: read
pull-requests: write
# The comment upsert below calls the REST `/issues/{n}/comments` endpoints
# (gh api ... issues/comments). With GITHUB_TOKEN those are gated by the
# `issues` permission, not `pull-requests` — without it the GET returns 401
# on every PR that produces eval artifacts (PRs with no artifacts exit
# early and never hit it, which is why this stayed hidden). See #1802 CI fix.
issues: write
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 1
- name: Download all eval artifacts
uses: actions/download-artifact@v8
with:
pattern: eval-*
path: /tmp/eval-results
merge-multiple: true
- name: Post PR comment
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# shellcheck disable=SC2086,SC2059
RESULTS=$(find /tmp/eval-results -name '*.json' 2>/dev/null | sort)
if [ -z "$RESULTS" ]; then
echo "No eval results found"
exit 0
fi
TOTAL=0; PASSED=0; FAILED=0; COST="0"
SUITE_LINES=""
for f in $RESULTS; do
if ! jq -e '.total_tests' "$f" >/dev/null 2>&1; then
echo "Skipping malformed JSON: $f"
continue
fi
T=$(jq -r '.total_tests // 0' "$f")
P=$(jq -r '.passed // 0' "$f")
F=$(jq -r '.failed // 0' "$f")
C=$(jq -r '.total_cost_usd // 0' "$f")
TIER=$(jq -r '.tier // "unknown"' "$f")
[ "$T" -eq 0 ] && continue
TOTAL=$((TOTAL + T))
PASSED=$((PASSED + P))
FAILED=$((FAILED + F))
COST=$(echo "$COST + $C" | bc)
STATUS_ICON="✅"
[ "$F" -gt 0 ] && STATUS_ICON="❌"
SUITE_LINES="${SUITE_LINES}| ${TIER} | ${P}/${T} | ${STATUS_ICON} | \$${C} |\n"
done
STATUS="✅ PASS"
[ "$FAILED" -gt 0 ] && STATUS="❌ FAIL"
BODY="## E2E Evals: ${STATUS}
**${PASSED}/${TOTAL}** tests passed | **\$${COST}** total cost
| Suite | Result | Status | Cost |
|-------|--------|--------|------|
$(echo -e "$SUITE_LINES")
---
*ubicloud-standard-8 runners (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite*"
if [ "$FAILED" -gt 0 ]; then
FAILURES=""
for f in $RESULTS; do
if ! jq -e '.failed' "$f" >/dev/null 2>&1; then continue; fi
F=$(jq -r '.failed // 0' "$f")
[ "$F" -eq 0 ] && continue
FAILS=$(jq -r '.tests[] | select(.passed == false) | "- ❌ \(.name): \(.exit_reason // "unknown")"' "$f" 2>/dev/null || echo "- ⚠️ $(basename "$f"): parse error")
FAILURES="${FAILURES}${FAILS}\n"
done
BODY="${BODY}
### Failures
$(echo -e "$FAILURES")"
fi
# Update existing comment or create new one
COMMENT_ID=$(gh api repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments \
--jq '.[] | select(.body | startswith("## E2E Evals")) | .id' | tail -1)
if [ -n "$COMMENT_ID" ]; then
gh api "repos/${{ github.repository }}/issues/comments/${COMMENT_ID}" \
-X PATCH -f body="$BODY"
else
gh pr comment "${{ github.event.pull_request.number }}" --body "$BODY"
fi
# ── Sliced lane (paid-CI re-platform, parity phase) ─────────────────────────
# ── Sliced lane (the ONLY paid lane; legacy 17-row matrix deleted) ──────────
# One PLANNER computes diff selection + the slice plan ONCE (killing
# per-slice selector divergence); K executors consume the manifest; the
# report reconciles results against it FAIL-CLOSED (a slice whose artifact
# never landed is a failure, a planned shard nobody reported is a failure —
# hollow lanes cannot aggregate green). Runs AFTER the matrix (`needs:
# evals`) so provider concurrency never doubles while both lanes coexist;
# once parity is demonstrated the matrix + its ratchets are deleted and this
# lane loses the needs edge. Engine: scripts/test-paid-shards.ts — the same
# runner local eval:bg:gate uses, so CI and local share one selection engine.
# hollow lanes cannot aggregate green). Engine: scripts/test-paid-shards.ts —
# the same runner local eval:bg:gate uses, so CI and local share one
# selection engine, and every gate-tier file is in the census by
# construction (no hand-enumerated rows to drift). The legacy matrix ran
# 18 enumerated files for 22.6 min/$21 per PR serialized AHEAD of this
# lane's 49-file diff-selected census; parity was demonstrated (sliced
# census ⊇ matrix files) and the matrix deleted — one revert restores it.
#
# Fork PRs never receive repository secrets (ANTHROPIC_API_KEY et al), so
# every API-calling eval fails at SDK auth before a model runs. Skip
# deterministically; fork work gets real coverage via a trusted base-repo
# branch (see CLAUDE.md's garrytan-agents workflow).
plan-slices:
runs-on: ubicloud-standard-8
needs: [build-image, evals]
if: always() && needs.build-image.result == 'success' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
needs: build-image
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
timeout-minutes: 10
permissions:
contents: read
@@ -520,7 +116,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}
options: --user runner
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# The planner is the ONE place that needs history: diff selection
# resolves a merge-base. Executors run from the manifest and stay
@@ -530,19 +126,14 @@ jobs:
persist-credentials: false
- name: Restore deps
run: |
if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then
cp -r /opt/node_modules_cache node_modules
else
bun install
fi
uses: ./.github/actions/restore-deps
- name: Emit run manifest
env:
EVALS_ALL: ${{ (github.event_name == 'workflow_dispatch' && inputs.evals_all) && '1' || '' }}
run: EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --emit-plan /tmp/paid-plan/manifest.json --slices 6
- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: paid-plan
path: /tmp/paid-plan/manifest.json
@@ -572,7 +163,7 @@ jobs:
matrix:
slice: [1, 2, 3, 4, 5, 6]
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# Full history: files with SELF-derived selection (the LLM-judge
# map, routing) walk git at module load, and selection is
@@ -583,74 +174,28 @@ jobs:
persist-credentials: false
- name: Fix bun temp
run: |
mkdir -p /home/runner/.cache/bun
{
echo "BUN_INSTALL_CACHE_DIR=/home/runner/.cache/bun"
echo "BUN_TMPDIR=/home/runner/.cache/bun"
echo "TMPDIR=/home/runner/.cache"
} >> "$GITHUB_ENV"
uses: ./.github/actions/fix-bun-temp
- name: Restore deps
run: |
if [ -d /opt/node_modules_cache ] && diff -q /opt/node_modules_cache/.bun.lock bun.lock >/dev/null 2>&1; then
cp -r /opt/node_modules_cache node_modules
else
bun install
fi
uses: ./.github/actions/restore-deps
- run: bun run build
# Any slice can host a PTY smoke, so the seed/registration steps run
# UNCONDITIONALLY (both are idempotent) — the old matrix keyed them on
# matrix.suite.name, which a sliced lane cannot do.
# matrix.suite.name, which a sliced lane cannot do. The register
# composite carries the fail-fast dangling-symlink/frontmatter
# verification loop, so a moved skill target fails HERE in seconds,
# not as a wedged PTY session at the shard wall.
- name: Seed claude interactive config
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
node -e '
const fs = require("fs"), os = require("os"), path = require("path");
const p = path.join(os.homedir(), ".claude.json");
const seed = fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, "utf8")) : {};
seed.hasCompletedOnboarding = true;
const key = process.env.ANTHROPIC_API_KEY || "";
if (key) seed.customApiKeyResponses = { approved: [key.slice(-20)], rejected: [] };
fs.writeFileSync(p, JSON.stringify(seed, null, 2));
console.log("seeded", p);
'
uses: ./.github/actions/seed-claude-config
with:
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
- name: Register gstack skills for PTY smokes
run: |
set -eu
SKILLS_DIR="$HOME/.claude/skills"
REPO="$GITHUB_WORKSPACE"
mkdir -p "$SKILLS_DIR"
ln -snf "$REPO" "$SKILLS_DIR/gstack"
for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do
rm -rf "${SKILLS_DIR:?}/$s"
mkdir -p "$SKILLS_DIR/$s"
cp "$REPO/$s/SKILL.md" "$SKILLS_DIR/$s/SKILL.md"
cp -R "$REPO/$s/sections" "$SKILLS_DIR/$s/sections"
done
PROJ_SKILLS="$REPO/.claude/skills"
mkdir -p "$PROJ_SKILLS"
for s in office-hours plan-ceo-review plan-eng-review plan-design-review; do
rm -rf "${PROJ_SKILLS:?}/$s"
mkdir -p "$PROJ_SKILLS/$s"
cp "$REPO/$s/SKILL.md" "$PROJ_SKILLS/$s/SKILL.md"
cp -R "$REPO/$s/sections" "$PROJ_SKILLS/$s/sections"
done
mkdir -p "$HOME/.gstack"
touch "$HOME/.gstack/.activated" \
"$HOME/.gstack/.first-loop-tip-shown" \
"$HOME/.gstack/.telemetry-prompted" \
"$HOME/.gstack/.proactive-prompted" \
"$HOME/.gstack/.completeness-intro-seen" \
"$HOME/.gstack/.plan-tune-nudge-shown"
touch "$SKILLS_DIR/gstack/.feature-prompted-continuous-checkpoint" \
"$SKILLS_DIR/gstack/.feature-prompted-model-overlay"
uses: ./.github/actions/register-gstack-skills
- uses: actions/download-artifact@v8
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: paid-plan
path: /tmp/paid-plan
@@ -668,7 +213,7 @@ jobs:
- name: Upload slice results
if: always()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: paid-slice-${{ matrix.slice }}
path: /tmp/paid-slice-results
@@ -678,7 +223,7 @@ jobs:
# later needs more than a summary line.
- name: Upload shard logs on failure
if: failure()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: paid-slice-${{ matrix.slice }}-logs
# The Fix-bun-temp step points TMPDIR at /home/runner/.cache, so the
@@ -697,29 +242,180 @@ jobs:
# missing slice artifact reading as green is the class this lane kills.
if: always() && needs.plan-slices.result == 'success'
timeout-minutes: 5
# contents:read ONLY — this job executes PR-authored code (bun install
# lifecycle scripts + the reconcile runner from the PR checkout), so it
# must never hold a write-scoped token. The PR comment lives in the
# separate slices-comment job below, which runs NO repo code: a
# $GITHUB_ENV/BASH_ENV persistence trick is job-scoped, so the split is
# the trust boundary (codex adversarial finding, 2026-08-31 — the old
# matrix-era report job had this separation and the consolidation had
# regressed it).
permissions:
contents: read
outputs:
reconcile-exit: ${{ steps.reconcile.outputs.exit }}
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- uses: oven-sh/setup-bun@v2
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install --frozen-lockfile
- uses: actions/download-artifact@v8
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: paid-plan
path: /tmp/paid-report
- uses: actions/download-artifact@v8
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: paid-slice-[0-9]*
path: /tmp/paid-report
merge-multiple: true
- name: Reconcile slices against the manifest (fail-closed)
run: EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --report /tmp/paid-report
id: reconcile
run: |
set +e
EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --report /tmp/paid-report | tee /tmp/report.txt
# PIPESTATUS[0], NOT $?: GitHub's default run-step shell is
# `bash -e {0}` with NO pipefail, so $? after the pipe is tee's
# exit (always 0) — the fail-closed gate was silently fail-open
# (caught by the ship review army; the wiring test now pins this).
echo "exit=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
- name: Upload reconciliation output for the comment job
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: report-verdict
path: /tmp/report.txt
if-no-files-found: ignore
retention-days: 30
- name: Fail the workflow when reconciliation failed
if: steps.reconcile.outputs.exit != '0'
run: exit 1
# PR comment in its OWN job with the write token and ZERO repo code: no
# checkout, no bun install — only downloaded artifacts, jq, and gh. See the
# trust-boundary note on slices-report.
slices-comment:
runs-on: ubicloud-standard-2
needs: slices-report
if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && needs.slices-report.result != 'skipped'
timeout-minutes: 5
permissions:
pull-requests: write
# The comment upsert calls the REST `/issues/{n}/comments` endpoints
# (gh api ... issues/comments). With GITHUB_TOKEN those are gated by the
# `issues` permission, not `pull-requests` (#1802 CI fix).
issues: write
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: paid-slice-[0-9]*
path: /tmp/paid-report
merge-multiple: true
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: report-verdict
path: /tmp/verdict
continue-on-error: true
# Sourced from the slice artifacts' eval-store JSONs. Keeps the
# "## E2E Evals" marker so the upsert keeps updating the same comment.
# Runs even when reconciliation failed — a red lane on the PR is the point.
- name: Post PR comment
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RECONCILE_EXIT: ${{ needs.slices-report.outputs.reconcile-exit }}
run: |
# shellcheck disable=SC2086,SC2059
RESULTS=$(find /tmp/paid-report -name '*.json' ! -name 'manifest.json' ! -name 'slice-*.json' ! -name '_partial*' 2>/dev/null | sort)
TOTAL=0; PASSED=0; FAILED=0; FLAKY=0; COST="0"
SUITE_LINES=""
for f in $RESULTS; do
if ! jq -e '.total_tests' "$f" >/dev/null 2>&1; then
echo "Skipping malformed JSON: $f"
continue
fi
# FINAL-attempt accounting: eval-store keeps EVERY retry attempt
# as its own record (that's the flake telemetry), so counting raw
# records marks a pass-on-retry as a failure and inflates totals.
# Group by test name, judge the LAST record — flaky passes render
# as the ⚠ line, never as ❌ (WS1 policy: recorded, not blocking).
# Guarded: a file with total_tests but a null/non-array `tests`
# passes the -e probe, the group_by then fails, and an empty $T
# would abort the whole step under bash -e ([ "" -eq 0 ] is an
# error) — killing the comment on exactly the corrupted-artifact
# runs where the red evidence matters (claude adversarial).
STATS=$(jq -r '[.tests | group_by(.name)[] | last] as $final | "\($final | length) \([$final[] | select(.passed)] | length) \([$final[] | select(.passed | not)] | length) \(.flaky_retries // [] | length)"' "$f" 2>/dev/null) || { echo "Skipping malformed tests[] in: $f"; continue; }
read -r T P F FL <<< "$STATS"
[ -z "$T" ] && { echo "Skipping malformed tests[] in: $f"; continue; }
C=$(jq -r '.total_cost_usd // 0' "$f")
TIER=$(jq -r '.tier // "unknown"' "$f")
SHARD=$(jq -r '.shard // "-"' "$f")
[ "$T" -eq 0 ] && continue
TOTAL=$((TOTAL + T))
PASSED=$((PASSED + P))
FAILED=$((FAILED + F))
FLAKY=$((FLAKY + FL))
COST=$(echo "$COST + $C" | bc)
STATUS_ICON="✅"
[ "$F" -gt 0 ] && STATUS_ICON="❌"
[ "$F" -eq 0 ] && [ "$FL" -gt 0 ] && STATUS_ICON="✅⚠"
SUITE_LINES="${SUITE_LINES}| ${TIER}/${SHARD} | ${P}/${T} | ${STATUS_ICON} | \$${C} |\n"
done
STATUS="✅ PASS"
if [ "${RECONCILE_EXIT:-1}" != "0" ] || [ "$FAILED" -gt 0 ]; then STATUS="❌ FAIL"; fi
BODY="## E2E Evals: ${STATUS}
**${PASSED}/${TOTAL}** tests passed | **\$${COST}** total cost | reconcile exit: ${RECONCILE_EXIT:-missing}$([ "$FLAKY" -gt 0 ] && printf ' | ⚠ %s flaky pass(es) — recorded, not blocking' "$FLAKY")
| Shard | Result | Status | Cost |
|-------|--------|--------|------|
$(echo -e "$SUITE_LINES")
<details><summary>Fail-closed reconciliation</summary>
\`\`\`
$(tail -c 4000 /tmp/verdict/report.txt 2>/dev/null || echo '(no reconciliation output)')
\`\`\`
</details>
---
*Sliced lane: diff-selected gate census via scripts/test-paid-shards.ts (planner → 6 executors → fail-closed report)*"
if [ "$FAILED" -gt 0 ]; then
FAILURES=""
for f in $RESULTS; do
if ! jq -e '.failed' "$f" >/dev/null 2>&1; then continue; fi
FAILS=$(jq -r '[.tests | group_by(.name)[] | last | select(.passed == false)][] | "- ❌ \(.name): \(.exit_reason // "unknown")"' "$f" 2>/dev/null || echo "- ⚠️ parse error")
FAILURES="${FAILURES}${FAILS}\n"
done
BODY="${BODY}
### Failures
$(echo -e "$FAILURES")"
fi
COMMENT_ID=$(gh api repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments \
--jq '.[] | select(.body | startswith("## E2E Evals")) | .id' | tail -1)
if [ -n "$COMMENT_ID" ]; then
gh api "repos/${{ github.repository }}/issues/comments/${COMMENT_ID}" \
-X PATCH -f body="$BODY"
else
# REST, not gh's pr-comment subcommand: this job runs with NO
# checkout (the token/exec split), and that subcommand resolves
# the repo FROM git — it dies with "not a git repository" here.
gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments" \
-X POST -f body="$BODY"
fi
.github/workflows/free-tests.yml
+23 -2
View File
@@ -48,7 +48,7 @@ jobs:
runs-on: ubicloud-standard-8
timeout-minutes: 20
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
@@ -129,6 +129,27 @@ jobs:
run: xvfb-run -a bun run test:free
env:
GSTACK_EXPECT_BINARIES: "1"
# WS1 flake telemetry: a single timing flake must not red the only
# required lane — the runner's attribution-gated retry pass (cap 5,
# truncation veto) re-runs failing files once, serially, and a
# clean retry downgrades to a LOUD flaky-pass. Every flaky-pass is
# appended to the ledger (single writer: the parent runner) and
# uploaded below, so repeat offenders are an enumerable series —
# recorded and ranked, never masked. Pinned by
# test/free-tests-workflow-wiring.test.ts.
GSTACK_FREE_RETRY_FLAKY: "1"
GSTACK_FLAKE_LEDGER: ${{ runner.temp }}/flake-ledger.jsonl
# Uploaded unconditionally (not just on failure): a flaky-pass run is
# GREEN — that's the point — so its evidence must survive green runs.
- name: Upload flake ledger
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: flake-ledger
path: ${{ runner.temp }}/flake-ledger.jsonl
if-no-files-found: ignore
retention-days: 90
# The runner streams the full child output to per-run logs under the OS
# tmpdir and prints only the quiet contract to the console. Without this
@@ -137,7 +158,7 @@ jobs:
# need a local re-run, which fork contributors can't do on this image.
- name: Upload shard logs on failure
if: failure()
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: free-test-shard-logs
path: /tmp/gstack-free-test-*.log
ARCHITECTURE.md
+1 -1
View File
@@ -424,7 +424,7 @@ The `EvalCollector` accumulates test results and writes them in two ways:
1. **Incremental:** `savePartial()` writes `_partial-e2e.json` after each test (atomic: write `.tmp`, `fs.renameSync`). Survives kills.
2. **Final:** `finalize()` writes a timestamped eval file (e.g. `e2e-20260314-143022.json`). The partial file is never cleaned up — it persists alongside the final file for observability.
`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack/projects/<slug>/evals/` (legacy fallback `~/.gstack-dev/evals/`). Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `<evalDir>/shards/<slug>/` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards/<slug>/` subdirectories. Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline.
`eval:compare` diffs two eval runs. `eval:summary` aggregates stats across all runs in `~/.gstack/projects/<slug>/evals/` (legacy fallback `~/.gstack-dev/evals/`). Both are shard-aware (v1.63.0.0): the sharded paid runner (`scripts/test-paid-shards.ts`, run via `test:gate:sharded` / `test:periodic:sharded` — the `eval:bg:gate` / `eval:bg:periodic` scripts now point at these) gives each shard's collector its own directory at `<evalDir>/shards/<slug>/` through the `GSTACK_EVAL_DIR` env var (honored by the `EvalCollector` constructor), and `eval:list` / `eval:compare` / `eval:summary` scan one level of `shards/<slug>/` subdirectories (`eval:flake-rank` reads the same tree recursively, plus the free-suite flake ledger). Baseline lookups exclude `_partial` accumulators (`isPartialEval` / `findLatestFinalizedRun` in `eval-store.ts`), so auto-comparison never uses the current run's own partial file as its baseline.
### Test tiers
CHANGELOG.md
+56
View File
@@ -1,5 +1,61 @@
# Changelog
## [1.77.0.0] - 2026-08-31
**Every PR stops paying for evals twice.**
**Flakes are now measured, killed at the root, and fenced.**
The test infrastructure got its overhaul, wave 1. The legacy 17-row eval matrix that ran serialized AHEAD of the sliced lane on every PR is deleted: one paid lane, its gate census derived from the runner itself, so a new gate test is in the census the moment its file lands. No hand-enumerated rows to drift, and the drift already tried, a new matrix row landed on main mid-branch and the merge resolved to the derived census that covers it by construction.
The flake war moved from anecdotes to instruments. Every retried pass is now recorded where it cannot hide (bun's own output shows a retry as a clean pass, we probed it), the free lane retries a failing file once, loudly, and appends every flaky pass to a per-project ledger uploaded from CI on green runs. `bun run eval:flake-rank` ranks the series. And the wedge class that hit main, a hung child under a blocking spawnSync that no in-process timeout can interrupt, is extinct: 499 timeout-less sync-spawn sites across 146 files (the branch tripwire's own count against main) swept to zero, with a ratcheted tripwire that failed its first real offender the day a timeout-less spawn arrived from a merge.
### The numbers that matter
Source: CI runs 33263204465 / 33262077256 (measured 2026-08-29), the repo census (`test/helpers/touchfiles-data.ts`), and the sweep tripwire (`test/spawnsync-timeout-tripwire.test.ts`).
| Metric | Before | After | Δ |
|---|---|---|---|
| Paid lanes per PR | 2 (serialized) | 1 | eval wall 35.5 → ~13 min (target) |
| Measured duplicate spend per PR | ~$20.94 | $0 | matrix deleted |
| Gate census keys | 86 (6 phantoms) | 77, all provably alive | reverse invariant enforces |
| Sync spawns that can wedge a shard | 499 unbounded | 0 (8 reasoned exemptions) | tripwire-ratcheted |
| Local gate worst case | ~6.5 h (4×4 jobs) | ~3.3 h (8×2) | isolation landed first |
| Retried-pass visibility | invisible | recorded + ranked | `eval:flake-rank` |
The phantom-census number is the quiet one that matters: six merge-blocking "tests" existed only as map keys. They are deleted, and a key without a living test now fails the free suite.
What this means for anyone shipping here: PRs get one honest paid verdict faster and cheaper, a flaky pass never blocks your merge but never disappears either, and a test that hangs takes down thirty seconds, not a shard. Run `bun run eval:flake-rank` when you want the flake ledger's verdict.
### Itemized changes
#### Added
- **Flake telemetry, end to end.** Eval-store records every retry attempt (`attempt`, `flaky_retries`), the paid report lists passed-only-on-retry tests as warnings, the free lane's flaky-retry pass is ON in CI with a single-writer JSONL ledger (branch + sha attributed, per-project by default) uploaded as an artifact on every run, and `bun run eval:flake-rank` aggregates the series with final-attempt accounting and a 60-day recency bound.
- **Two-phase session timeouts.** A silent API dies at a startup grace (90s local, 300s CI floor, a real `Math.max` floor) with the distinct reason `timeout_startup` instead of burning a 600s work budget into an opaque `0 turns / $0.00` failure; the work budget arms on the first byte and the total wall never grows.
- **Green-by-skip census.** "Ran N tests" counts skips, so a codex/gemini file whose every test self-skipped used to read as coverage. The classifier now parses bun's skip/pass recap and the paid report labels an all-skipped pass "verified nothing."
- **Sync-spawn timeout tripwire** (spawnSync / execSync / execFileSync / Bun.spawnSync, comment-aware, 30-line window, shrink-only exemption ratchet) plus a raw-SHA fixture ban (`git show <sha>:path` fixtures must be vendored; the one live offender now reads a committed fixture).
- **Behavioral kill-semantics tests**: a fake-claude shim proves a timed-out session leaves neither the CLI nor its grandchild alive; two gstack-detach watchdog tests cover TERM-immune grandchildren and the leader-dies-first case.
- **CLI version stamping**: every eval run records `claude --version` (resolved once in the runner parent), so the next TUI-drift flake hunt is a grep, not archaeology.
#### Changed
- **One paid lane.** The legacy evals.yml matrix is deleted (pure deletion, one revert restores it) after a static parity receipt: the sliced lane's 49-file derived census strictly contained the matrix's 18 files. The PR comment moved into the sliced lane with final-attempt accounting and a fail-closed reconciliation verdict.
- **Paid runner defaults 4×4 → 8×2**: ~10-13 real in-flight sessions (under the documented-safe 15) instead of ~4-6; per-shard TMPDIR/Chromium-profile isolation and a kill-path cleanup backstop landed first, deliberately.
- **CI setup deduplicated into four composite actions**; the register-skills composite carries the fail-fast dangling-symlink verification loop that only the deleted matrix copy had, so the surviving lanes inherit it. Rerun-safe, frozen-lockfile fallback, input-validated.
- **Supply chain pinned**: the claude CLI in the CI image is an exact version (bumps ride PRs that run the PTY gate, ending the weekly-latest drift that broke the harness three times), and every action in the secrets-bearing and image-publishing workflows is SHA-pinned.
- **Routing journeys lost their answer key**: the fixture no longer ships a prompt→skill lookup table, so a regressed skill description can actually fail the test again, at roughly half the previous per-journey cost (2 turns, [Skill, Read] only).
- Decided A/B experiments retired (auq-repetition-cut, preamble-script, the opus-47 single-run fanout comparison): one-shot questions answered months ago no longer re-run weekly as coin flips. `plan-ceo-review-expansion-energy` and `ios-qa-e2e` moved to the periodic tier with reasons.
#### Fixed
- **A fail-open reconcile gate**: GitHub's default run-step shell has no pipefail, so the fail-closed report's exit was read from `tee` (always 0) in both paid lanes. Now `PIPESTATUS[0]`, pinned by a wiring test.
- **A write-token trust boundary**: the job that executes PR-authored code no longer holds the PR-comment write token; commenting moved to a job that runs zero repo code.
- **Provider-runner orphans**: timeouts kill the whole process group (claude, codex, gemini, and gstack-detach's watchdog with the group id captured at spawn), and the codex/gemini runners inherited the orphan-drain hardening only the claude copy had. The observed 600s-timeout-stretching-past-1400s class is gone, with a regression net.
- **Selection integrity**: 17 phantom selection keys deleted (a reverse invariant now requires every key to name a living test), gitignored `.agents/**` dep patterns that could never match a git diff replaced with their generators, and the codex/gemini local touchfile forks now derive from the canonical map.
- **A cross-shard SKILL.md race**: the opus-47 eval regenerated the live tree's skill files mid-run; it now renders into a scratch dir via `--out-dir`.
#### For contributors
- `bun run eval:flake-rank` (with `--json`, `--dir`, `--since-days`) is the promotion-clock dial; the WS16 required-check decision reads it.
- The overhaul plan (16 workstreams, reviewed by CEO + eng passes with two cross-model outside voices) continues: budget-aware shard walls, PTY readiness events, free-suite splits, judge determinism, and required-check promotion are the next waves.
## [1.76.0.0] - 2026-08-31
**Ship's doc-sync now survives Conductor.**
CLAUDE.md
+4 -3
View File
@@ -22,6 +22,7 @@ bun run dev:skill # watch mode: auto-regen + validate on change
bun run eval:list # list all eval runs from ~/.gstack/projects/<slug>/evals/
bun run eval:compare # compare two eval runs (auto-picks most recent)
bun run eval:summary # aggregate stats across all eval runs
bun run eval:flake-rank # rank tests by flake signal (retried passes first; --json, --dir, --since-days)
bun run slop # full slop-scan report (all files)
bun run slop:diff # slop findings in files changed on this branch only
```
@@ -647,9 +648,9 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
floor enforced against the live shard census by
test/eval-detach-timeout-floor.test.ts)
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
process count (default 4); `EVALS_CONCURRENCY` is bun's --max-concurrency
WITHIN a shard (default 4) — they are deliberately separate knobs. `eval:list` / `eval:compare` /
`eval:summary` read the shard dirs too. Or call
process count (default 8); `EVALS_CONCURRENCY` is bun's --max-concurrency
WITHIN a shard (default 2) — they are deliberately separate knobs. `eval:list` / `eval:compare` /
`eval:summary` / `eval:flake-rank` read the shard dirs too. Or call
`gstack-detach [--lock NAME] [--timeout SECS] [--label LBL] --
<cmd>` directly for any long agent job. Export `ANTHROPIC_API_KEY` first (never
pass keys in argv).
CONTRIBUTING.md
+7 -4
View File
@@ -168,7 +168,9 @@ silent truncation can never report green. Pass `--verbose` to forward the full
child stream; `--wall-timeout <secs>` overrides the per-shard kill deadline.
`GSTACK_FREE_JOBS=<n>` overrides the shard count (digits only, loud on garbage),
and `GSTACK_FREE_RETRY_FLAKY=1` opts into one serial retry pass for
syscall-supervised sandboxes (off by default — dev boxes should see flakes).
syscall-supervised sandboxes (off by default locally — dev boxes should see
flakes; the required CI free lane turns it on and uploads every flaky pass
in a JSONL ledger artifact that `bun run eval:flake-rank` folds in).
Working in a cloud sandbox? Run `scripts/sandbox-doctor.sh` once per boot to
make the suite run green (details in
[docs/TESTING_INTERNALS.md](docs/TESTING_INTERNALS.md)).
@@ -236,6 +238,7 @@ When E2E tests run, they produce machine-readable artifacts in `~/.gstack-dev/`:
bun run eval:list # list all eval runs (turns, duration, cost per run)
bun run eval:compare # compare two runs — shows per-test deltas + Takeaway commentary
bun run eval:summary # aggregate stats + per-test efficiency averages across runs
bun run eval:flake-rank # rank tests by flake signal: retried passes first, then failure rate (--json, --dir, --since-days)
```
**Detached runs for agents and long suites.** When an agent (or you, for a run
@@ -264,9 +267,9 @@ distinguishes failed vs timed-out vs never-started shards. The runner also
selects by diff: shards untouched by your branch are reported as
skipped-by-diff, with a selection banner naming the reason (`EVALS_ALL=1`
forces everything). `EVALS_JOBS` sets how many shard processes run at once
(default 4); `EVALS_CONCURRENCY` is bun's concurrency WITHIN a shard — they
are deliberately separate knobs. `eval:list`,
`eval:compare`, and `eval:summary` are shard-aware. Humans running
(default 8); `EVALS_CONCURRENCY` is bun's concurrency WITHIN a shard
(default 2) — they are deliberately separate knobs. `eval:list`,
`eval:compare`, `eval:summary`, and `eval:flake-rank` are shard-aware. Humans running
`bun run test:evals` foreground in their own terminal don't need this — Ctrl-C
is intended there.
TODOS.md
+21 -9
View File
@@ -567,14 +567,15 @@ duration-packed free shards, the sharded paid runner as the CI engine
coverage contract + gate census, eval-budget timeout tiers, and the
coverage fill. Remaining, in rough priority order:
- **P1 — Delete the legacy evals.yml matrix after parity.** The sliced lane
runs alongside the 18-row matrix (`needs: evals`, so provider concurrency
never doubles). After 1-2 PR cycles of parity (compare executed-test sets:
intersection strict + the 8 KNOWN_MATRIX_GAPS files as expected additions;
stochastic outcomes informational), delete the matrix as a PURE-DELETION
commit (one revert restores it), drop the `needs: evals` edge, rewrite
test/evals-workflow-matrix.test.ts into a runner-wiring pin, and retire
KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET wholesale. Effort S.
- **DONE (v1.77.0.0 test-infra wave 1) — Delete the legacy evals.yml matrix after
parity.** Deleted as a pure-deletion commit (one revert restores it) after
a static parity receipt: sliced gate census (49 files) ⊇ matrix files (18),
31 files of extra coverage. `needs: evals` edge dropped, PR comment moved
into slices-report, KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET retired,
test/evals-workflow-matrix.test.ts rewritten as
test/evals-workflow-wiring.test.ts. The register-skills fail-fast
verification loop was ported to the surviving lanes FIRST via the shared
.github/actions/register-gstack-skills composite.
- **P1 — Maintainer decision: make `slices-report` a required check** once
post-migration flake data exists (the Codex outside-voice's "green means
green is not delivered while paid stays advisory" point — correct, and
@@ -812,7 +813,18 @@ SKILL.md untouched). `bun test` is green again.
## Scope-gate follow-ups (filed via /plan-eng-review on the plan-mode auto-select-B change)
### P2: SDK eval budgets charge API-queue latency to the work budget — pick a structural fix
### DONE (v1.77.0.0) — SDK eval budgets charge API-queue latency to the work budget
**Shipped shape:** the two-phase timer landed WITHOUT the codemod this entry
feared: the total wall stays <= timeout (work phase = remainder after first
byte), so every outer/inner bun-timeout relationship is untouched; a silent
API now dies EARLY at the startup grace (90s local / 300s CI floor, enforced
Math.max) with the distinct reason 'timeout_startup'. Option (b)'s 300s CI
floor is in (test/session-runner-startup-grace.test.ts pins it). The
budget-EXTENSION variant (work budget = full timeout from first byte, which
DOES need the tier/wall reshape) remains wave-2 scope in the overhaul plan.
Original entry follows for context:
**What:** `runSkillTest`'s single `setTimeout(timeout)` arms at spawn, so session
startup AND the model's first-completion queue time are charged against the
VERSION
+1 -1
View File
@@ -1 +1 @@
1.76.0.0
1.77.0.0
agents-digest/gstack-AGENTS.md
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.76.0.0 — regenerate/re-copy after upgrading gstack
# gstack digest v1.77.0.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows,
bin/gstack-detach
+18 -3
View File
@@ -96,20 +96,35 @@ def child_run(args, log):
proc = subprocess.Popen(
cmd, stdout=f, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, start_new_session=True
)
# Capture the PGID AT SPAWN (== proc.pid: start_new_session makes the
# child a session/group leader). Resolving it later via
# os.getpgid(proc.pid) raises ESRCH once the leader exits — a leader
# that died on the SIGTERM while a TERM-immune grandchild survived
# left that grandchild alive forever (codex adversarial finding).
pgid = proc.pid
if args.timeout and args.timeout > 0:
try:
code = proc.wait(timeout=args.timeout)
except subprocess.TimeoutExpired:
log_line(log, f"### gstack-detach WATCHDOG fired after {args.timeout}s — killing ### {_now()}")
try:
os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
os.killpg(pgid, signal.SIGTERM)
except Exception:
pass
time.sleep(5)
# Group SIGKILL after the grace, not a direct-child kill:
# eval runs spawn claude/codex grandchildren that survive a
# proc.kill() and burn cores + API for hours (the observed
# 15-hour-orphan class). ESRCH here just means the group
# honored the SIGTERM. Uses the SAVED pgid so a dead leader
# cannot orphan its group.
try:
proc.kill()
os.killpg(pgid, signal.SIGKILL)
except Exception:
pass
try:
proc.kill()
except Exception:
pass
code = "timeout"
else:
code = proc.wait()
browse/test/build.test.ts
+1 -1
View File
@@ -13,7 +13,7 @@ describe('build: server-node.mjs', () => {
// Skip rather than fail so plain `bun test` without a prior build passes.
return;
}
expect(() => execSync(`node --check ${SERVER_NODE}`, { stdio: 'pipe' })).not.toThrow();
expect(() => execSync(`node --check ${SERVER_NODE}`, { stdio: 'pipe', timeout: 30_000 })).not.toThrow();
});
test('does not inline @ngrok/ngrok (must be external)', () => {
browse/test/bun-polyfill.test.ts
+14 -14
View File
@@ -27,7 +27,7 @@ describe('bun-polyfill', () => {
const elapsed = Date.now() - start;
console.log(elapsed >= 40 ? 'OK' : 'TOO_FAST');
})();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.stdout.toString().trim()).toBe('OK');
expect(result.exitCode).toBe(0);
});
@@ -38,7 +38,7 @@ describe('bun-polyfill', () => {
const r = Bun.spawnSync(['echo', 'hello'], { stdout: 'pipe' });
console.log(r.stdout.toString().trim());
console.log('exit:' + r.exitCode);
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const lines = result.stdout.toString().trim().split('\n');
expect(lines[0]).toBe('hello');
expect(lines[1]).toBe('exit:0');
@@ -51,7 +51,7 @@ describe('bun-polyfill', () => {
console.log(typeof p.pid === 'number' ? 'HAS_PID' : 'NO_PID');
console.log(typeof p.kill === 'function' ? 'HAS_KILL' : 'NO_KILL');
console.log(typeof p.unref === 'function' ? 'HAS_UNREF' : 'NO_UNREF');
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const lines = result.stdout.toString().trim().split('\n');
expect(lines[0]).toBe('HAS_PID');
expect(lines[1]).toBe('HAS_KILL');
@@ -70,7 +70,7 @@ describe('bun-polyfill', () => {
console.log(typeof p.exited === 'object' && typeof p.exited.then === 'function' ? 'IS_PROMISE' : 'NOT_PROMISE');
console.log('exit:' + await p.exited);
})();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const lines = result.stdout.toString().trim().split('\n');
expect(lines[0]).toBe('IS_PROMISE');
expect(lines[1]).toBe('exit:0');
@@ -83,7 +83,7 @@ describe('bun-polyfill', () => {
const p = Bun.spawn(['node', '-e', 'process.exit(3)'], { stdio: ['ignore', 'ignore', 'ignore'] });
console.log('exit:' + await p.exited);
})();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.stdout.toString().trim()).toBe('exit:3');
});
@@ -100,7 +100,7 @@ describe('bun-polyfill', () => {
const out = await new Response(p.stdout).text();
console.log(out + ':' + code);
})();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.stdout.toString().trim()).toBe('ready:0');
});
@@ -120,7 +120,7 @@ describe('bun-polyfill', () => {
]).catch(() => 'TIMEOUT');
console.log('exit:' + code);
})();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
// Anything other than 'TIMEOUT' (and ideally a non-zero number) means the
// lifecycle promise resolved on the spawn error.
const out = result.stdout.toString().trim();
@@ -139,7 +139,7 @@ describe('bun-polyfill', () => {
setTimeout(() => p.kill('SIGTERM'), 150);
console.log('exit:' + await p.exited);
})();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
// SIGTERM = 15 → 128 + 15 = 143.
expect(result.stdout.toString().trim()).toBe('exit:143');
});
@@ -166,7 +166,7 @@ describe('bun-polyfill', () => {
const out = await new Response(p.stdout).text();
console.log(out.length + ':' + code);
})();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.stdout.toString().trim()).toBe('1024:0');
});
@@ -197,7 +197,7 @@ describe('bun-polyfill', () => {
const out = await new Response(p.stdout).text();
console.log(out.length + ':' + code);
})().catch((e) => { console.log('THREW:' + e.message); });
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.stdout.toString().trim()).toBe('1048576:0');
}, 15000);
@@ -217,7 +217,7 @@ describe('bun-polyfill', () => {
console.log(typeof server.stop === 'function' ? 'HAS_STOP' : 'NO_STOP');
console.log(typeof server.port === 'number' ? 'HAS_PORT' : 'NO_PORT');
server.stop();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const lines = result.stdout.toString().trim().split('\n');
expect(lines[0]).toBe('HAS_STOP');
expect(lines[1]).toBe('HAS_PORT');
@@ -237,7 +237,7 @@ describe('bun-polyfill', () => {
require(${JSON.stringify(polyfillPath)});
Bun.spawn(['node', '-e', ''], { stdio: ['ignore', 'ignore', 'ignore'] });
console.log('windowsHide:' + seen.windowsHide);
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.stdout.toString().trim()).toBe('windowsHide:true');
});
@@ -250,7 +250,7 @@ describe('bun-polyfill', () => {
require(${JSON.stringify(polyfillPath)});
Bun.spawnSync(['node', '-e', '']);
console.log('windowsHide:' + seen.windowsHide);
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.stdout.toString().trim()).toBe('windowsHide:true');
});
@@ -263,7 +263,7 @@ describe('bun-polyfill', () => {
require(${JSON.stringify(polyfillPath)});
Bun.spawn(['node', '-e', ''], { stdio: ['ignore', 'ignore', 'ignore'], windowsHide: false });
console.log('windowsHide:' + seen.windowsHide);
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
expect(result.stdout.toString().trim()).toBe('windowsHide:false');
});
});
browse/test/config.test.ts
+8 -8
View File
@@ -85,7 +85,7 @@ describe('config', () => {
// it (the exact bug the fix removed) would skip the guard here.
const tmpDir = path.join(os.tmpdir(), `browse-gitignored-repo-test-${Date.now()}`);
fs.mkdirSync(tmpDir, { recursive: true });
Bun.spawnSync(['git', 'init'], { cwd: tmpDir, stdout: 'ignore', stderr: 'ignore' });
Bun.spawnSync(['git', 'init'], { cwd: tmpDir, stdout: 'ignore', stderr: 'ignore', timeout: 30_000 });
fs.writeFileSync(path.join(tmpDir, '.gitignore'), '.gstack/\n');
const config = resolveConfig({ BROWSE_STATE_FILE: path.join(tmpDir, '.gstack', 'browse.json') });
ensureStateDir(config);
@@ -166,22 +166,22 @@ describe('config', () => {
fs.mkdirSync(tmpDir, { recursive: true });
// Set up a real git repo
spawnSync('git', ['init', '-q'], { cwd: tmpDir });
spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir });
spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir });
spawnSync('git', ['init', '-q'], { cwd: tmpDir, timeout: 30_000 });
spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, timeout: 30_000 });
spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, timeout: 30_000 });
// Write a global excludes file that ignores .gstack/
const excludesFile = path.join(tmpDir, 'global-gitignore');
fs.writeFileSync(excludesFile, '.gstack/\n');
spawnSync('git', ['config', 'core.excludesFile', excludesFile], { cwd: tmpDir });
spawnSync('git', ['config', 'core.excludesFile', excludesFile], { cwd: tmpDir, timeout: 30_000 });
// .gitignore exists but does NOT contain .gstack/
fs.writeFileSync(path.join(tmpDir, '.gitignore'), 'node_modules/\n');
spawnSync('git', ['add', '.gitignore'], { cwd: tmpDir });
spawnSync('git', ['commit', '-qm', 'init'], { cwd: tmpDir });
spawnSync('git', ['add', '.gitignore'], { cwd: tmpDir, timeout: 30_000 });
spawnSync('git', ['commit', '-qm', 'init'], { cwd: tmpDir, timeout: 30_000 });
// Verify git knows .gstack/ is ignored
const check = spawnSync('git', ['check-ignore', '-q', '.gstack/'], { cwd: tmpDir });
const check = spawnSync('git', ['check-ignore', '-q', '.gstack/'], { cwd: tmpDir, timeout: 30_000 });
expect(check.status).toBe(0);
const config = resolveConfig({ BROWSE_STATE_FILE: path.join(tmpDir, '.gstack', 'browse.json') });
browse/test/file-permissions.test.ts
+2 -2
View File
@@ -87,7 +87,7 @@ describe('restrictDirectoryPermissions', () => {
fs.mkdirSync(d);
// System chmod, not fs.chmodSync: Bun masks the sticky bit off chmod/
// mkdir modes, so 0o1777 through the fs API lands as 0o777.
Bun.spawnSync(['chmod', '1777', d]);
Bun.spawnSync(['chmod', '1777', d], { timeout: 30_000 });
expect(fs.statSync(d).mode & 0o7777).toBe(0o1777); // fixture took
restrictDirectoryPermissions(d);
expect(fs.statSync(d).mode & 0o7777).toBe(0o1777);
@@ -244,7 +244,7 @@ describe('mkdirSecure', () => {
fs.mkdirSync(d);
// System chmod: Bun's fs API masks the sticky bit off modes (see the
// restrictDirectoryPermissions sticky-dir test).
Bun.spawnSync(['chmod', '1777', d]);
Bun.spawnSync(['chmod', '1777', d], { timeout: 30_000 });
expect(fs.statSync(d).mode & 0o7777).toBe(0o1777); // fixture took
mkdirSecure(d);
expect(fs.statSync(d).mode & 0o7777).toBe(0o1777);
browse/test/findport.test.ts
+3 -3
View File
@@ -160,7 +160,7 @@ describe('findPort / isPortAvailable', () => {
}
test();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const output = result.stdout.toString().trim();
// Confirms the polyfill's stop() is fire-and-forget — callers
@@ -171,7 +171,7 @@ describe('findPort / isPortAvailable', () => {
test('net.createServer approach does not have the race condition', async () => {
// Prove the fix: net.createServer with proper async bind/close
// releases the port cleanly
const result = Bun.spawnSync(['node', '-e', `
const result = Bun.spawnSync(['node', '-e', ` // timeout in trailing options
const net = require('net');
async function testFix() {
@@ -205,7 +205,7 @@ describe('findPort / isPortAvailable', () => {
}
testFix();
`], { stdout: 'pipe', stderr: 'pipe' });
`], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
const output = result.stdout.toString().trim();
expect(output).toBe('FIX_WORKS');
browse/test/gstack-config.test.ts
+1
View File
@@ -31,6 +31,7 @@ function run(args: string[] = [], extraEnv: Record<string, string> = {}) {
env,
stdout: 'pipe',
stderr: 'pipe',
timeout: 30_000,
});
return {
exitCode: result.exitCode,
browse/test/gstack-update-check.test.ts
+1
View File
@@ -34,6 +34,7 @@ function run(extraEnv: Record<string, string> = {}, args: string[] = []) {
env,
stdout: 'pipe',
stderr: 'pipe',
timeout: 30_000,
});
return {
exitCode: result.exitCode,
browse/test/temp-dirs.test.ts
+1 -1
View File
@@ -107,7 +107,7 @@ describe('untrustable TMPDIR values never widen the allowlist', () => {
const r = Bun.spawnSync([
process.execPath, '-e',
"import { TEMP_DIRS } from './browse/src/platform'; console.log(JSON.stringify(TEMP_DIRS));",
], { env: { ...process.env, TMPDIR: tmpdir }, cwd: path.resolve(import.meta.dir, '..', '..') });
], { env: { ...process.env, TMPDIR: tmpdir }, cwd: path.resolve(import.meta.dir, '..', '..'), timeout: 30_000 });
return JSON.parse(r.stdout.toString().trim().split('\n').pop()!);
};
browse/test/windows-spawn-hide.test.ts
+1
View File
@@ -70,6 +70,7 @@ describe('windowsHide on Windows-reachable spawns (#1835)', () => {
const EXEMPT: Array<{ file: string; needle: string; reason: string }> = [
{
file: 'domain-skill-commands.ts',
// tripwire-exempt: grep NEEDLE string for this census, not a call
needle: 'spawnSync(editor',
reason: "interactive $EDITOR with stdio:'inherit' — windowsHide would detach a console editor into an invisible console",
},
browse/test/xvfb.test.ts
+1 -1
View File
@@ -11,7 +11,7 @@ import {
const HAS_XVFB = (() => {
if (process.platform !== 'linux') return false;
const result = Bun.spawnSync(['which', 'Xvfb'], { stdout: 'pipe', stderr: 'pipe' });
const result = Bun.spawnSync(['which', 'Xvfb'], { stdout: 'pipe', stderr: 'pipe', timeout: 30_000 });
return result.exitCode === 0;
})();
docs/PROJECT_STRUCTURE.md
+2 -1
View File
@@ -28,6 +28,7 @@ gstack/
│ ├── skill-check.ts # Health dashboard
│ ├── test-free-shards.ts # Strict parallel free-suite runner (GSTACK_FREE_JOBS, opt-in flaky retry)
│ ├── test-paid-shards.ts # Sharded paid-tier runner (one Bun process per shard)
│ ├── eval-flake-rank.ts # Flake-telemetry dial: ranks tests by retried passes across eval runs + the free-lane ledger
│ ├── sandbox-doctor.sh # One-command cloud-sandbox fixer: makes the free suite run green
│ └── dev-skill.ts # Watch mode
├── test/ # Skill validation + eval tests
@@ -71,7 +72,7 @@ gstack/
├── patches/ # bun `patchedDependencies` patches (playwright-core windowsHide)
├── docs/designs/ # Design documents
├── setup-deploy/ # /setup-deploy skill (one-time deploy config)
├── .github/ # CI workflows + Docker image
├── .github/ # CI workflows + shared composite actions (.github/actions/) + Docker image (claude CLI pinned)
│ ├── workflows/ # evals.yml (E2E on Ubicloud), quality-gate.yml (secret scan), dependency-review.yml, osv-scanner.yml, skill-docs.yml, actionlint.yml, and 8 more (windows, periodic evals, release gates, ci-image)
│ └── docker/ # Dockerfile.ci (pre-baked toolchain + Playwright/Chromium)
├── contrib/ # Contributor-only tools (never installed for users)
docs/TESTING_INTERNALS.md
+41 -6
View File
@@ -68,6 +68,18 @@ failure), never-started/timed-out taxonomy, and parent-computed diff
selection propagated to children via `EVALS_SELECTION_JSON` (fail-open: a
child that can't parse it recomputes locally with one warning). Retry parity
lives in `RETRY_OVERRIDES` (literals; old matrix rows' earned `retries: 2`).
Flake telemetry rides the store: every recorded test carries its 1-based
`attempt` (a pass-on-attempt-2 stays visible forever — bun's own stream hides
it), runs list `flaky_retries`, the report warns on passed-only-on-retry
tests, and `bun run eval:flake-rank` ranks the series (retried passes first,
then failure rate; 60-day recency bound on eval files; the free lane's flake
ledger is folded in from `flakeLedgerPath()` — override with
`GSTACK_FLAKE_LEDGER`, the same env var the CI free lane sets before
uploading the ledger as the `flake-ledger` artifact). Census integrity is
enforced from the free suite: every `E2E_TOUCHFILES` / `LLM_JUDGE_TOUCHFILES`
key must name a living paid test (`test/touchfiles.test.ts`'s reverse
invariant), and `git show <sha>:path` fixtures are banned — vendor the bytes
instead (`test/git-ref-fixture-tripwire.test.ts`).
**CI planner/executor/report.** `--emit-plan <path> --slices K` computes
selection + the slice plan ONCE (killing per-slice selector divergence);
@@ -76,19 +88,39 @@ slice-result artifacts; `--report <dir>` reconciles them FAIL-CLOSED (a slice
whose artifact never landed, or a planned shard nobody reported, is a
failure). Under `EVALS_ALL` the hollow-shard guard marks exit-0 shards with
ZERO executed tests `passed-empty` (a failure) — census-health, not just
test runs. evals.yml runs the sliced gate lane per PR (parity phase:
alongside the legacy matrix, `needs:`-sequenced so provider concurrency
never doubles; the matrix and its `KNOWN_MATRIX_GAPS`/`KNOWN_TIER_UNSET`
ratchets are deleted after demonstrated parity). evals-periodic.yml runs ALL
test runs. evals.yml runs the sliced gate lane per PR — the ONLY paid lane
since the legacy 17-row matrix (22.6 min/$21 per PR serialized ahead of the
slices) was deleted after demonstrated parity; its
`KNOWN_MATRIX_GAPS`/`KNOWN_TIER_UNSET` ratchets retired with it and
`test/evals-workflow-wiring.test.ts` pins the surviving wiring (slice-count
agreement, tier consistency, the shared register-skills composite with its
fail-fast verification loop). evals-periodic.yml runs ALL
periodic-tier files weekly (the coverage contract) minus the reasoned
exclusions in `test/helpers/periodic-exclude-data.ts` (reason + tracking
required per entry; removal re-activates the file), plus a weekly
`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks.
`EVALS_ALL` gate census, plus a tracking-issue UPSERT on red weeks. The CI
image pins the claude CLI to an exact version (`.github/docker/Dockerfile.ci`,
enforced by `test/ci-image-cli-pin.test.ts` — bumps ride PRs that run the PTY
gate), and every eval-store run records `claude --version`, resolved once in
the runner parent and handed to shard children as `GSTACK_CLAUDE_CLI_VERSION`
(never spawned on a test thread), so a TUI-drift flake hunt is a grep, not
archaeology.
**Timeout policy.** Paid tests use the tiers in
`test/helpers/eval-budgets.ts` (JUDGE/CAPTURE/CAPTURE_LONG/PTY/PTY_LONG);
`test/eval-budgets-policy.test.ts` pins that every tier fits the shard wall
minus overhead and ratchets raw literals. Budget above the wall is fiction.
Session timeouts are two-phase: a silent API dies at the startup grace (90s
local / 300s CI floor, distinct exit reason `timeout_startup`) and the work
budget arms on the first byte — the total wall never grows
(`test/session-runner-startup-grace.test.ts` pins the floor). A timed-out
session kills its whole detached process group (claude, codex, and gemini
runners alike — `test/session-runner-groupkill.test.ts`), so a stray
grandchild can't stretch a 600s budget past 1400s. And sync spawns can't
wedge a shard: every `spawnSync`/`execSync`/`execFileSync`/`Bun.spawnSync`
in the test trees must carry a `timeout`, enforced by
`test/spawnsync-timeout-tripwire.test.ts` with a shrink-only exemption
ratchet.
## Cloud sandboxes (Vercel / Conductor cloud workspaces)
@@ -110,4 +142,7 @@ Two runner knobs exist for these environments (both no-ops unless set):
serial mega-shard and 6-way sharding both saturate the per-process syscall
supervisor), and `GSTACK_FREE_RETRY_FLAKY=1` re-runs attributed failures once
serially, downgrading a clean retry to a loud FLAKY-PASS (capped at 5 files so
a broken tree can't masquerade as flaky).
a broken tree can't masquerade as flaky). The required CI free lane sets the
retry knob too, appending every flaky pass to the JSONL ledger it uploads
(`GSTACK_FLAKE_LEDGER`) — a flaky pass never reds the lane, but it never
disappears either.
ios-qa/daemon/test/cli-mint.test.ts
+2 -1
View File
@@ -14,7 +14,7 @@ const MINT_BIN = join(ROOT, 'bin', 'gstack-ios-qa-mint');
const DAEMON_BIN = join(ROOT, 'bin', 'gstack-ios-qa-daemon');
function runMint(args: string[]) {
return spawnSync(MINT_BIN, args, { stdio: 'pipe', encoding: 'utf-8' });
return spawnSync(MINT_BIN, args, { stdio: 'pipe', encoding: 'utf-8', timeout: 30_000 });
}
describe('bin/gstack-ios-qa-mint launcher', () => {
@@ -112,6 +112,7 @@ describe('bin/gstack-ios-qa-daemon launcher', () => {
stdio: 'pipe',
encoding: 'utf-8',
env: { PATH: '/usr/bin:/bin' },
timeout: 30_000,
});
expect(r.status).not.toBe(0);
expect(r.stderr).toContain('bun');
ios-qa/scripts/gen-accessors.test.ts
+9 -7
View File
@@ -612,6 +612,7 @@ class AppState {
'--output', outputDir,
], {
encoding: 'utf8',
timeout: 30_000,
env: { ...process.env, GSTACK_IOS_CACHE_ROOT: join(workDir, 'cache') },
});
expect(result.status).toBe(4);
@@ -717,7 +718,7 @@ describe('render', () => {
});
test('typechecks beside an internal @Observable app state using a comment marker', () => {
if (spawnSync('swiftc', ['--version'], { encoding: 'utf8' }).status !== 0) return;
if (spawnSync('swiftc', ['--version'], { encoding: 'utf8', timeout: 30_000 }).status !== 0) return;
const coreSource = join(workDir, 'DebugBridgeCore.swift');
const coreModule = join(workDir, 'DebugBridgeCore.swiftmodule');
@@ -752,7 +753,7 @@ public final class StateServer {
'-module-name', 'DebugBridgeCore',
coreSource,
'-emit-module-path', coreModule,
], { encoding: 'utf8' });
], { encoding: 'utf8', timeout: 120_000 });
if (emitModule.status !== 0) {
throw new Error(`failed to build DebugBridgeCore test stub:\n${emitModule.stderr}`);
}
@@ -775,7 +776,7 @@ ${render([{
'-D', 'DEBUG',
'-I', workDir,
appSource,
], { encoding: 'utf8' });
], { encoding: 'utf8', timeout: 120_000 });
if (typecheck.status !== 0) {
throw new Error(`generated accessor failed Swift type checking:\n${typecheck.stderr}`);
}
@@ -783,7 +784,7 @@ ${render([{
test('strict JSON typing and cross-model validate-before-apply restore run correctly', () => {
if (process.platform !== 'darwin') return;
if (spawnSync('swiftc', ['--version'], { encoding: 'utf8' }).status !== 0) return;
if (spawnSync('swiftc', ['--version'], { encoding: 'utf8', timeout: 30_000 }).status !== 0) return;
const coreSource = join(workDir, 'DebugBridgeCore.swift');
const coreModule = join(workDir, 'DebugBridgeCore.swiftmodule');
@@ -830,7 +831,7 @@ public final class StateServer {
'-module-name', 'DebugBridgeCore', coreSource,
'-emit-module-path', coreModule,
'-o', coreLibrary,
], { encoding: 'utf8' });
], { encoding: 'utf8', timeout: 120_000 });
if (emitCore.status !== 0) throw new Error(`failed to build runtime stub:\n${emitCore.stderr}`);
const appSource = join(workDir, 'OptionalRoundTrip.swift');
@@ -915,10 +916,11 @@ struct Runner {
const compile = spawnSync('swiftc', [
'-D', 'DEBUG', '-I', workDir, '-L', workDir, '-lDebugBridgeCore',
'-parse-as-library', appSource, '-o', executable,
], { encoding: 'utf8' });
], { encoding: 'utf8', timeout: 120_000 });
if (compile.status !== 0) throw new Error(`generated Optional accessor failed compilation:\n${compile.stderr}`);
const run = spawnSync(executable, [], {
encoding: 'utf8',
timeout: 30_000,
env: { ...process.env, DYLD_LIBRARY_PATH: workDir },
});
if (run.status !== 0) throw new Error(`generated Optional accessor failed at runtime:\n${run.stderr}`);
@@ -928,7 +930,7 @@ struct Runner {
describe('SwiftSyntax generator parity', () => {
test('isolates canonical markers and rejects inaccessible fields', () => {
if (process.platform !== 'darwin') return;
if (spawnSync('swift', ['--version'], { encoding: 'utf8' }).status !== 0) return;
if (spawnSync('swift', ['--version'], { encoding: 'utf8', timeout: 30_000 }).status !== 0) return;
const packageDir = join(import.meta.dir, 'gen-accessors-tool');
const inputDir = join(workDir, 'swift-syntax-input');
ios-qa/scripts/gen-accessors.ts
+2 -1
View File
@@ -739,7 +739,7 @@ export function render(specs: AccessorSpec[], buildId: string, accessorHash: str
function detectSwiftVersion(): string {
if (process.env.SWIFT_VERSION) return process.env.SWIFT_VERSION;
try {
const out = execSync('swift --version', { stdio: ['ignore', 'pipe', 'ignore'] }).toString();
const out = execSync('swift --version', { stdio: ['ignore', 'pipe', 'ignore'], timeout: 30_000 }).toString();
const m = out.match(/Apple Swift version (\d+\.\d+\.\d+)/);
if (m) return m[1]!;
} catch {
@@ -754,6 +754,7 @@ function detectToolGitRev(): string {
return execSync('git rev-parse --short HEAD', {
cwd: dirname(new URL(import.meta.url).pathname),
stdio: ['ignore', 'pipe', 'ignore'],
timeout: 30_000,
}).toString().trim();
} catch {
return 'dev';
make-pdf/test/e2e/combined-gate.test.ts
+1
View File
@@ -50,6 +50,7 @@ describe("combined-features copy-paste gate", () => {
try {
execFileSync(PDF_BIN, ["generate", FIXTURE, outputPdf, "--quiet"], {
encoding: "utf8",
timeout: 30_000,
env: { ...process.env, BROWSE_BIN },
stdio: ["ignore", "pipe", "pipe"],
});
make-pdf/test/e2e/diagram-gate.test.ts
+1
View File
@@ -83,6 +83,7 @@ describe("diagram render gate", () => {
env: { ...process.env, BROWSE_BIN },
stdout: "pipe",
stderr: "pipe",
timeout: 120_000,
});
const stderr = new TextDecoder().decode(run.stderr);
if (run.exitCode !== 0) {
package.json
+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gstack",
"version": "1.76.0",
"version": "1.77.0",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT",
"type": "module",
@@ -44,6 +44,7 @@
"eval:list": "bun run scripts/eval-list.ts",
"eval:compare": "bun run scripts/eval-compare.ts",
"eval:summary": "bun run scripts/eval-summary.ts",
"eval:flake-rank": "bun run scripts/eval-flake-rank.ts",
"eval:watch": "bun run scripts/eval-watch.ts",
"eval:select": "bun run scripts/eval-select.ts",
"analytics": "bun run scripts/analytics.ts",
scripts/eval-flake-rank.ts
+144
View File
@@ -0,0 +1,144 @@
#!/usr/bin/env bun
/**
* eval-flake-rank — the flake-telemetry dial (WS1).
*
* Aggregates per-test series across every FINALIZED eval-store run on this
* machine (default: ~/.gstack/projects/<slug>/evals/, shard dirs included)
* plus the free suite's flake ledger, and ranks tests by flake signal:
* retried passes first (a test that needs attempt 2 to go green is the
* definition of a flake), then failure rate.
*
* This is the readable dial behind two policies:
* - a flaky pass never blocks a merge, but it is recorded and RANKED here;
* - the required-check promotion (WS16) needs weeks of clean flake-rank,
* not vibes.
*
* Usage:
* bun run eval:flake-rank # project eval dir
* bun run eval:flake-rank --dir <path> # e.g. downloaded CI artifacts
* bun run eval:flake-rank --json # machine-readable
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { getProjectEvalDir, isPartialEval, isFinalizedEvalResultFile, type EvalResult } from '../test/helpers/eval-store';
import { flakeLedgerPath, type FlakeLedgerEntry } from './test-free-shards';
interface TestSeries {
name: string;
runs: number;
passes: number;
fails: number;
retriedPasses: number;
totalAttempts: number;
totalCostUsd: number;
totalDurationMs: number;
lastSeen: string;
}
export function aggregate(evalFiles: string[]): Map<string, TestSeries> {
const series = new Map<string, TestSeries>();
for (const file of evalFiles) {
let run: EvalResult;
try {
run = JSON.parse(fs.readFileSync(file, 'utf-8'));
} catch { continue; }
if (isPartialEval(run, file)) continue; // in-progress accumulators are not runs
if (!Array.isArray(run.tests)) continue;
// Group this run's entries by name so N attempts = 1 run of that test.
const byName = new Map<string, typeof run.tests>();
for (const t of run.tests) {
const list = byName.get(t.name) ?? [];
list.push(t);
byName.set(t.name, list);
}
for (const [name, entries] of byName) {
const s = series.get(name) ?? {
name, runs: 0, passes: 0, fails: 0, retriedPasses: 0,
totalAttempts: 0, totalCostUsd: 0, totalDurationMs: 0, lastSeen: '',
};
const final = entries[entries.length - 1];
s.runs += 1;
s.totalAttempts += entries.length;
if (final.passed) s.passes += 1; else s.fails += 1;
if (final.passed && entries.length > 1) s.retriedPasses += 1;
for (const e of entries) {
s.totalCostUsd += e.cost_usd || 0;
s.totalDurationMs += e.duration_ms || 0;
}
if (run.timestamp > s.lastSeen) s.lastSeen = run.timestamp;
series.set(name, s);
}
}
return series;
}
export function collectEvalFiles(dir: string, sinceDays = 60): string[] {
if (!fs.existsSync(dir)) return [];
const cutoff = Date.now() - sinceDays * 86_400_000;
const out: string[] = [];
for (const name of fs.readdirSync(dir, { recursive: true }) as string[]) {
if (!isFinalizedEvalResultFile(name)) continue;
const full = path.join(dir, name);
try {
// Recency bound (review finding): E2E results embed full transcripts
// (MBs each) and the scan is otherwise unbounded over all-time history.
if (fs.statSync(full).mtimeMs < cutoff) continue;
} catch { continue; }
out.push(full);
}
return out;
}
function readFreeLedger(): FlakeLedgerEntry[] {
// Per-LINE parse: one malformed JSONL line (torn write, manual edit) must
// drop that line, never vanish the whole series (codex adversarial finding).
let raw: string;
try {
raw = fs.readFileSync(flakeLedgerPath(), 'utf-8');
} catch { return []; }
const out: FlakeLedgerEntry[] = [];
for (const line of raw.split('\n')) {
if (!line.trim()) continue;
try { out.push(JSON.parse(line)); } catch { /* torn line — skip */ }
}
return out;
}
if (import.meta.main) {
const argv = process.argv.slice(2);
const dirFlag = argv.indexOf('--dir');
const dir = dirFlag !== -1 ? argv[dirFlag + 1] : getProjectEvalDir();
const asJson = argv.includes('--json');
const sinceFlag = argv.indexOf('--since-days');
const sinceDays = sinceFlag !== -1 ? Number(argv[sinceFlag + 1]) || 60 : 60;
const files = collectEvalFiles(dir, sinceDays);
const series = [...aggregate(files).values()]
.sort((a, b) => b.retriedPasses - a.retriedPasses || (b.fails / b.runs) - (a.fails / a.runs));
const ledger = readFreeLedger();
if (asJson) {
console.log(JSON.stringify({ dir, runsScanned: files.length, tests: series, freeLedger: ledger }, null, 2));
} else {
console.log(`flake-rank: ${files.length} finalized run file(s) under ${dir}`);
const flaky = series.filter((s) => s.retriedPasses > 0 || s.fails > 0);
if (flaky.length === 0) {
console.log(' no retried passes and no failures recorded — clean series');
} else {
console.log(' retries fails/runs avg-dur test');
for (const s of flaky.slice(0, 30)) {
console.log(` ${String(s.retriedPasses).padStart(7)} ${String(s.fails).padStart(5)}/${String(s.runs).padEnd(4)} `
+ `${Math.round(s.totalDurationMs / s.totalAttempts / 1000).toString().padStart(5)}s ${s.name}`);
}
}
if (ledger.length > 0) {
const byFile = new Map<string, number>();
for (const e of ledger) byFile.set(e.file, (byFile.get(e.file) ?? 0) + 1);
console.log(`free-suite flaky-passes (${flakeLedgerPath()}):`);
for (const [file, n] of [...byFile.entries()].sort((a, b) => b[1] - a[1])) {
console.log(` ${String(n).padStart(3)}x ${file}`);
}
}
}
}
scripts/test-free-shards.ts
+76
View File
@@ -1022,6 +1022,62 @@ export function buildRunEpilogue(
export type FreeShardStatus = 'passed' | 'failed' | 'timed-out';
// ─── Flake ledger (WS1 telemetry) ───────────────────────────────────────────
// Single-writer JSONL: ONLY this parent runner appends (never shards, never
// tests — no concurrent-append hazard by construction). CI points
// GSTACK_FLAKE_LEDGER at $RUNNER_TEMP and uploads it as an artifact every
// run, so repeat offenders become an enumerable series instead of console
// scrollback. Fail-open with a loud stderr warning: a broken ledger must
// never red the only required lane.
export interface FlakeLedgerEntry {
ts: string;
runner: 'free';
kind: 'flaky-pass';
file: string;
/** Shard the original failure surfaced in, when attributable. */
shard?: number;
/** Code-state attribution (review finding): without branch/sha the series
* can't tie an entry to the state that produced it, and the WS16
* promotion evidence needs exactly that. */
branch?: string;
git_sha?: string;
}
export function flakeLedgerPath(env: NodeJS.ProcessEnv = process.env): string {
if (env.GSTACK_FLAKE_LEDGER) return env.GSTACK_FLAKE_LEDGER;
// Local default: per-PROJECT, not the machine-global tmpdir — sibling
// Conductor worktrees of DIFFERENT repos must not interleave into one
// series (review finding). CI always sets GSTACK_FLAKE_LEDGER explicitly.
try {
const slug = spawnSync('bash', ['-c', '~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null'], { stdio: 'pipe', timeout: 3000 })
.stdout?.toString().match(/^SLUG=(.+)$/m)?.[1];
if (slug) {
const dir = path.join(os.homedir(), '.gstack', 'projects', slug);
fs.mkdirSync(dir, { recursive: true });
return path.join(dir, 'flake-ledger.jsonl');
}
} catch { /* fall through */ }
return path.join(os.tmpdir(), 'gstack-flake-ledger.jsonl');
}
export function appendFlakeLedger(
entries: FlakeLedgerEntry[],
ledgerPath: string,
warn: (line: string) => void = (line) => console.error(line),
): boolean {
if (entries.length === 0) return true;
try {
fs.mkdirSync(path.dirname(ledgerPath), { recursive: true });
fs.appendFileSync(ledgerPath, entries.map((e) => JSON.stringify(e)).join('\n') + '\n');
return true;
} catch (error) {
warn(`[test:free] WARNING: could not append flake ledger at ${ledgerPath} `
+ `(${error instanceof Error ? error.message : String(error)}) — flaky-pass telemetry lost for this run, verdict unaffected`);
return false;
}
}
export interface FreeShardOutcome {
shard: number;
files: string[];
@@ -1511,6 +1567,26 @@ async function main(): Promise<number> {
if (retryOutcome.status === 'passed') {
console.log(`[test:free] FLAKY-PASS — ${flakyFiles.length} file(s) failed once and passed on serial retry: ${flakyFiles.join(', ')}`);
console.log('[test:free] treat repeat offenders as real flakes worth fixing, not noise.');
// Durable record (WS1): console lines vanish with the scrollback; the
// ledger makes repeat offenders rankable across runs (eval:flake-rank).
const ts = new Date().toISOString();
// Two separate calls: `rev-parse --abbrev-ref HEAD HEAD` abbreviates
// BOTH revs, printing the branch twice — git_sha recorded the branch
// name (codex adversarial finding).
const ledgerBranch = (spawnSync('git', ['rev-parse', '--abbrev-ref', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5000 }).stdout ?? '').trim();
const ledgerSha = (spawnSync('git', ['rev-parse', 'HEAD'], { cwd: ROOT, encoding: 'utf8', timeout: 5000 }).stdout ?? '').trim();
appendFlakeLedger(
flakyFiles.map((file) => ({
ts,
runner: 'free' as const,
kind: 'flaky-pass' as const,
file,
shard: outcomes.find((o) => o.failingFiles.includes(file))?.shard,
...(ledgerBranch ? { branch: ledgerBranch } : {}),
...(ledgerSha ? { git_sha: ledgerSha.slice(0, 12) } : {}),
})),
flakeLedgerPath(),
);
worst = 0;
} else {
console.error('[test:free] flaky-retry FAILED — the failures reproduce serially; not flaky.');
scripts/test-paid-shards.ts
+104 -11
View File
@@ -64,7 +64,7 @@ import {
} from './test-strict-output';
import { PAID_TEST_GLOBS, isPaidTestFile } from '../test/helpers/paid-test-set';
import { PERIODIC_CI_EXCLUDE } from '../test/helpers/periodic-exclude-data';
import { getProjectEvalDir } from '../test/helpers/eval-store';
import { getProjectEvalDir, getClaudeCliVersion, isFinalizedEvalResultFile } from '../test/helpers/eval-store';
import { preflightAnthropicApi } from '../test/helpers/anthropic-preflight';
import {
detectBaseBranch,
@@ -85,11 +85,16 @@ export type PaidTier = 'gate' | 'periodic';
export const DEFAULT_TIER: PaidTier = 'gate';
export const DEFAULT_SHARD_TIMEOUT_MS = 30 * 60_000;
export const DEFAULT_MAX_FILES_PER_SHARD = 1;
export const DEFAULT_JOBS = 4;
// Within one shard's bun process. 4 jobs × 4 ≈ the legacy single-process
// default of 15, keeping total in-flight `claude` sessions inside known-safe
// API rate headroom.
export const DEFAULT_WITHIN_SHARD_CONCURRENCY = 4;
// 8 jobs × 2 within-shard ≈ 10-13 real in-flight sessions (39 of 75
// skill-e2e files hold exactly ONE test, so within-shard concurrency is
// dead weight for most shards) — under the documented-safe ~15 the legacy
// 40-way runner established. The old 4×4 yielded only ~4-6 in-flight and a
// 13-wave local gate worst case (~6.5h); 8×2 halves it. Watch the WS1
// flake telemetry for sustained 429 storms across 2 PR cycles — that is
// the rollback trigger. Prerequisite (landed): per-shard TMPDIR/
// CHROMIUM_PROFILE isolation in runPaidShard.
export const DEFAULT_JOBS = 8;
export const DEFAULT_WITHIN_SHARD_CONCURRENCY = 2;
export function collectPaidTestFiles(rootDir = ROOT): string[] {
const testDir = path.join(rootDir, 'test');
@@ -393,6 +398,24 @@ export interface ShardOutcome {
groupPid: number | null;
/** Tests bun reported executing ("Ran N tests ..."), null when unknown. */
executedTests: number | null;
/** Tests bun reported skipping (" N skip" count line), null when unknown.
* "Ran N tests" COUNTS skips, so executedTests alone cannot distinguish a
* shard that verified work from one whose every test self-skipped —
* codex/gemini files green-by-skip on every CI runner (no binary) and the
* weekly census read them as covered. */
skippedTests: number | null;
}
/**
* True when a shard "passed" without verifying anything: every test bun ran
* was a skip. Legitimate for external-service files on hosts without the
* binary, but it must surface as a census warning, never read as coverage.
*/
export function isAllSkippedPass(outcome: Pick<ShardOutcome, 'status' | 'executedTests' | 'skippedTests'>): boolean {
return outcome.status === 'passed'
&& outcome.executedTests !== null
&& outcome.executedTests > 0
&& outcome.skippedTests === outcome.executedTests;
}
export interface ShardCommand {
@@ -475,6 +498,28 @@ export async function runPaidShard(
if (options.evalDirBase) {
env.GSTACK_EVAL_DIR = path.join(options.evalDirBase, 'shards', shardSlug(files));
}
// Resolve `claude --version` ONCE in the parent (cached across shards) and
// hand it to every child: eval-store's fallback is a synchronous spawn on
// the same thread that polls PTY sessions, so children must never pay it.
if (!env.GSTACK_CLAUDE_CLI_VERSION) {
env.GSTACK_CLAUDE_CLI_VERSION = getClaudeCliVersion();
}
// Per-shard temp + Chromium-profile isolation — the free runner treats
// this as mandatory (test-free-shards.ts: two concurrent shards on one
// profile dir kill each other's browser; shared tmp cross-contaminates),
// and the paid lane had NONE of it. Doubly load-bearing here: when a
// shard hits its 30-min wall the group-SIGKILL means per-test afterAll
// cleanup never runs — the rmSync backstop below is the only thing
// stopping wedged runs from accumulating full git-repo workspaces in the
// shared tmpdir forever. Prerequisite for raising EVALS_JOBS (more
// concurrency on shared state amplifies exactly the opus-47 race class).
const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-paid-shard-'));
const childTmp = path.join(stateDir, 'tmp');
fs.mkdirSync(childTmp);
env.TMPDIR = childTmp;
env.TEMP = childTmp;
env.TMP = childTmp;
env.CHROMIUM_PROFILE = path.join(stateDir, 'chromium-profile');
const startedAt = Date.now();
log(`${label} START ${files.join(' ')} (timeout ${Math.round(timeoutMs / 1000)}s)`);
@@ -529,6 +574,16 @@ export async function runPaidShard(
} finally {
// Close the spool even when the spawn itself failed.
await new Promise<void>((resolve) => logStream.end(() => resolve()));
try {
// async rm: a SIGKILLed shard can leave a full git workspace + Chromium
// profile here; a synchronous recursive delete on the parent's event
// loop would stall every sibling shard's stream classification and
// wall timers for seconds (review finding).
await fs.promises.rm(stateDir, { recursive: true, force: true });
} catch {
// Best-effort: a locked file must not turn a real verdict into an
// exception (same posture as the free runner's cleanup).
}
}
const summary = classifier.end();
@@ -562,7 +617,8 @@ export async function runPaidShard(
const executedTests = summary.terminalTestCounts.length > 0
? summary.terminalTestCounts.reduce((a, b) => a + b, 0)
: null;
return { shard: shardNumber, files, status, exitCode, elapsedMs, groupPid, executedTests };
const skippedTests = summary.terminalTestCounts.length > 0 ? summary.skippedTests : null;
return { shard: shardNumber, files, status, exitCode, elapsedMs, groupPid, executedTests, skippedTests };
}
export interface RunSummary {
@@ -636,6 +692,7 @@ export async function runPaidShards(
elapsedMs: 0,
groupPid: null,
executedTests: null,
skippedTests: null,
}));
let next = 0;
@@ -659,6 +716,7 @@ export async function runPaidShards(
elapsedMs: 0,
groupPid: null,
executedTests: null,
skippedTests: null,
};
console.error(`[test:paid] shard ${index + 1} could not run: ${error instanceof Error ? error.message : String(error)}`);
}
@@ -678,9 +736,14 @@ export function formatSummary(summary: RunSummary): string[] {
+ `${summary.skippedByDiff} skipped by diff`,
];
for (const outcome of summary.outcomes) {
// A pass whose every test skipped is labeled distinctly: it exited 0 but
// verified NOTHING (codex/gemini files on hosts without the binary).
// Status stays 'passed' — availability of an external service is not a
// repo regression — but the census must never read it as coverage.
const allSkipped = isAllSkippedPass(outcome) ? ` ⚠ all ${outcome.executedTests} tests SKIPPED — verified nothing` : '';
lines.push(
` ${outcome.status.padEnd(15)} ${String(Math.round(outcome.elapsedMs / 1000)).padStart(5)}s `
+ outcome.files.join(' '),
+ outcome.files.join(' ') + allSkipped,
);
}
return lines;
@@ -786,7 +849,7 @@ export interface SliceResult {
tier: PaidTier;
sliceIndex: number;
sliceCount: number;
outcomes: Array<Pick<ShardOutcome, 'files' | 'status' | 'exitCode' | 'elapsedMs' | 'executedTests'>>;
outcomes: Array<Pick<ShardOutcome, 'files' | 'status' | 'exitCode' | 'elapsedMs' | 'executedTests' | 'skippedTests'>>;
}
/**
@@ -970,6 +1033,35 @@ async function main(): Promise<number> {
for (const problem of verdict.problems) console.error(` ✗ ${problem}`);
return 1;
}
// Flake honesty (WS1): surface every test that needed a retry to pass.
// WARNS, never fails — a flaky pass must not block merges; it must also
// never be invisible (bun's own output hides retried passes entirely).
// Source: the finalized eval-store JSONs inside the slice artifacts.
const flaky: Array<{ name: string; attempts: number; file: string }> = [];
for (const name of fs.readdirSync(options.reportDir, { recursive: true }) as string[]) {
if (!isFinalizedEvalResultFile(name)) continue;
try {
const parsed = JSON.parse(fs.readFileSync(path.join(options.reportDir, name), 'utf-8')) as { flaky_retries?: Array<{ name: string; attempts: number }> };
for (const f of parsed.flaky_retries ?? []) flaky.push({ ...f, file: name });
} catch { /* non-eval JSON — not this report's business */ }
}
if (flaky.length > 0) {
console.log(`[test:paid] report: ⚠ ${flaky.length} test(s) passed only on retry this run (recorded, not blocking):`);
for (const f of flaky) console.log(` ⚠ ${f.name} (x${f.attempts}) — ${f.file}`);
}
// Census honesty: a 'passed' shard whose every test skipped verified
// nothing (external-service binary absent on the runner). Not a failure —
// service availability is host state, not a repo regression — but the
// report must say so, or the weekly lane reads codex/gemini as covered
// on runners that never install them.
const allSkipped = results.flatMap((r) => r.outcomes.filter(isAllSkippedPass));
if (allSkipped.length > 0) {
console.log(`[test:paid] report: ⚠ ${allSkipped.length} shard(s) passed with EVERY test skipped — they verified nothing:`);
for (const outcome of allSkipped) {
console.log(` ⚠ ${outcome.files.join(' ')} (${outcome.executedTests} skipped — external service missing or tier mismatch)`);
}
}
console.log('[test:paid] report: every planned shard accounted and passed');
return 0;
}
@@ -1023,8 +1115,8 @@ async function main(): Promise<number> {
tier: manifest.tier,
sliceIndex: options.sliceIndex,
sliceCount: manifest.sliceCount,
outcomes: guarded.map(({ files, status, exitCode, elapsedMs, executedTests }) =>
({ files, status, exitCode, elapsedMs, executedTests })),
outcomes: guarded.map(({ files, status, exitCode, elapsedMs, executedTests, skippedTests }) =>
({ files, status, exitCode, elapsedMs, executedTests, skippedTests })),
};
fs.mkdirSync(evalDirBase, { recursive: true });
const sliceResultPath = path.join(evalDirBase, `slice-${options.sliceIndex}.json`);
@@ -1102,6 +1194,7 @@ async function main(): Promise<number> {
elapsedMs: 0,
groupPid: null,
executedTests: null,
skippedTests: null,
}));
const guardedOutcomes = applyHollowShardGuard(runSummary.outcomes, {
evalsAll: process.env.EVALS_ALL === '1',
scripts/test-strict-output.ts
+23
View File
@@ -20,6 +20,15 @@ const ANSI_ESCAPE = /\u001B\[[0-?]*[ -/]*[@-~]/g;
const BUN_FAIL_RESULT = /^\(fail\) .+ \[(?:\d+(?:\.\d+)?)(?:ns|us|µs|ms|s)\]$/;
const BUN_BETWEEN_TESTS_ERROR = '# Unhandled error between tests';
const BUN_TERMINAL_SUMMARY = /^Ran (\d+) tests? across (\d+) files?\. \[(?:\d+(?:\.\d+)?)(?:ns|us|µs|ms|s)\]$/;
// The counts block bun prints just before the terminal summary (" 1 pass",
// " 2 skip", " 0 fail"). "Ran N tests" COUNTS skipped tests, so N alone
// cannot distinguish a shard that verified work from one whose every test
// self-skipped (external-service binary missing, tier mismatch) — the
// green-by-skip class. Anchored to whole-line matches; nested bun-test
// children can still contribute counts (same known limit as the terminal
// summary — see the last-summary-anchoring TODO in the audit).
const BUN_SKIP_COUNT = /^\s*(\d+) skip$/;
const BUN_PASS_COUNT = /^\s*(\d+) pass$/;
export type BunTestOutputFinding = 'failed-test' | 'unhandled-between-tests';
@@ -29,6 +38,11 @@ export interface BunTestOutputSummary {
terminalFileCounts: number[];
/** Test counts from the same terminal lines — feeds the hollow-shard guard. */
terminalTestCounts: number[];
/** Sum of bun's " N skip" count lines. "Ran N tests" includes skips, so
* this is what separates verified work from green-by-skip. */
skippedTests: number;
/** Sum of bun's " N pass" count lines. */
passedTests: number;
}
export type ForwardedTerminationSignal = 'SIGINT' | 'SIGTERM';
@@ -229,6 +243,8 @@ export class BunTestOutputClassifier {
private unhandledBetweenTests = 0;
private terminalFileCounts: number[] = [];
private terminalTestCounts: number[] = [];
private skippedTests = 0;
private passedTests = 0;
write(chunk: Uint8Array | string, origin: ClassifierOrigin = 'stdout'): void {
this.pending[origin] += typeof chunk === 'string'
@@ -252,6 +268,8 @@ export class BunTestOutputClassifier {
unhandledBetweenTests: this.unhandledBetweenTests,
terminalFileCounts: [...this.terminalFileCounts],
terminalTestCounts: [...this.terminalTestCounts],
skippedTests: this.skippedTests,
passedTests: this.passedTests,
};
}
@@ -268,6 +286,11 @@ export class BunTestOutputClassifier {
const finding = classifyBunTestOutputLine(line);
if (finding === 'failed-test') this.failedTests += 1;
if (finding === 'unhandled-between-tests') this.unhandledBetweenTests += 1;
const stripped = stripAnsiLine(line);
const skip = BUN_SKIP_COUNT.exec(stripped);
if (skip !== null) this.skippedTests += Number.parseInt(skip[1], 10);
const pass = BUN_PASS_COUNT.exec(stripped);
if (pass !== null) this.passedTests += Number.parseInt(pass[1], 10);
const terminal = parseBunTerminalSummary(line);
if (terminal !== null) {
this.terminalFileCounts.push(terminal.files);
test/artifacts-init-migration.test.ts
+2
View File
@@ -22,6 +22,7 @@ function runMigration(fakeHome: string): { code: number; stdout: string; stderr:
env: { ...process.env, HOME: fakeHome },
stdout: 'pipe',
stderr: 'pipe',
timeout: 30_000,
});
return {
code: proc.exitCode ?? -1,
@@ -217,6 +218,7 @@ function runMigrationV140(fakeHome: string): { code: number; stdout: string; std
env: { ...process.env, HOME: fakeHome },
stdout: 'pipe',
stderr: 'pipe',
timeout: 30_000,
});
return {
code: proc.exitCode ?? -1,
test/auq-error-fallback-hook.test.ts
+1
View File
@@ -113,6 +113,7 @@ function runHook(stdin: object, env: Record<string, string>): { additionalContex
input: JSON.stringify(stdin),
encoding: 'utf-8',
env: { PATH: process.env.PATH ?? '/usr/bin:/bin', ...env },
timeout: 30_000,
});
const parsed = JSON.parse(res.stdout || '{}');
return parsed.hookSpecificOutput ?? {};
test/bin-context-windows-slug.test.ts
+21 -21
View File
@@ -69,8 +69,8 @@ describe("native slug fallback mirrors bin/gstack-slug", () => {
["https://gitlab.com/acme/Widget", "acme-Widget"],
] as const) {
const cwd = fs.mkdtempSync(path.join(tmp, "repo-"));
spawnSync("git", ["init", "-q"], { cwd });
spawnSync("git", ["remote", "add", "origin", url], { cwd });
spawnSync("git", ["init", "-q"], { cwd, timeout: 30_000 });
spawnSync("git", ["remote", "add", "origin", url], { cwd, timeout: 30_000 });
expect(slugFromEnvironment(path.join(tmp, "home2"), cwd)).toBe(want);
}
});
@@ -231,8 +231,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => {
const projectRoot = path.join(tmp, "realgit");
const subdir = path.join(projectRoot, "src", "deep");
fs.mkdirSync(subdir, { recursive: true });
spawnSync("git", ["init", "-q", projectRoot]);
spawnSync("git", ["-C", projectRoot, "remote", "add", "origin", "https://github.com/foo/bar.git"]);
spawnSync("git", ["init", "-q", projectRoot], { timeout: 30_000 });
spawnSync("git", ["-C", projectRoot, "remote", "add", "origin", "https://github.com/foo/bar.git"], { timeout: 30_000 });
expectBoth(subdir, "foo-bar");
});
@@ -269,10 +269,10 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => {
const outer = path.join(tmp, "outer-project");
const inner = path.join(outer, "vendor", "inner-lib");
fs.mkdirSync(inner, { recursive: true });
spawnSync("git", ["init", "-q", outer]);
spawnSync("git", ["-C", outer, "remote", "add", "origin", "git@github.com:acme/outer.git"]);
spawnSync("git", ["init", "-q", inner]);
spawnSync("git", ["-C", inner, "remote", "add", "origin", "git@github.com:vendor/inner.git"]);
spawnSync("git", ["init", "-q", outer], { timeout: 30_000 });
spawnSync("git", ["-C", outer, "remote", "add", "origin", "git@github.com:acme/outer.git"], { timeout: 30_000 });
spawnSync("git", ["init", "-q", inner], { timeout: 30_000 });
spawnSync("git", ["-C", inner, "remote", "add", "origin", "git@github.com:vendor/inner.git"], { timeout: 30_000 });
expectBoth(inner, "acme-outer");
});
@@ -286,8 +286,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => {
fs.mkdirSync(path.join(strayHome, ".git"), { recursive: true }); // empty — invalid repo
const repo = path.join(strayHome, "work", "repo");
fs.mkdirSync(repo, { recursive: true });
spawnSync("git", ["init", "-q", repo]);
spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/garrytan/gstack"]);
spawnSync("git", ["init", "-q", repo], { timeout: 30_000 });
spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/garrytan/gstack"], { timeout: 30_000 });
expectBoth(repo, "garrytan-gstack");
expect(slugFromEnvironment(nativeHome(), repo)).not.toBe("strayhome");
});
@@ -299,9 +299,9 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => {
const outer = path.join(tmp, "outer-plain");
const inner = path.join(outer, "vendor", "inner-lib");
fs.mkdirSync(inner, { recursive: true });
spawnSync("git", ["init", "-q", outer]); // no origin — marker-only repo
spawnSync("git", ["init", "-q", inner]);
spawnSync("git", ["-C", inner, "remote", "add", "origin", "git@github.com:vendor/inner.git"]);
spawnSync("git", ["init", "-q", outer], { timeout: 30_000 }); // no origin — marker-only repo
spawnSync("git", ["init", "-q", inner], { timeout: 30_000 });
spawnSync("git", ["-C", inner, "remote", "add", "origin", "git@github.com:vendor/inner.git"], { timeout: 30_000 });
expectBoth(inner, "vendor-inner");
});
@@ -313,8 +313,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => {
fs.mkdirSync(path.join(strayHome, ".git"), { recursive: true });
const repo = path.join(strayHome, "git", "proj");
fs.mkdirSync(repo, { recursive: true });
spawnSync("git", ["init", "-q", repo]);
spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/garrytan/gstack"]);
spawnSync("git", ["init", "-q", repo], { timeout: 30_000 });
spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/garrytan/gstack"], { timeout: 30_000 });
const cacheDir = path.join(nativeHome(), "slug-cache");
fs.mkdirSync(cacheDir, { recursive: true });
@@ -335,8 +335,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => {
const inner = path.join(wrapper, "apps", "web");
fs.mkdirSync(inner, { recursive: true });
fs.writeFileSync(path.join(wrapper, "package.json"), '{"name":"wrapper"}\n');
spawnSync("git", ["init", "-q", inner]);
spawnSync("git", ["-C", inner, "remote", "add", "origin", "https://github.com/acme/web.git"]);
spawnSync("git", ["init", "-q", inner], { timeout: 30_000 });
spawnSync("git", ["-C", inner, "remote", "add", "origin", "https://github.com/acme/web.git"], { timeout: 30_000 });
const cacheDir = path.join(nativeHome(), "slug-cache");
fs.mkdirSync(cacheDir, { recursive: true });
@@ -361,8 +361,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => {
// fire even though cached == basename(project root).
const repo = path.join(tmp, "stickyproj");
fs.mkdirSync(repo, { recursive: true });
spawnSync("git", ["init", "-q", repo]);
spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/x/y.git"]);
spawnSync("git", ["init", "-q", repo], { timeout: 30_000 });
spawnSync("git", ["-C", repo, "remote", "add", "origin", "https://github.com/x/y.git"], { timeout: 30_000 });
const cacheDir = path.join(nativeHome(), "slug-cache");
fs.mkdirSync(cacheDir, { recursive: true });
@@ -379,8 +379,8 @@ describe("walk-up parity with bin/gstack-slug (outermost project root)", () => {
// implementations must reject it and fall through to the basename.
const repo = path.join(tmp, "dotty");
fs.mkdirSync(repo, { recursive: true });
spawnSync("git", ["init", "-q", repo]);
spawnSync("git", ["-C", repo, "remote", "add", "origin", ".."]);
spawnSync("git", ["init", "-q", repo], { timeout: 30_000 });
spawnSync("git", ["-C", repo, "remote", "add", "origin", ".."], { timeout: 30_000 });
expectBoth(repo, "dotty");
});
test/binding-template-drift.test.ts
+2 -2
View File
@@ -102,7 +102,7 @@ describe('content-binding template drift', () => {
const abort = spawnSync('bash', ['-c', scriptFor(
'clean body\n',
'body with UNTRUSTED TRACKER CONTENT banner leak\n',
)], { encoding: 'utf-8' });
)], { encoding: 'utf-8', timeout: 30_000 });
expect(abort.stderr).toContain('ABORT: envelope banner leaked');
expect(abort.stdout).not.toContain('banner tripwire clean');
@@ -110,7 +110,7 @@ describe('content-binding template drift', () => {
const clean = spawnSync('bash', ['-c', scriptFor(
'clean body\n',
'also clean body\n',
)], { encoding: 'utf-8' });
)], { encoding: 'utf-8', timeout: 30_000 });
expect(clean.stdout).toContain('banner tripwire clean');
expect(clean.stderr).not.toContain('ABORT');
} finally {
test/brain-sync.test.ts
+13 -11
View File
@@ -43,12 +43,13 @@ function run(argv: string[], opts: { env?: Record<string, string>; input?: strin
encoding: 'utf-8',
input: opts.input,
cwd: ROOT,
timeout: 30_000,
});
return { stdout: res.stdout || '', stderr: res.stderr || '', status: res.status ?? -1 };
}
function git(args: string[], cwd?: string) {
const res = spawnSync('git', args, { cwd: cwd || tmpHome, encoding: 'utf-8' });
const res = spawnSync('git', args, { cwd: cwd || tmpHome, encoding: 'utf-8', timeout: 30_000 });
return { stdout: res.stdout || '', stderr: res.stderr || '', status: res.status ?? -1 };
}
@@ -77,7 +78,7 @@ function seedSpool(record: string): string {
beforeEach(() => {
tmpHome = fs.mkdtempSync(path.join(os.tmpdir(), 'brain-sync-home-'));
bareRemote = fs.mkdtempSync(path.join(os.tmpdir(), 'brain-sync-remote-'));
spawnSync('git', ['init', '--bare', '-q', '-b', 'main', bareRemote]);
spawnSync('git', ['init', '--bare', '-q', '-b', 'main', bareRemote], { timeout: 30_000 });
});
afterEach(() => {
@@ -198,6 +199,7 @@ describe('gstack-brain-enqueue', () => {
const r = spawnSync(path.join(BIN, 'gstack-brain-enqueue'), [`file-${i}.jsonl`], {
env: { ...process.env, GSTACK_HOME: tmpHome },
encoding: 'utf-8',
timeout: 30_000,
});
resolve();
}));
@@ -245,7 +247,7 @@ describe('gstack-jsonl-merge', () => {
const lines = fs.readFileSync(ours, 'utf-8').trim().split('\n');
expect(lines.length).toBe(3);
// Order is deterministic (sha256 of each line).
const again = spawnSync(path.join(BIN, 'gstack-jsonl-merge'), [base, ours, theirs]);
const again = spawnSync(path.join(BIN, 'gstack-jsonl-merge'), [base, ours, theirs], { timeout: 30_000 });
// (re-running doesn't change the order since same input → same output)
});
});
@@ -271,7 +273,7 @@ describe('init + sync + restore round-trip', () => {
test('refuses init on different remote', () => {
run(['gstack-artifacts-init', '--remote', bareRemote]);
const otherRemote = fs.mkdtempSync(path.join(os.tmpdir(), 'brain-other-'));
spawnSync('git', ['init', '--bare', '-q', '-b', 'main', otherRemote]);
spawnSync('git', ['init', '--bare', '-q', '-b', 'main', otherRemote], { timeout: 30_000 });
const r = run(['gstack-artifacts-init', '--remote', otherRemote]);
expect(r.status).not.toBe(0);
expect(r.stderr).toContain('already a git repo pointing at');
@@ -288,7 +290,7 @@ describe('init + sync + restore round-trip', () => {
const r = run(['gstack-brain-sync', '--once']);
expect(r.status).toBe(0);
// Check the remote got the commit.
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' });
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 });
expect(log.stdout).toMatch(/sync: 1 file/);
});
@@ -311,7 +313,7 @@ describe('init + sync + restore round-trip', () => {
const restored = fs.readFileSync(path.join(machineB, 'projects/myproj/learnings.jsonl'), 'utf-8');
expect(restored).toContain('machine A wisdom');
// Merge drivers re-registered on B.
const cfg = spawnSync('git', ['-C', machineB, 'config', '--get', 'merge.jsonl-append.driver'], { encoding: 'utf-8' });
const cfg = spawnSync('git', ['-C', machineB, 'config', '--get', 'merge.jsonl-append.driver'], { encoding: 'utf-8', timeout: 30_000 });
expect(cfg.stdout).toContain('gstack-jsonl-merge');
fs.rmSync(machineB, { recursive: true, force: true });
});
@@ -399,7 +401,7 @@ describe('gstack-brain-sync egress receipt gate', () => {
// No local commit was created.
expect(git(['rev-list', '--count', 'HEAD']).stdout.trim()).toBe(commitsBefore);
// Nothing reached the remote.
const remoteLog = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' });
const remoteLog = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 });
expect(remoteLog.stdout).not.toMatch(/sync: 1 file/);
const status = JSON.parse(fs.readFileSync(path.join(tmpHome, '.brain-sync-status.json'), 'utf-8'));
expect(status.status).toBe('push_failed');
@@ -411,7 +413,7 @@ describe('gstack-brain-sync egress receipt gate', () => {
// Next run (ledger writable again) drains the intact queue and pushes.
const retry = run(['gstack-brain-sync', '--once']);
expect(retry.status).toBe(0);
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' });
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 });
expect(log.stdout).toMatch(/sync: 1 file/);
});
@@ -583,7 +585,7 @@ describe('#2549 queue integrity', () => {
expect(r.status).toBe(0);
expect(spoolText()).not.toContain('learnings.jsonl'); // synced, removed
expect(spoolText()).toContain('timeline.jsonl'); // held, retained
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' });
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 });
expect(log.stdout).toMatch(/sync: 1 file/);
});
@@ -619,7 +621,7 @@ describe('#2549 queue integrity', () => {
fs.rmSync(hook);
const retry = run(['gstack-brain-sync', '--once']);
expect(retry.status).toBe(0);
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' });
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 });
expect(log.stdout).toMatch(/sync: 1 file/);
expect(git(['rev-list', '--count', 'origin/main..HEAD']).stdout.trim()).toBe('0');
});
@@ -751,7 +753,7 @@ describe('C12 spool queue', () => {
run(['gstack-config', 'set', 'artifacts_sync_mode', mode]);
}
const remoteLog = () =>
spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' }).stdout;
spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8', timeout: 30_000 }).stdout;
test('two rapid enqueues of different paths create two spool files; one drain syncs both', () => {
initWithMode('full');
test/branch-slug-hygiene.test.ts
+5 -5
View File
@@ -37,7 +37,7 @@ const FILENAME_PREFIX = /\$\{?_BRANCH\}?[A-Za-z0-9._-]*\.(?:jsonl|json|md|txt|lo
function renderedSkillFiles(): string[] {
const out = execSync(
`find "${ROOT}" -name 'SKILL.md' -not -path '*/node_modules/*' -not -path '*/.claude/*' ; find "${ROOT}" -path '*/sections/*.md' -not -path '*/node_modules/*' -not -path '*/.claude/*'`,
{ encoding: 'utf-8' },
{ encoding: 'utf-8', timeout: 30_000 },
);
return out.split('\n').filter(Boolean);
}
@@ -91,18 +91,18 @@ describe('branch slug hygiene (#2550, #1851)', () => {
const env = { ...process.env, GSTACK_HOME: home };
execSync(
'git init -q && git -c user.email=t@t -c user.name=t commit -q --allow-empty -m init && git checkout -q -b feat/slug-hygiene',
{ cwd: repo, encoding: 'utf-8' },
{ cwd: repo, encoding: 'utf-8', timeout: 30_000 },
);
// Writer: the real gstack-review-log (canonicalizes via gstack-slug).
execSync(
`"${path.join(ROOT, 'bin', 'gstack-review-log')}" '{"skill":"ship","status":"ok"}'`,
{ cwd: repo, env, encoding: 'utf-8' },
{ cwd: repo, env, encoding: 'utf-8', timeout: 30_000 },
);
// The slug-canonical filename must exist; the raw form must not.
const slugVars = execSync(`"${path.join(ROOT, 'bin', 'gstack-slug')}"`, {
cwd: repo, env, encoding: 'utf-8',
cwd: repo, env, encoding: 'utf-8', timeout: 30_000,
});
const slug = slugVars.match(/^SLUG=(.*)$/m)![1];
const branch = slugVars.match(/^BRANCH=(.*)$/m)![1];
@@ -120,7 +120,7 @@ describe('branch slug hygiene (#2550, #1851)', () => {
.find((l) => l.includes('-reviews.jsonl'))!;
const script = `_PROJ="${proj}"\nBRANCH="${branch}"\n${probeLine.trim()}`;
const out = execSync(`bash -c '${script.replace(/'/g, `'\\''`)}'`, {
cwd: repo, encoding: 'utf-8',
cwd: repo, encoding: 'utf-8', timeout: 30_000,
});
expect(out).toContain('REVIEWS: 1 entries');
test/builder-profile.test.ts
+1 -1
View File
@@ -16,7 +16,7 @@ function runProfile(): Record<string, string> {
encoding: 'utf-8',
timeout: 15000,
};
const stdout = execSync(`${BIN}/gstack-builder-profile`, execOpts).trim();
const stdout = execSync(`${BIN}/gstack-builder-profile`, execOpts).trim(); // timeout via execOpts
const result: Record<string, string> = {};
for (const line of stdout.split('\n')) {
const idx = line.indexOf(':');
test/ci-image-cli-pin.test.ts
+67
View File
@@ -0,0 +1,67 @@
/**
* Provider CLIs baked into the CI image must be pinned to EXACT versions.
*
* The PTY harness (test/helpers/claude-pty-runner.ts) screen-scrapes the
* claude CLI's TUI — trust dialog, input-prompt ready marker, spinner glyphs.
* The image used to install `npm i -g @anthropic-ai/claude-code` UNPINNED and
* rebuild weekly "to pick up CLI updates", while bun sat carefully pinned at
* 1.3.13 two RUN lines above — the exact drift class the bun pin exists for.
* Receipts: TUI drift broke the harness three separate times (welcome-screen
* wedge vs CLI 2.1.233, skillify HOME discovery on 2.1.237, guard/freeze
* hooks on 2.1.162), each debugged as a "flake" before being traced to an
* unpinned weekly-latest CLI.
*
* This tripwire fails the free suite when any globally-installed npm package
* in Dockerfile.ci lacks an exact `@X.Y.Z` pin. Bumps are deliberate: edit
* the pin in a PR and run the PTY gate against the new TUI before merging.
*/
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
const ROOT = path.resolve(__dirname, '..');
const DOCKERFILE = path.join(ROOT, '.github', 'docker', 'Dockerfile.ci');
/** Package specs from every `npm i -g` / `npm install -g` in the Dockerfile. */
export function globalNpmInstallSpecs(source: string): string[] {
const specs: string[] = [];
for (const match of source.matchAll(/npm\s+(?:i|install)\s+(?:-g|--global)\s+([^\n\\&|;]+)/g)) {
for (const spec of match[1].trim().split(/\s+/)) {
if (spec.startsWith('-')) continue; // flags like --no-fund
specs.push(spec);
}
}
return specs;
}
/** Exact pin = a trailing @<semver> with no range operator (no ^ ~ x *). */
export function isExactlyPinned(spec: string): boolean {
// Scoped (@scope/name@1.2.3) or bare (name@1.2.3); version must be exact.
const at = spec.lastIndexOf('@');
if (at <= 0) return false; // no version at all (or a bare scope)
const version = spec.slice(at + 1);
return /^\d+\.\d+\.\d+(?:-[\w.]+)?$/.test(version);
}
describe('ci image provider-CLI pins', () => {
const source = fs.readFileSync(DOCKERFILE, 'utf-8');
const specs = globalNpmInstallSpecs(source);
test('the image installs at least the claude CLI globally (scan must not rot)', () => {
expect(
specs.some((s) => s.startsWith('@anthropic-ai/claude-code@')),
`expected a pinned @anthropic-ai/claude-code install in ${path.relative(ROOT, DOCKERFILE)}; found: ${specs.join(', ') || '(none)'}`,
).toBe(true);
});
test('every global npm install carries an exact @X.Y.Z pin', () => {
const unpinned = specs.filter((s) => !isExactlyPinned(s));
expect(
unpinned,
`unpinned global npm installs in Dockerfile.ci: ${unpinned.join(', ')}\n`
+ 'Pin the exact version (name@X.Y.Z) and bump via a PR that runs the '
+ 'PTY gate against the new TUI — weekly-latest CLI drift broke the '
+ 'harness three times before this tripwire existed.',
).toHaveLength(0);
});
});
test/code-intelligence.test.ts
+10 -10
View File
@@ -153,9 +153,9 @@ describe("session-start indexing offer (suggest)", () => {
home = fs.mkdtempSync(path.join(os.tmpdir(), "ci-home-"));
repo = fs.mkdtempSync(path.join(os.tmpdir(), "ci-repo-"));
env = { ...process.env, GSTACK_HOME: home };
Bun.spawnSync(["git", "init", "-q", repo]);
Bun.spawnSync(["git", "init", "-q", repo], { timeout: 30_000 });
for (const name of ["a.ts", "b.ts", "c.ts"]) fs.writeFileSync(path.join(repo, name), "x\n");
Bun.spawnSync(["git", "-C", repo, "add", "-A"]);
Bun.spawnSync(["git", "-C", repo, "add", "-A"], { timeout: 30_000 });
});
afterEach(() => {
fs.rmSync(home, { recursive: true, force: true });
@@ -399,7 +399,7 @@ describe("consent unification — deny tier wins (R1)", () => {
function makeRepo(dir: string, url: string): string {
const repo = path.join(dir, "repo");
fs.mkdirSync(repo, { recursive: true });
const git = (...a: string[]) => execFileSync("git", a, { cwd: repo });
const git = (...a: string[]) => execFileSync("git", a, { cwd: repo, timeout: 30_000 });
git("init", "-q", ".");
git("remote", "add", "origin", url);
return repo;
@@ -423,11 +423,11 @@ describe("consent unification — deny tier wins (R1)", () => {
const env = { ...process.env, GSTACK_HOME: home };
const repo = makeRepo(home, URL);
setConsent(repo, true, env);
execFileSync(POLICY_BIN, ["set", URL, "deny"], { env, encoding: "utf-8" });
execFileSync(POLICY_BIN, ["set", URL, "deny"], { env, encoding: "utf-8", timeout: 30_000 });
expect(hasConsent(repo, env)).toBe(false);
// Flipping the tier back restores the recorded consent — the veto is
// live policy, not a destructive rewrite of the consent store.
execFileSync(POLICY_BIN, ["set", URL, "read-write"], { env, encoding: "utf-8" });
execFileSync(POLICY_BIN, ["set", URL, "read-write"], { env, encoding: "utf-8", timeout: 30_000 });
expect(hasConsent(repo, env)).toBe(true);
} finally { fs.rmSync(home, { recursive: true, force: true }); }
});
@@ -439,7 +439,7 @@ describe("consent unification — deny tier wins (R1)", () => {
const env = { ...process.env, GSTACK_HOME: home };
const repo = makeRepo(home, URL);
setConsent(repo, true, env);
execFileSync(POLICY_BIN, ["set", URL, "read-write"], { env, encoding: "utf-8" });
execFileSync(POLICY_BIN, ["set", URL, "read-write"], { env, encoding: "utf-8", timeout: 30_000 });
fs.chmodSync(path.join(home, "gbrain-repo-policy.json"), 0o000);
try {
expect(hasConsent(repo, env)).toBe(false);
@@ -460,7 +460,7 @@ describe("consent unification — deny tier wins (R1)", () => {
const env = { ...process.env, GSTACK_HOME: home };
const repo = makeRepo(home, URL);
setConsent(repo, true, env);
execFileSync(POLICY_BIN, ["set", URL, "read-only"], { env, encoding: "utf-8" });
execFileSync(POLICY_BIN, ["set", URL, "read-only"], { env, encoding: "utf-8", timeout: 30_000 });
// Default op class is write — a caller that doesn't say gets fail-closed.
expect(hasConsent(repo, env)).toBe(false);
expect(hasConsent(repo, env, "write")).toBe(false);
@@ -475,7 +475,7 @@ describe("consent unification — deny tier wins (R1)", () => {
const env = { ...process.env, GSTACK_HOME: home };
const repo = makeRepo(home, URL);
setConsent(repo, true, env);
execFileSync(POLICY_BIN, ["set", URL, "deny"], { env, encoding: "utf-8" });
execFileSync(POLICY_BIN, ["set", URL, "deny"], { env, encoding: "utf-8", timeout: 30_000 });
expect(hasConsent(repo, env, "write")).toBe(false);
expect(hasConsent(repo, env, "read")).toBe(false);
} finally { fs.rmSync(home, { recursive: true, force: true }); }
@@ -971,9 +971,9 @@ exit 1
function makeRepoWithFiles(count: number): string {
const repo = fs.mkdtempSync(path.join(os.tmpdir(), "ci-cli-suggest-"));
Bun.spawnSync(["git", "init", "-q", repo]);
Bun.spawnSync(["git", "init", "-q", repo], { timeout: 30_000 });
for (let i = 0; i < count; i++) fs.writeFileSync(path.join(repo, `f${i}.ts`), "x\n");
Bun.spawnSync(["git", "-C", repo, "add", "-A"]);
Bun.spawnSync(["git", "-C", repo, "add", "-A"], { timeout: 30_000 });
return repo;
}
test/codex-e2e-plan-format.test.ts
+1 -1
View File
@@ -43,7 +43,7 @@ const ROOT = path.resolve(import.meta.dir, '..');
const CODEX_AVAILABLE = (() => {
try {
const result = Bun.spawnSync(['which', 'codex']);
const result = Bun.spawnSync(['which', 'codex'], { timeout: 30_000 });
return result.exitCode === 0;
} catch { return false; }
})();
test/codex-e2e-recommendation-substance.test.ts
+1 -1
View File
@@ -31,7 +31,7 @@ const ROOT = path.resolve(import.meta.dir, '..');
const CODEX_AVAILABLE = (() => {
try {
return Bun.spawnSync(['which', 'codex']).exitCode === 0;
return Bun.spawnSync(['which', 'codex'], { timeout: 30_000 }).exitCode === 0;
} catch {
return false;
}
test/codex-e2e-sol-scope.test.ts
+2 -2
View File
@@ -21,12 +21,12 @@ import { EvalCollector } from './helpers/eval-store';
import { selectTests, detectBaseBranch, getChangedFiles, GLOBAL_TOUCHFILES } from './helpers/touchfiles';
const ROOT = path.resolve(import.meta.dir, '..');
const CODEX_AVAILABLE = spawnSync('which', ['codex']).status === 0;
const CODEX_AVAILABLE = spawnSync('which', ['codex'], { timeout: 30_000 }).status === 0;
// The run pins the model with --ignore-user-config; older codex CLIs reject
// the flag with an argv error indistinguishable from a Sol regression, so
// probe support and skip (not fail) on old CLIs.
const IGNORE_USER_CONFIG_SUPPORTED = CODEX_AVAILABLE
&& (spawnSync('codex', ['exec', '--help'], { encoding: 'utf8' }).stdout ?? '').includes('--ignore-user-config');
&& (spawnSync('codex', ['exec', '--help'], { encoding: 'utf8', timeout: 120_000 }).stdout ?? '').includes('--ignore-user-config');
const evalsEnabled = !!process.env.EVALS;
// External-service test — periodic tier only (CLAUDE.md tiering rule 3). The
// positive guard shape below is what classifyPaidTestFile greps to exclude
test/codex-e2e.test.ts
+13 -6
View File
@@ -32,7 +32,7 @@ const ROOT = path.resolve(import.meta.dir, '..');
const CODEX_AVAILABLE = (() => {
try {
const result = Bun.spawnSync(['which', 'codex']);
const result = Bun.spawnSync(['which', 'codex'], { timeout: 30_000 });
return result.exitCode === 0;
} catch { return false; }
})();
@@ -63,11 +63,18 @@ if (!evalsEnabled) {
// --- Diff-based test selection ---
// Codex E2E touchfiles — keyed by test name, same pattern as E2E_TOUCHFILES
const CODEX_E2E_TOUCHFILES: Record<string, string[]> = {
'codex-discover-skill': ['codex/**', '.agents/skills/**', 'test/helpers/codex-session-runner.ts'],
'codex-review-findings': ['review/**', '.agents/skills/gstack-review/**', 'codex/**', 'test/helpers/codex-session-runner.ts'],
};
// Codex E2E touchfiles — DERIVED from the canonical map, never a local fork.
// The old hand-copy drifted (it kept gitignored '.agents/skills/**' patterns
// that can never match a git diff, and missed deps the canonical map gained
// like lib/worktree.ts and this test file itself), so review-template edits
// silently stopped selecting these tests. Deriving keeps one source of truth
// and puts these keys under the tier-alignment + dep-existence invariants.
const CODEX_E2E_TOUCHFILES: Record<string, string[]> = Object.fromEntries(
(['codex-discover-skill', 'codex-review-findings'] as const).map((key) => {
if (!E2E_TOUCHFILES[key]) throw new Error(`canonical E2E_TOUCHFILES lost key '${key}' — fix the map, not this file`);
return [key, E2E_TOUCHFILES[key]];
}),
);
let selectedTests: string[] | null = null; // null = run all
test/codex-generation-model.test.ts
+2
View File
@@ -116,6 +116,7 @@ model = "gpt-5.6-terra"
cwd: ROOT,
encoding: 'utf8',
env: { ...process.env, CODEX_HOME: home },
timeout: 30_000,
});
expect(ok.status).toBe(0);
expect(ok.stdout).toBe(`gpt-5.6-sol\t${path.join(home, 'config.toml')}\n`);
@@ -123,6 +124,7 @@ model = "gpt-5.6-terra"
const bad = spawnSync('bun', ['run', 'scripts/resolve-codex-generation-model.ts', '--explicit', 'llama-local'], {
cwd: ROOT,
encoding: 'utf8',
timeout: 30_000,
});
expect(bad.status).not.toBe(0);
expect(bad.stderr).toContain('Unknown model');
test/codex-hardening.test.ts
+1 -1
View File
@@ -297,7 +297,7 @@ describe('gstack-codex-probe: timeout wrapper + namespace hygiene', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-watchdog-'));
try {
const which = (tool: string) =>
spawnSync('bash', ['-c', `command -v ${tool}`]).stdout.toString().trim() || `/bin/${tool}`;
spawnSync('bash', ['-c', `command -v ${tool}`], { timeout: 30_000 }).stdout.toString().trim() || `/bin/${tool}`;
fs.symlinkSync(which('bash'), path.join(dir, 'bash'));
fs.symlinkSync(which('sleep'), path.join(dir, 'sleep'));
const r = runProbe({
test/codex-resume-flag-semantics.test.ts
+1 -1
View File
@@ -15,7 +15,7 @@
import { describe, test, expect } from 'bun:test';
import { spawnSync } from 'child_process';
const codexPath = spawnSync('which', ['codex'], { encoding: 'utf-8' }).stdout.trim();
const codexPath = spawnSync('which', ['codex'], { encoding: 'utf-8', timeout: 30_000 }).stdout.trim();
const codexAvailable = codexPath.length > 0;
describe.skipIf(!codexAvailable)(
test/codex-web-search-flag.test.ts
+1 -1
View File
@@ -23,7 +23,7 @@ function grepRepo(pattern: string, includes: string[]): string[] {
const includeArgs = includes.map((i) => `--include='${i}'`).join(' ');
const out = execSync(
`grep -rln ${includeArgs} -e '${pattern}' "${ROOT}" || true`,
{ encoding: 'utf-8' },
{ encoding: 'utf-8', timeout: 30_000 },
);
return out
.split('\n')
test/context-bill.test.ts
+1 -1
View File
@@ -718,7 +718,7 @@ describe("CLI plumbing", () => {
});
it("bin/gstack-context-bill runs standalone", () => {
const result = Bun.spawnSync([path.join(ROOT, "bin", "gstack-context-bill"), TREE_A]);
const result = Bun.spawnSync([path.join(ROOT, "bin", "gstack-context-bill"), TREE_A], { timeout: 30_000 });
expect(result.exitCode).toBe(0);
expect(result.stdout.toString()).toContain("ALWAYS-ON");
expect(result.stdout.toString()).toContain("EAGER");
test/design-flag-utils.test.ts
+1 -1
View File
@@ -99,7 +99,7 @@ describe("normalizeIntFlag CLI wrapper (exit-1 semantics)", () => {
const v = normalizeIntFlag(${rawExpr}, ${specExpr});
console.log("VALUE:" + v);
`;
const res = spawnSync("bun", ["-e", script], { encoding: "utf-8", cwd: ROOT });
const res = spawnSync("bun", ["-e", script], { encoding: "utf-8", cwd: ROOT, timeout: 30_000 });
return { status: res.status ?? -1, stderr: res.stderr ?? "" };
}
test/diagram-render-drift.test.ts
+1 -1
View File
@@ -86,7 +86,7 @@ describe("diagram-render bundle drift", () => {
"deep: fresh build reproduces committed dist",
async () => {
const before = await Bun.file(BUILD_INFO).json();
const proc = Bun.spawnSync(["bun", "run", "scripts/build.ts"], { cwd: ROOT });
const proc = Bun.spawnSync(["bun", "run", "scripts/build.ts"], { cwd: ROOT, timeout: 120_000 });
expect(proc.exitCode).toBe(0);
const after = await Bun.file(BUILD_INFO).json();
expect(after.sha256).toBe(before.sha256);
test/distill-apply.test.ts
+1 -1
View File
@@ -60,7 +60,7 @@ function run(args: string[]): { stdout: string; stderr: string; status: number }
env.GSTACK_STATE_ROOT = stateRoot;
env.GSTACK_QUESTION_LOG_NO_DERIVE = '1';
delete env.GSTACK_HOME;
const res = spawnSync(BIN, args, { env, encoding: 'utf-8', cwd: fixtureCwd });
const res = spawnSync(BIN, args, { env, encoding: 'utf-8', cwd: fixtureCwd, timeout: 30_000 });
return {
stdout: res.stdout ?? '',
stderr: res.stderr ?? '',
test/distill-free-text.test.ts
+5 -3
View File
@@ -47,6 +47,7 @@ function run(args: string[]): { stdout: string; stderr: string; status: number }
env: makeEnv(),
encoding: 'utf-8',
cwd: fixtureCwd,
timeout: 30_000,
});
return {
stdout: res.stdout ?? '',
@@ -75,6 +76,7 @@ function writeAuqOtherEvent(text: string): void {
env: makeEnv(),
cwd: fixtureCwd,
encoding: 'utf-8',
timeout: 30_000,
},
);
}
@@ -150,7 +152,7 @@ describe('no-event paths', () => {
tool_use_id: 'tu-x',
}),
],
{ env: makeEnv(), cwd: fixtureCwd, encoding: 'utf-8' },
{ env: makeEnv(), cwd: fixtureCwd, encoding: 'utf-8', timeout: 30_000 },
);
const r = run([]);
expect(r.status).toBe(0);
@@ -169,7 +171,7 @@ describe('--dry-run', () => {
// Strip ANTHROPIC_API_KEY to prove no API call happens.
const env = makeEnv();
delete env.ANTHROPIC_API_KEY;
const res = spawnSync(BIN, ['--dry-run'], { env, cwd: fixtureCwd, encoding: 'utf-8' });
const res = spawnSync(BIN, ['--dry-run'], { env, cwd: fixtureCwd, encoding: 'utf-8', timeout: 30_000 });
expect(res.status).toBe(0);
expect(res.stdout).toContain('DISTILL PROMPT');
expect(res.stdout).toContain('always include tests');
@@ -185,7 +187,7 @@ describe('API auth', () => {
writeAuqOtherEvent('Some free text response that needs distilling');
const env = makeEnv();
delete env.ANTHROPIC_API_KEY;
const res = spawnSync(BIN, [], { env, cwd: fixtureCwd, encoding: 'utf-8' });
const res = spawnSync(BIN, [], { env, cwd: fixtureCwd, encoding: 'utf-8', timeout: 30_000 });
expect(res.status).not.toBe(0);
expect(res.stderr).toMatch(/ANTHROPIC_API_KEY/);
expect(res.stderr).toMatch(/separate billing/);
test/e2e-tier-alignment.test.ts
+4 -4
View File
@@ -42,8 +42,8 @@ const SELF_GATE_RE = /EVALS_TIER\s*===\s*['"](gate|periodic)['"]/g;
const HELPER_GATE_RE = /\b(?:describeE2ETier|e2eTierEnabled)\(\s*['"](gate|periodic)['"]/g;
/**
* Ratchet, not amnesty (same contract as KNOWN_MATRIX_GAPS in
* test/evals-workflow-matrix.test.ts): skill-e2e files that are named in NO
* Ratchet, not amnesty (the contract KNOWN_MATRIX_GAPS pioneered before the
* legacy eval matrix and its test retired): skill-e2e files that are named in NO
* touchfiles dep list because no E2E map key exists for them. Every entry
* carries a one-line reason. Do NOT add new files here — give the test an
* E2E map key (touchfiles + tier) and register the file in its dep list.
@@ -141,8 +141,8 @@ describe('E2E tier alignment (touchfiles declaration vs test self-gate)', () =>
expect(misaligned).toEqual([]);
});
// Ratchet cleanup enforcement (same contract as evals-workflow-matrix's
// burn-down test): a KNOWN_UNREGISTERED entry whose file was deleted, or
// Ratchet cleanup enforcement (same shrink-only contract as the retired
// matrix test's burn-down): a KNOWN_UNREGISTERED entry whose file was deleted, or
// whose file is now named in a dep list, is stale — delete the entry so
// the set can only shrink.
test('KNOWN_UNREGISTERED holds only live, still-unregistered files', () => {
test/egress-receipt.test.ts
+4 -4
View File
@@ -222,12 +222,12 @@ describe('gstack-egress-receipt shell bridge', () => {
fs.writeFileSync(payload, '[{"v":1}]');
const write = spawnSync(bin, ['write', '--sink', 'telemetry-sync', '--host', '127.0.0.1:8399',
'--class', 'telemetry-events', '--payload-file', payload, '--consent', 'telemetry=community'],
{ encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } });
{ encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GSTACK_HOME: home } });
expect(write.status).toBe(0);
const id = write.stdout.trim();
expect(id).toMatch(/^[0-9a-f]{64}$/);
const outcome = spawnSync(bin, ['outcome', id, '204'],
{ encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } });
{ encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GSTACK_HOME: home } });
expect(outcome.status).toBe(0);
const receipts = listReceipts(home);
expect(receipts.length).toBe(1);
@@ -239,7 +239,7 @@ describe('gstack-egress-receipt shell bridge', () => {
test('--no-payload records sha256:null (git-class: a subprocess owns the bytes)', () => {
const write = spawnSync(bin, ['write', '--sink', 'brain-sync', '--host', 'github.com',
'--class', 'git-push', '--no-payload', '--consent', 'artifacts_sync_mode=auto'],
{ encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } });
{ encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GSTACK_HOME: home } });
expect(write.status).toBe(0);
const receipts = listReceipts(home);
expect(receipts.length).toBe(1);
@@ -251,7 +251,7 @@ describe('gstack-egress-receipt shell bridge', () => {
if (!canRevokeWrites()) return; // chmod is advisory here (win32, root, DAC-override containers)
fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 });
const write = spawnSync(bin, ['write', '--sink', 's', '--host', 'h', '--class', 'c', '--no-payload'],
{ encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } });
{ encoding: 'utf-8', timeout: 30_000, env: { ...process.env, GSTACK_HOME: home } });
expect(write.status).toBe(3);
expect(write.stderr).toContain('EGRESS_RECEIPT_FAILED');
fs.chmodSync(path.join(home, 'security'), 0o700);
test/empty-find-fallthrough.test.ts
+3 -2
View File
@@ -35,7 +35,7 @@ describe('empty find must not fall through to cwd (#2483)', () => {
test('no resolver emits a bare `xargs ls -t` (must be `xargs -r ls -t`)', () => {
const out = execSync(
`grep -rn "xargs ls -t" "${path.join(ROOT, 'scripts')}" "${path.join(ROOT, 'bin')}" || true`,
{ encoding: 'utf-8' },
{ encoding: 'utf-8', timeout: 30_000 },
);
expect(out.trim()).toBe('');
});
@@ -67,6 +67,7 @@ describe('empty find must not fall through to cwd (#2483)', () => {
const out = execSync(`bash -c '${script.replace(/'/g, `'\\''`)}'`, {
cwd,
encoding: 'utf-8',
timeout: 30_000,
});
expect(out).toContain('LATEST_CP=[]');
expect(out).not.toContain('DECOY.md');
@@ -79,7 +80,7 @@ describe('empty find must not fall through to cwd (#2483)', () => {
test('no generated SKILL.md carries the unguarded form', () => {
const out = execSync(
`grep -rln "xargs ls -t" --include=SKILL.md "${ROOT}" || true`,
{ encoding: 'utf-8' },
{ encoding: 'utf-8', timeout: 30_000 },
);
// node_modules and vendored trees are not generated output; nothing in
// the repo's generated skills may carry the unguarded form.
test/eval-budgets-policy.test.ts
+1 -1
View File
@@ -41,7 +41,7 @@ describe('eval budget tiers', () => {
});
test('no paid-test timeout literal exceeds the ceiling tier', () => {
const out = spawnSync('git', ['ls-files', 'test/*.test.ts'], { cwd: ROOT, encoding: 'utf-8' });
const out = spawnSync('git', ['ls-files', 'test/*.test.ts'], { cwd: ROOT, encoding: 'utf-8', timeout: 30_000 });
const files = out.stdout.split('\n').filter((f) => f && isPaidTestFile(f));
expect(files.length).toBeGreaterThan(50); // scan-rot guard
test/eval-flake-rank.test.ts
+70
View File
@@ -0,0 +1,70 @@
/**
* Unit tests for the flake-rank aggregator (WS1's dial). The CLI ranks tests
* by retried passes (the flake signature) across finalized eval-store runs —
* these pin the accounting: N attempt records = 1 run of that test, the
* FINAL attempt decides pass/fail, retried passes count separately, partials
* and runner artifacts are excluded, shard dirs recurse, and the recency
* bound drops stale files.
*/
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { aggregate, collectEvalFiles } from '../scripts/eval-flake-rank';
const entry = (name: string, passed: boolean, attempt: number) => ({
name, suite: 's', tier: 'e2e', passed, attempt, duration_ms: 1000, cost_usd: 0.1,
});
const run = (tests: object[], extra: object = {}) => JSON.stringify({
schema_version: 2, version: '1.0.0', branch: 'b', git_sha: 'x', hostname: 'h',
timestamp: '2026-08-31T00:00:00Z', tier: 'e2e',
total_tests: tests.length, passed: 0, failed: 0, total_cost_usd: 0, total_duration_ms: 0,
tests, ...extra,
});
describe('eval-flake-rank aggregate', () => {
test('final attempt decides; retried pass counts as retriedPass, not a fail', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flakerank-'));
fs.writeFileSync(path.join(dir, 'run1.json'), run([
entry('flaky', false, 1), entry('flaky', true, 2), // pass on retry
entry('steady', true, 1),
entry('broken', false, 1), entry('broken', false, 2), // fails even retried
]));
fs.writeFileSync(path.join(dir, 'run2.json'), run([
entry('flaky', true, 1), entry('steady', true, 1),
]));
const series = aggregate(collectEvalFiles(dir));
expect(series.get('flaky')).toMatchObject({ runs: 2, passes: 2, fails: 0, retriedPasses: 1, totalAttempts: 3 });
expect(series.get('steady')).toMatchObject({ runs: 2, passes: 2, fails: 0, retriedPasses: 0 });
expect(series.get('broken')).toMatchObject({ runs: 1, passes: 0, fails: 1, retriedPasses: 0, totalAttempts: 2 });
fs.rmSync(dir, { recursive: true, force: true });
});
test('partials and runner artifacts are excluded; shard dirs recurse', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flakerank-'));
fs.mkdirSync(path.join(dir, 'shards', 'slug-a'), { recursive: true });
fs.writeFileSync(path.join(dir, 'shards', 'slug-a', 'run.json'), run([entry('sharded', true, 1)]));
fs.writeFileSync(path.join(dir, '_partial-e2e.json'), run([entry('inflight', false, 1)], { _partial: true }));
fs.writeFileSync(path.join(dir, 'manifest.json'), '{"version":1}');
fs.writeFileSync(path.join(dir, 'slice-3.json'), '{"version":1}');
const files = collectEvalFiles(dir);
expect(files).toHaveLength(1);
const series = aggregate(files);
expect(series.has('sharded')).toBe(true);
expect(series.has('inflight')).toBe(false);
fs.rmSync(dir, { recursive: true, force: true });
});
test('recency bound drops files older than sinceDays', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flakerank-'));
const stale = path.join(dir, 'old.json');
fs.writeFileSync(stale, run([entry('ancient', true, 1)]));
const old = new Date(Date.now() - 90 * 86_400_000);
fs.utimesSync(stale, old, old);
fs.writeFileSync(path.join(dir, 'new.json'), run([entry('recent', true, 1)]));
const files = collectEvalFiles(dir, 60);
expect(files.map((f) => path.basename(f))).toEqual(['new.json']);
fs.rmSync(dir, { recursive: true, force: true });
});
});
test/eval-list-cli.test.ts
+1
View File
@@ -65,6 +65,7 @@ function runEvalList(...args: string[]): { stdout: string; stderr: string; statu
GSTACK_HOME: path.join(tmpHome, '.gstack'),
},
encoding: 'utf-8',
timeout: 30_000,
});
return {
stdout: result.stdout ?? '',
test/evals-workflow-matrix.test.ts
-197
View File
@@ -1,197 +0,0 @@
/**
* CI eval-matrix completeness tripwire — kills the silent-skip class where a
* gate-tier test exists in the repo but the hand-enumerated matrix in
* .github/workflows/evals.yml never runs it, so "gate tier blocks merge" is
* quietly false in CI. This has happened before (see the "rehomed from the
* deleted pre-split monolith" comment in evals.yml) and was found again on
* PR #2700: nine gate-hosting files absent from the matrix, plus matrix rows
* whose whole-file tier guards can never fire because the Run step exported
* no EVALS_TIER.
*
* Ratchet, not amnesty: the KNOWN_* lists below enumerate the PRE-EXISTING
* gaps with reasons, so no NEW gap can land while the backlog burns down
* (same pattern as SCANNER_EXEMPT in egress-receipt-wiring). If you fix a
* listed gap (add its matrix row / tier property), this test FAILS until you
* remove the entry — stale exemptions are enforced, not decorative.
*
* Wiring pinned:
* - every matrix `file:` path exists on disk (no stale rows),
* - every gate-hosting paid file (whole-file gate self-gate, or named in the
* dep list of a gate-tier E2E_TOUCHFILES key) appears in the matrix or in
* KNOWN_MATRIX_GAPS,
* - every matrix file with a whole-file tier guard has a matching row-level
* `tier:` property (else the suite self-skips and the job is hollow-green)
* or sits in KNOWN_TIER_UNSET.
*/
import { describe, test, expect } from 'bun:test';
import * as fs from 'fs';
import * as path from 'path';
import { E2E_TOUCHFILES, E2E_TIERS } from './helpers/touchfiles-data';
import { isPaidTestFile } from './helpers/paid-test-set';
const ROOT = path.join(import.meta.dir, '..');
const WORKFLOW = path.join(ROOT, '.github', 'workflows', 'evals.yml');
/**
* Pre-existing gate-hosting files with no matrix row (found 2026-08-26,
* PR #2700). Adding a row activates real paid runs on every PR — a cost and
* flake-surface decision per file, tracked in TODOS.md ("CI gate-lane
* hollow-coverage burn-down"). Fix = add a matrix row (plus `tier: gate` when
* the file is whole-file gated), then DELETE the entry here.
*/
const KNOWN_MATRIX_GAPS = new Set([
'test/skill-e2e-ask-user-question-format-compliance.test.ts',
'test/skill-e2e-hermetic-canary.test.ts',
'test/skill-e2e-ios.test.ts',
'test/skill-e2e-plan-ceo-finding-floor.test.ts',
'test/skill-e2e-plan-ceo-plan-mode.test.ts',
'test/skill-e2e-plan-design-with-ui.test.ts',
'test/skill-e2e-plan-devex-finding-floor.test.ts',
'test/skill-e2e-plan-devex-plan-mode.test.ts',
// Exposed by the 2026-08 dep-list self-registration sweep: these eight had
// zero gate-key dep-list membership before it, so the census never saw
// them as gate-hosting. Their gate tests run in NO CI lane today. The
// paid-lane re-platform (test-paid-shards.ts as the CI engine) runs every
// gate-tier file by construction and retires this whole ratchet.
'test/skill-e2e-cso.test.ts',
'test/skill-e2e-diagram.test.ts',
'test/skill-e2e-learnings.test.ts',
'test/skill-e2e-plan-tune.test.ts',
'test/skill-e2e-plan-tune-cathedral.test.ts',
'test/skill-e2e-review-army.test.ts',
'test/skill-e2e-session-intelligence.test.ts',
'test/skill-e2e-skillify.test.ts',
]);
/**
* Matrix files whose whole-file tier guard has no matching row `tier:`
* property. Burned down to empty 2026-08-29: the vestigial codex/gemini rows
* were deleted (periodic-tier files, zero tests per PR) and
* e2e-pty-plan-smoke gained its `tier: gate`. The ratchet stays so a future
* row/file tier mismatch fails the suite instead of shipping hollow green.
*/
const KNOWN_TIER_UNSET = new Map<string, string>([]);
interface MatrixRow {
name: string;
files: string[];
tier?: string;
}
/** Parse the `matrix: suite:` rows (name / file / optional tier) from evals.yml. */
function parseMatrixRows(source: string): MatrixRow[] {
const rows: MatrixRow[] = [];
let current: MatrixRow | null = null;
for (const line of source.split('\n')) {
const name = line.match(/^\s+- name: (\S+)\s*$/);
if (name) {
if (current) rows.push(current);
current = { name: name[1], files: [] };
continue;
}
if (!current) continue;
const file = line.match(/^\s+file: (.+?)\s*$/);
if (file) current.files.push(...file[1].trim().split(/\s+/));
const tier = line.match(/^\s+tier: (\S+)\s*$/);
if (tier) current.tier = tier[1];
// `steps:` ends the strategy block — stop before step-level keys leak in.
if (/^\s{4}steps:\s*$/.test(line)) break;
}
if (current) rows.push(current);
return rows.filter((r) => r.files.length > 0);
}
const wholeFileTier = (source: string): string | null => {
const m =
/\b(?:describeE2ETier|e2eTierEnabled)\(\s*['"`](gate|periodic)['"`]/.exec(source) ||
/EVALS_TIER\s*===\s*['"`](gate|periodic)['"`]/.exec(source);
return m ? m[1] : null;
};
const workflowSource = fs.readFileSync(WORKFLOW, 'utf-8');
const rows = parseMatrixRows(workflowSource);
const matrixFiles = new Map<string, MatrixRow>();
for (const row of rows) for (const f of row.files) matrixFiles.set(f, row);
const paidFiles = fs
.readdirSync(path.join(ROOT, 'test'))
.filter((f) => f.endsWith('.test.ts'))
.map((f) => `test/${f}`)
.filter(isPaidTestFile);
describe('evals.yml matrix completeness (gate-lane silent-skip tripwire)', () => {
test('matrix parse sanity: rows and known suites present', () => {
expect(rows.length).toBeGreaterThanOrEqual(15);
expect(matrixFiles.has('test/skill-e2e-workflow.test.ts')).toBe(true);
expect(matrixFiles.has('test/skill-e2e-ship-docsync.test.ts')).toBe(true);
});
test('every matrix file exists on disk', () => {
const missing = [...matrixFiles.keys()].filter(
(f) => !fs.existsSync(path.join(ROOT, f))
);
expect(missing).toEqual([]);
});
test('every gate-hosting paid file is in the matrix (or the documented backlog)', () => {
const gaps: string[] = [];
for (const file of paidFiles) {
const source = fs.readFileSync(path.join(ROOT, file), 'utf-8');
const declaresGate = wholeFileTier(source) === 'gate';
const inGateDeps = Object.entries(E2E_TOUCHFILES).some(
([key, deps]) =>
(E2E_TIERS as Record<string, string>)[key] === 'gate' &&
(deps as string[]).includes(file)
);
if (!declaresGate && !inGateDeps) continue;
if (matrixFiles.has(file) || KNOWN_MATRIX_GAPS.has(file)) continue;
gaps.push(file);
}
expect(
gaps,
`Gate-hosting test file(s) missing from the evals.yml matrix — CI will ` +
`never run them and "gate tier blocks merge" becomes silently false. ` +
`Add a matrix row (with tier: gate when the file is whole-file gated). ` +
`Do NOT extend KNOWN_MATRIX_GAPS for new files.`
).toEqual([]);
});
test('matrix rows for whole-file-gated files carry a matching tier property', () => {
const mismatches: string[] = [];
for (const [file, row] of matrixFiles) {
if (!fs.existsSync(path.join(ROOT, file))) continue;
const declared = wholeFileTier(fs.readFileSync(path.join(ROOT, file), 'utf-8'));
if (!declared) continue;
if (row.tier === declared) continue;
if (KNOWN_TIER_UNSET.get(file) === declared && row.tier === undefined) continue;
mismatches.push(`${file} declares '${declared}' but row '${row.name}' has tier: ${row.tier ?? 'unset'}`);
}
expect(
mismatches,
`A whole-file tier guard with no matching row tier means the suite ` +
`self-skips and the CI job reports a hollow green. Set tier: <declared> ` +
`on the row (the Run step exports it as EVALS_TIER).`
).toEqual([]);
});
test('burn-down lists hold only live gaps (ratchet cleanup enforcement)', () => {
const staleGaps = [...KNOWN_MATRIX_GAPS].filter(
(f) => matrixFiles.has(f) || !fs.existsSync(path.join(ROOT, f))
);
expect(
staleGaps,
'Entry fixed or file removed — delete it from KNOWN_MATRIX_GAPS.'
).toEqual([]);
const staleTiers = [...KNOWN_TIER_UNSET.entries()].filter(([f, declared]) => {
const row = matrixFiles.get(f);
if (!row) return true; // row deleted — entry no longer applies
if (row.tier === declared) return true; // fixed — entry must go
if (!fs.existsSync(path.join(ROOT, f))) return true;
return wholeFileTier(fs.readFileSync(path.join(ROOT, f), 'utf-8')) !== declared;
});
expect(
staleTiers.map(([f]) => f),
'Entry fixed, row removed, or guard changed — delete it from KNOWN_TIER_UNSET.'
).toEqual([]);
});
});
test/evals-workflow-wiring.test.ts
+169
View File
@@ -0,0 +1,169 @@
/**
* Sliced-lane wiring pins for the paid CI workflows — the successor to
* evals-workflow-matrix.test.ts, which enforced completeness of a
* hand-enumerated 17-row matrix (and carried KNOWN_MATRIX_GAPS /
* KNOWN_TIER_UNSET burn-down ratchets for the files that matrix missed).
* The matrix is deleted: the sliced lane's planner derives the gate census
* from the runner itself (collectPaidTestFiles + tier selection), so "every
* gate-hosting file is in the census" is true BY CONSTRUCTION and the
* burn-down ratchets retired with the rows.
*
* What still needs pinning is the WIRING — the yml plumbing that free tests
* are the only guard for:
* - the legacy matrix (and its `needs: evals` serialization) stays deleted,
* - planner/executor/report all run tier=gate and agree on the slice count,
* - both surviving lanes register skills through the SHARED composite that
* carries the fail-fast dangling-symlink/frontmatter verification loop
* (the sliced + periodic copies had silently dropped it — the loop was
* written after a silent "Unknown command" + 35-min-timeout incident),
* - the PR comment survives the matrix-report deletion (it moved into
* slices-report, keyed on the same "## E2E Evals" upsert marker).
*/
import { describe, test, expect } from 'bun:test';
import * as fs from 'fs';
import * as path from 'path';
const ROOT = path.join(import.meta.dir, '..');
const read = (rel: string) => fs.readFileSync(path.join(ROOT, rel), 'utf-8');
const evalsYml = read('.github/workflows/evals.yml');
const periodicYml = read('.github/workflows/evals-periodic.yml');
const registerAction = read('.github/actions/register-gstack-skills/action.yml');
/** Slice count the planner emits (`--slices N`) in a workflow source. */
function plannedSlices(source: string): number[] {
return [...source.matchAll(/--emit-plan\s+\S+\s+--slices\s+(\d+)/g)].map((m) => Number(m[1]));
}
/** The executor matrix's slice list (`slice: [1, 2, ...]`). */
function matrixSlices(source: string): number[][] {
return [...source.matchAll(/^\s+slice: \[([\d,\s]+)\]\s*$/gm)].map((m) =>
m[1].split(',').map((n) => Number(n.trim())),
);
}
describe('evals.yml sliced-lane wiring (post-matrix)', () => {
test('the legacy matrix job stays deleted', () => {
// Row-enumeration shapes from the deleted matrix. Any reappearance means
// someone is re-growing a hand-maintained enumeration next to a lane
// whose census is derived — the drift class the deletion killed.
expect(evalsYml).not.toMatch(/^\s+suite:\s*$/m);
expect(evalsYml).not.toMatch(/^\s+file: test\//m);
expect(evalsYml).not.toContain('needs: [build-image, evals]');
expect(evalsYml).not.toMatch(/^\s+needs: evals\s*$/m);
});
test('no workflow-level EVALS_TIER env (each command sets its own)', () => {
// The workflow-level `EVALS_TIER: gate` was dead config once every
// consumer set its own; a resurrected copy would silently leak gate
// semantics into steps that must choose explicitly.
expect(evalsYml).not.toMatch(/^env:[\s\S]{0,120}^\s+EVALS_TIER:/m);
});
test('planner, executors, and report all run tier=gate on the shared runner', () => {
expect(evalsYml).toMatch(/EVALS_TIER=gate bun run scripts\/test-paid-shards\.ts --tier gate --emit-plan/);
expect(evalsYml).toMatch(/EVALS_TIER=gate bun run scripts\/test-paid-shards\.ts --tier gate --plan .* --slice /);
expect(evalsYml).toMatch(/EVALS_TIER=gate bun run scripts\/test-paid-shards\.ts --tier gate --report /);
});
test('executor matrix slice list matches the planner --slices count', () => {
const planned = plannedSlices(evalsYml);
const matrices = matrixSlices(evalsYml);
expect(planned, 'expected exactly one --emit-plan site in evals.yml').toHaveLength(1);
expect(matrices, 'expected exactly one slice matrix in evals.yml').toHaveLength(1);
const n = planned[0];
expect(matrices[0]).toEqual(Array.from({ length: n }, (_, i) => i + 1));
});
test('reconcile exit is captured via PIPESTATUS, never $? after a pipe', () => {
// GitHub's default run-step shell is `bash -e {0}` with NO pipefail, so
// `$?` after `... | tee` is tee's exit — always 0. That made the
// fail-closed reconcile gate silently fail-open (ship review army,
// 2026-08-31). Both lanes must read PIPESTATUS[0].
for (const [name, source] of [['evals.yml', evalsYml], ['evals-periodic.yml', periodicYml]] as const) {
const reconcileBlocks = [...source.matchAll(/--report[^\n]*\| tee[^\n]*\n([\s\S]{0,400}?)GITHUB_OUTPUT/g)];
expect(reconcileBlocks.length, `${name}: expected a tee'd reconcile step`).toBeGreaterThanOrEqual(1);
for (const block of reconcileBlocks) {
expect(block[1], `${name} reconcile captures tee's exit, not the runner's`).toContain('PIPESTATUS[0]');
expect(block[1]).not.toMatch(/exit=\$\?/);
}
}
});
test('the PR comment survived the matrix-report deletion (moved to slices-comment)', () => {
// Keyed on the upsert marker so the migration keeps updating the SAME
// comment; and the job holding it needs the issues permission (#1802).
expect(evalsYml).toContain('## E2E Evals');
expect(evalsYml).toMatch(/pull-requests: write/);
expect(evalsYml).toMatch(/issues: write/);
});
test('the write-token job runs ZERO repo code (token/exec separation)', () => {
// slices-report executes PR-authored code (bun install + the reconcile
// runner), so it must hold contents:read ONLY; the write token lives in
// slices-comment, which may only download artifacts and run jq/gh —
// $GITHUB_ENV persistence is job-scoped, so this split IS the trust
// boundary (codex adversarial, 2026-08-31; the matrix-era report job had
// this property and the consolidation briefly regressed it).
const commentJob = evalsYml.slice(evalsYml.indexOf(' slices-comment:'));
expect(commentJob.length).toBeGreaterThan(100);
expect(commentJob).not.toContain('actions/checkout');
expect(commentJob).not.toContain('bun install');
expect(commentJob).not.toMatch(/run: .*bun run/);
expect(commentJob).not.toContain('uses: ./');
// No checkout also means no git context: `gh pr comment` resolves the
// repo FROM git and dies with "not a git repository" here (PR #2746's
// first run). Every comment call must be explicit-repo REST (gh api).
expect(commentJob).not.toContain('gh pr comment');
// And the code-executing report job must NOT hold write scopes.
const reportJob = evalsYml.slice(evalsYml.indexOf(' slices-report:'), evalsYml.indexOf(' slices-comment:'));
expect(reportJob).not.toMatch(/pull-requests: write/);
expect(reportJob).not.toMatch(/issues: write/);
});
});
describe('evals-periodic.yml sliced-lane wiring', () => {
test('planner/executor/report tier=periodic and slice counts agree', () => {
expect(periodicYml).toMatch(/EVALS_TIER=periodic bun run scripts\/test-paid-shards\.ts --tier periodic --emit-plan/);
expect(periodicYml).toMatch(/EVALS_TIER=periodic bun run scripts\/test-paid-shards\.ts --tier periodic --plan .* --slice /);
expect(periodicYml).toMatch(/EVALS_TIER=periodic bun run scripts\/test-paid-shards\.ts --tier periodic --report /);
const planned = plannedSlices(periodicYml);
const matrices = matrixSlices(periodicYml);
expect(planned).toHaveLength(1);
expect(matrices).toHaveLength(1);
expect(matrices[0]).toEqual(Array.from({ length: planned[0] }, (_, i) => i + 1));
});
});
describe('shared setup composites (both surviving lanes)', () => {
test('both lanes register skills through the shared composite', () => {
for (const [name, source] of [['evals.yml', evalsYml], ['evals-periodic.yml', periodicYml]] as const) {
expect(source, `${name} must use the register-gstack-skills composite`)
.toContain('uses: ./.github/actions/register-gstack-skills');
// No inline re-implementation creeping back beside the composite.
expect(source, `${name} re-inlines the skill registry instead of using the composite`)
.not.toContain('ln -snf "$REPO" "$SKILLS_DIR/gstack"');
}
});
test('the register composite carries the fail-fast verification loop', () => {
// The loop is the POINT of the composite: a dangling symlink or renamed
// committed target fails in seconds with a named path, never as a wedged
// PTY session at the shard wall. Pin its load-bearing markers.
expect(registerAction).toContain('skill registry OK');
expect(registerAction).toContain('skill-registry target missing');
expect(registerAction).toContain('gstack root symlink dangles');
expect(registerAction).toMatch(/grep -m1 "\^name: \$s\\\$"/);
});
test('seed/deps/temp composites exist and both lanes use them', () => {
for (const action of ['seed-claude-config', 'restore-deps', 'fix-bun-temp']) {
expect(fs.existsSync(path.join(ROOT, '.github', 'actions', action, 'action.yml')), `missing composite: ${action}`).toBe(true);
}
for (const [name, source] of [['evals.yml', evalsYml], ['evals-periodic.yml', periodicYml]] as const) {
expect(source, `${name} must use seed-claude-config`).toContain('uses: ./.github/actions/seed-claude-config');
expect(source, `${name} must use restore-deps`).toContain('uses: ./.github/actions/restore-deps');
expect(source, `${name} must use fix-bun-temp`).toContain('uses: ./.github/actions/fix-bun-temp');
}
});
});
test/flake-ledger.test.ts
+64
View File
@@ -0,0 +1,64 @@
/**
* WS1 flake-ledger unit tests: the free runner's FLAKY-PASS events become a
* durable JSONL series (single writer: the parent runner). Fail-open is the
* contract — a broken ledger warns loudly but must never turn a real verdict
* into a failure on the only required lane.
*/
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { appendFlakeLedger, flakeLedgerPath, type FlakeLedgerEntry } from '../scripts/test-free-shards';
const entry = (file: string): FlakeLedgerEntry => ({
ts: '2026-08-31T00:00:00.000Z',
runner: 'free',
kind: 'flaky-pass',
file,
shard: 2,
});
describe('flake ledger', () => {
test('appends one JSONL line per entry, creating parent dirs', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flake-ledger-'));
const ledger = path.join(dir, 'nested', 'ledger.jsonl');
expect(appendFlakeLedger([entry('test/a.test.ts')], ledger)).toBe(true);
expect(appendFlakeLedger([entry('test/b.test.ts'), entry('test/c.test.ts')], ledger)).toBe(true);
const lines = fs.readFileSync(ledger, 'utf-8').trim().split('\n');
expect(lines).toHaveLength(3);
expect(JSON.parse(lines[0])).toMatchObject({ runner: 'free', kind: 'flaky-pass', file: 'test/a.test.ts', shard: 2 });
fs.rmSync(dir, { recursive: true, force: true });
});
test('empty entry list is a no-op success (no file created)', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flake-ledger-'));
const ledger = path.join(dir, 'ledger.jsonl');
expect(appendFlakeLedger([], ledger)).toBe(true);
expect(fs.existsSync(ledger)).toBe(false);
fs.rmSync(dir, { recursive: true, force: true });
});
test('FAIL-OPEN: an unwritable path warns and returns false, never throws', () => {
const warnings: string[] = [];
// A path whose parent is a FILE cannot be mkdir'd — deterministic EEXIST/ENOTDIR.
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'flake-ledger-'));
const blocker = path.join(dir, 'blocker');
fs.writeFileSync(blocker, 'not a dir');
const ledger = path.join(blocker, 'ledger.jsonl');
const ok = appendFlakeLedger([entry('test/a.test.ts')], ledger, (l) => warnings.push(l));
expect(ok).toBe(false);
expect(warnings).toHaveLength(1);
expect(warnings[0]).toContain('verdict unaffected');
fs.rmSync(dir, { recursive: true, force: true });
});
test('env override wins; local default is project-scoped, never machine-global', () => {
expect(flakeLedgerPath({ GSTACK_FLAKE_LEDGER: '/x/y.jsonl' } as NodeJS.ProcessEnv)).toBe('/x/y.jsonl');
// Without the env override, the default lives under the PROJECT dir
// (sibling worktrees of different repos must not interleave one series);
// tmpdir is only the last-resort fallback when slug detection fails.
const local = flakeLedgerPath({} as NodeJS.ProcessEnv);
expect(local).toMatch(/flake-ledger\.jsonl$/);
expect(local.includes(path.join('.gstack', 'projects')) || local.startsWith(os.tmpdir())).toBe(true);
});
});
test/founder-resources-optout.test.ts
+2
View File
@@ -30,6 +30,7 @@ afterEach(() => { fs.rmSync(tmpHome, { recursive: true, force: true }); });
function cfg(args: string[]): string {
return execFileSync(CONFIG_BIN, args, {
timeout: 30_000,
encoding: "utf-8",
env: { ...process.env, GSTACK_HOME: tmpHome },
}).trim();
@@ -47,6 +48,7 @@ describe("founder_resources config key (#538)", () => {
test("invalid values are rejected to the default, never persisted as-is", () => {
execFileSync(CONFIG_BIN, ["set", "founder_resources", "banana"], {
timeout: 30_000,
encoding: "utf-8", stdio: ["ignore", "pipe", "pipe"],
env: { ...process.env, GSTACK_HOME: tmpHome },
});
test/free-tests-workflow-wiring.test.ts
+11
View File
@@ -54,6 +54,17 @@ describe('free-tests workflow wiring', () => {
}
});
test('flake telemetry stays wired: retry flag, single-writer ledger, unconditional artifact', () => {
// WS1: a timing flake must not red the required lane, but every
// flaky-pass must be recorded and uploaded — a green run is exactly when
// the evidence matters. Removing any of these silently returns flakes to
// either merge-blocking (flag off) or invisibility (ledger/artifact off).
expect(source).toMatch(/GSTACK_FREE_RETRY_FLAKY:\s*"1"/);
expect(source).toMatch(/GSTACK_FLAKE_LEDGER:\s*\$\{\{ runner\.temp \}\}\/flake-ledger\.jsonl/);
expect(source).toContain('name: flake-ledger');
expect(source).toMatch(/name: Upload flake ledger\s*\n\s*if: always\(\)/);
});
test('least-privilege token: contents read-only, credentials not persisted', () => {
// The job executes PR-controlled code (install lifecycle scripts + the
// suite itself). A default-grant GITHUB_TOKEN persisted into .git/config
test/fs-utils.test.ts
+4 -3
View File
@@ -74,11 +74,11 @@ describe("swept mkdirp sites under bun-on-Windows EEXIST semantics (#2635)", ()
fs.mkdirSync(work, { recursive: true });
const payload = '{"decision":"eexist probe","rationale":"r","scope":"repo","source":"user"}';
const env = { ...process.env, HOME: base };
const first = spawnSync("bun", [DECISION_LOG, payload], { cwd: work, encoding: "utf8", env });
const first = spawnSync("bun", [DECISION_LOG, payload], { cwd: work, encoding: "utf8", env, timeout: 30_000 });
expect(first.status).toBe(0);
const second = spawnSync(
"bun", ["--preload", EEXIST_PRELOAD, DECISION_LOG, payload],
{ cwd: work, encoding: "utf8", env },
{ cwd: work, encoding: "utf8", env, timeout: 30_000 },
);
expect(second.status).toBe(0);
expect(second.stderr ?? "").not.toContain("EEXIST");
@@ -93,7 +93,7 @@ describe("install-prepush-hook under bun-on-Windows EEXIST semantics (#2635)", (
const base = tmpdir();
try {
const repo = path.join(base, "repo");
spawnSync("git", ["init", "-q", repo]);
spawnSync("git", ["init", "-q", repo], { timeout: 30_000 });
const hookDir = path.join(repo, ".git", "hooks");
fs.mkdirSync(hookDir, { recursive: true });
const hookPath = path.join(hookDir, "pre-push");
@@ -105,6 +105,7 @@ describe("install-prepush-hook under bun-on-Windows EEXIST semantics (#2635)", (
const r = spawnSync("bun", ["--preload", EEXIST_PRELOAD, REDACT, "install-prepush-hook"], {
cwd: repo,
encoding: "utf8",
timeout: 30_000,
});
expect(r.status).toBe(0);
expect(r.stderr ?? "").not.toContain("EEXIST");
test/gbrain-detect-install.test.ts
+1
View File
@@ -51,6 +51,7 @@ function run(bin: string, args: string[], opts: RunOpts = {}) {
env,
cwd: opts.cwd,
encoding: 'utf-8',
timeout: 30_000,
});
return {
stdout: (res.stdout || '').trim(),
test/gbrain-exec-invariant.test.ts
+3
View File
@@ -35,9 +35,12 @@ const GUARDED_FILES = [
// Patterns that would bypass lib/gbrain-exec.ts. Match the literal `"gbrain"`
// as the first argument since these helpers are the failure mode.
const BANNED_PATTERNS: Array<{ name: string; regex: RegExp }> = [
// tripwire-exempt: grep-needle STRING for this invariant, not a process spawn
{ name: 'spawnSync("gbrain", ...)', regex: /spawnSync\s*\(\s*["']gbrain["']/g },
{ name: 'spawn("gbrain", ...)', regex: /\bspawn\s*\(\s*["']gbrain["']/g },
// tripwire-exempt: grep-needle STRING for this invariant, not a process spawn
{ name: 'execFileSync("gbrain", ...)', regex: /execFileSync\s*\(\s*["']gbrain["']/g },
// tripwire-exempt: grep-needle STRING for this invariant, not a process spawn
{ name: 'execSync("...gbrain...")', regex: /execSync\s*\(\s*["'`][^"'`]*\bgbrain\b/g },
];
test/gbrain-init-rollback.test.ts
+1
View File
@@ -122,6 +122,7 @@ echo "ok"
HOME: env.home,
PATH: `${env.bindir}:/usr/bin:/bin`,
},
timeout: 30_000,
});
return {
exitCode: result.status ?? 1,
test/gbrain-init-voyage-code-3.test.ts
+3 -1
View File
@@ -121,6 +121,7 @@ gbrain init --pglite --json "$@"
const result = spawnSync(shell, ["-c", script], {
encoding: "utf-8",
env: baseEnv,
timeout: 30_000,
});
if (result.status !== 0) {
throw new Error(`init script exited ${result.status}: ${result.stderr}`);
@@ -133,7 +134,7 @@ function lastArgc(env: FakeEnv): number {
return parseInt(lines[lines.length - 1], 10);
}
const HAVE_ZSH = spawnSync("zsh", ["-c", "true"]).status === 0;
const HAVE_ZSH = spawnSync("zsh", ["-c", "true"], { timeout: 30_000 }).status === 0;
describe("voyage-code-3 default for gstack-driven PGLite init", () => {
it("passes voyage-code-3 flags when VOYAGE_API_KEY is set", () => {
@@ -193,6 +194,7 @@ gbrain init --pglite --json $GBRAIN_EMBED_FLAGS
const result = spawnSync("zsh", ["-c", brokenShape], {
encoding: "utf-8",
env: { ...process.env, HOME: env.home, PATH: `${env.bindir}:/usr/bin:/bin` },
timeout: 30_000,
});
expect(result.status).toBe(0);
expect(lastArgc(env)).toBe(4); // init, --pglite, --json, "<entire flag string>"
test/gbrain-lib-verify.test.ts
+2
View File
@@ -28,6 +28,7 @@ function runVerify(arg: string, stdin?: string) {
const res = spawnSync(VERIFY, arg === '' ? [] : [arg], {
input: stdin,
encoding: 'utf-8',
timeout: 30_000,
});
return {
stdout: (res.stdout || '').trim(),
@@ -43,6 +44,7 @@ function runLibSnippet(snippet: string, stdin: string = '') {
const res = spawnSync('bash', ['-c', script], {
input: stdin,
encoding: 'utf-8',
timeout: 30_000,
});
return {
stdout: (res.stdout || '').trim(),
test/gbrain-local-status.test.ts
+4
View File
@@ -210,6 +210,7 @@ describe("lib/gbrain-local-status — status classification", () => {
);
expect(source).not.toContain('command -v gbrain');
// tripwire-exempt: string assertion on lib source text, not a call
expect(source).toContain('execFileSync("gbrain", ["--version"]');
});
@@ -538,6 +539,7 @@ describe("lib/gbrain-local-status — thin-client (#2051)", () => {
GSTACK_HOME: env.gstackHome,
GSTACK_DETECT_NO_CACHE: "1",
},
timeout: 30_000,
});
expect(r.status).toBe(0);
});
@@ -554,6 +556,7 @@ describe("lib/gbrain-local-status — thin-client (#2051)", () => {
GSTACK_HOME: env.gstackHome,
GSTACK_DETECT_NO_CACHE: "1",
},
timeout: 30_000,
});
expect(r.status).toBe(1);
});
@@ -790,6 +793,7 @@ describe("lib/gbrain-local-status — bearer-token thin-client (#2520)", () => {
GSTACK_HOME: env.gstackHome,
GSTACK_DETECT_NO_CACHE: "1",
},
timeout: 30_000,
});
expect(r.status).toBe(0);
});
test/gbrain-repo-policy-client.test.ts
+1 -1
View File
@@ -32,7 +32,7 @@ function env(): NodeJS.ProcessEnv {
}
function run(args: string[], input?: string) {
const res = spawnSync(BIN, args, { env: env(), encoding: "utf-8", input });
const res = spawnSync(BIN, args, { env: env(), encoding: "utf-8", input, timeout: 30_000 });
return {
stdout: res.stdout || "",
stderr: res.stderr || "",
test/gbrain-repo-policy.test.ts
+3 -1
View File
@@ -29,6 +29,7 @@ function run(args: string[], opts: { env?: Record<string, string> } = {}) {
const res = spawnSync(BIN, args, {
env: { ...process.env, GSTACK_HOME: tmpHome, ...(opts.env || {}) },
encoding: 'utf-8',
timeout: 30_000,
});
return {
stdout: (res.stdout || '').trim(),
@@ -263,6 +264,7 @@ describe('get without arg (auto-detect from current dir)', () => {
env: { ...process.env, GSTACK_HOME: tmpHome },
cwd: cwdTmp,
encoding: 'utf-8',
timeout: 30_000,
});
expect((res.stdout || '').trim()).toBe('unset');
} finally {
@@ -288,7 +290,7 @@ describe('gstack-gbrain-sync code stage honors the repo policy (#2140 sync path)
function makeRepo(): void {
repoDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-policy-repo-'));
const git = (...args: string[]) =>
spawnSync('git', args, { cwd: repoDir, encoding: 'utf-8' });
spawnSync('git', args, { cwd: repoDir, encoding: 'utf-8', timeout: 30_000 });
git('init', '-q', '.');
git('remote', 'add', 'origin', REPO_URL);
fs.writeFileSync(path.join(repoDir, 'README.md'), 'fixture\n');
test/gbrain-source-worktree-advance.test.ts
+2 -1
View File
@@ -23,12 +23,13 @@ function run(argv: string[], env: Record<string, string> = {}) {
env: { ...process.env, HOME: tmpHome, GSTACK_HOME: tmpHome, ...env },
encoding: 'utf-8',
cwd: ROOT,
timeout: 30_000,
});
return { stdout: res.stdout || '', stderr: res.stderr || '', status: res.status ?? -1 };
}
function git(args: string[], cwd: string) {
const res = spawnSync('git', args, { cwd, encoding: 'utf-8' });
const res = spawnSync('git', args, { cwd, encoding: 'utf-8', timeout: 30_000 });
return { stdout: (res.stdout || '').trim(), status: res.status ?? -1 };
}
test/gbrain-sync-skip.test.ts
+3 -2
View File
@@ -25,7 +25,7 @@ import { join } from "path";
import { execFileSync, spawnSync } from "child_process";
const SCRIPT = join(import.meta.dir, "..", "bin", "gstack-gbrain-sync.ts");
const BUN_BIN = execFileSync("sh", ["-c", "command -v bun"], { encoding: "utf-8" }).trim();
const BUN_BIN = execFileSync("sh", ["-c", "command -v bun"], { encoding: "utf-8", timeout: 30_000 }).trim();
interface FakeEnv {
tmp: string;
@@ -112,9 +112,10 @@ function runOrchestrator(
): { stdout: string; stderr: string; exitCode: number } {
// Initialize a git repo in the sandbox so repoRoot() finds it (otherwise
// code stage skips with "not in git repo" before our check ever fires).
spawnSync("git", ["init", "-q", env.home], { encoding: "utf-8" });
spawnSync("git", ["init", "-q", env.home], { encoding: "utf-8", timeout: 30_000 });
spawnSync("git", ["-C", env.home, "commit", "--allow-empty", "-m", "init", "-q"], {
encoding: "utf-8",
timeout: 30_000,
env: { ...process.env, GIT_AUTHOR_NAME: "T", GIT_AUTHOR_EMAIL: "t@t", GIT_COMMITTER_NAME: "T", GIT_COMMITTER_EMAIL: "t@t" },
});
test/gbrain-sync-voyage-code-3-integration.test.ts
+6 -6
View File
@@ -44,7 +44,7 @@ import { tmpdir } from "os";
import { join } from "path";
import { spawnSync } from "child_process";
const gbrainPath = spawnSync("which", ["gbrain"], { encoding: "utf-8" }).stdout.trim();
const gbrainPath = spawnSync("which", ["gbrain"], { encoding: "utf-8", timeout: 30_000 }).stdout.trim();
const gbrainAvailable = gbrainPath.length > 0;
const voyageKey = process.env.VOYAGE_API_KEY?.trim() ?? "";
const voyageKeyPresent = voyageKey.length > 0;
@@ -114,14 +114,14 @@ The queue module is a simple FIFO job queue.
);
// Make it a git repo because gbrain's code-sync strategy expects one.
const gitInit = spawnSync("git", ["init", "-q"], { cwd: fixtureDir, encoding: "utf-8" });
const gitInit = spawnSync("git", ["init", "-q"], { cwd: fixtureDir, encoding: "utf-8", timeout: 30_000 });
if (gitInit.status !== 0) {
throw new Error(`git init failed: ${gitInit.stderr}`);
}
spawnSync("git", ["config", "user.email", "test@example.invalid"], { cwd: fixtureDir });
spawnSync("git", ["config", "user.name", "test"], { cwd: fixtureDir });
spawnSync("git", ["add", "."], { cwd: fixtureDir });
spawnSync("git", ["commit", "-q", "-m", "fixture"], { cwd: fixtureDir });
spawnSync("git", ["config", "user.email", "test@example.invalid"], { cwd: fixtureDir, timeout: 30_000 });
spawnSync("git", ["config", "user.name", "test"], { cwd: fixtureDir, timeout: 30_000 });
spawnSync("git", ["add", "."], { cwd: fixtureDir, timeout: 30_000 });
spawnSync("git", ["commit", "-q", "-m", "fixture"], { cwd: fixtureDir, timeout: 30_000 });
return {
root,
test/gemini-e2e.test.ts
+12 -6
View File
@@ -19,7 +19,7 @@ import { JUDGE_MS } from './helpers/eval-budgets';
import { runGeminiSkill } from './helpers/gemini-session-runner';
import type { GeminiResult } from './helpers/gemini-session-runner';
import { EvalCollector } from './helpers/eval-store';
import { selectTests, detectBaseBranch, getChangedFiles, GLOBAL_TOUCHFILES } from './helpers/touchfiles';
import { selectTests, detectBaseBranch, getChangedFiles, E2E_TOUCHFILES, GLOBAL_TOUCHFILES } from './helpers/touchfiles';
import { createTestWorktree, harvestAndCleanup } from './helpers/e2e-helpers';
import * as path from 'path';
@@ -29,7 +29,7 @@ const ROOT = path.resolve(import.meta.dir, '..');
const GEMINI_AVAILABLE = (() => {
try {
const result = Bun.spawnSync(['which', 'gemini']);
const result = Bun.spawnSync(['which', 'gemini'], { timeout: 30_000 });
return result.exitCode === 0;
} catch { return false; }
})();
@@ -74,10 +74,16 @@ if (!evalsEnabled) {
// --- Diff-based test selection ---
// Gemini E2E touchfiles — keyed by test name
const GEMINI_E2E_TOUCHFILES: Record<string, string[]> = {
'gemini-smoke': ['.agents/skills/**', 'test/helpers/gemini-session-runner.ts'],
};
// Gemini E2E touchfiles — DERIVED from the canonical map, never a local fork
// (the old hand-copy kept a gitignored '.agents/skills/**' pattern that can
// never match a git diff and missed canonical deps — same drift class as the
// codex copy).
const GEMINI_E2E_TOUCHFILES: Record<string, string[]> = Object.fromEntries(
(['gemini-smoke'] as const).map((key) => {
if (!E2E_TOUCHFILES[key]) throw new Error(`canonical E2E_TOUCHFILES lost key '${key}' — fix the map, not this file`);
return [key, E2E_TOUCHFILES[key]];
}),
);
let selectedTests: string[] | null = null; // null = run all
test/gen-skill-docs-import-purity.test.ts
+1 -1
View File
@@ -39,7 +39,7 @@ describe('gen-skill-docs import purity', () => {
}
console.log('IMPORT_PURE');
`;
const out = Bun.spawnSync(['bun', '-e', probe], { cwd: ROOT });
const out = Bun.spawnSync(['bun', '-e', probe], { cwd: ROOT, timeout: 120_000 });
const stdout = out.stdout.toString();
const stderr = out.stderr.toString();
expect(stderr, stderr).not.toContain('import mutated');
test/gen-skill-docs-out-dir.test.ts
+1 -1
View File
@@ -21,7 +21,7 @@ describe('gen-skill-docs --out-dir (B2 render isolation)', () => {
}
function porcelain(): string {
const r = spawnSync('git', ['status', '--porcelain'], { cwd: ROOT, encoding: 'utf-8' });
const r = spawnSync('git', ['status', '--porcelain'], { cwd: ROOT, encoding: 'utf-8', timeout: 30_000 });
return r.status === 0 ? r.stdout : '';
}
test/gen-skill-docs.test.ts
+12 -7
View File
@@ -139,7 +139,7 @@ const EXTERNAL_OUT = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-gen-docs-out-
{
const render = Bun.spawnSync(
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--out-dir', EXTERNAL_OUT],
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe' },
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 },
);
if (render.exitCode !== 0) {
throw new Error(
@@ -315,6 +315,7 @@ describe('gen-skill-docs', () => {
cwd: ROOT,
stdout: 'pipe',
stderr: 'pipe',
timeout: 120_000,
});
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
@@ -1965,7 +1966,7 @@ describe('Codex generation (--host codex)', () => {
'/tmp/gstack-claude-error-XXXXXX',
'/tmp/gstack-claude-diff-XXXXXX',
]) {
const result = spawnSync('mktemp', [template], { encoding: 'utf-8' });
const result = spawnSync('mktemp', [template], { encoding: 'utf-8', timeout: 30_000 });
expect(result.status).toBe(0);
const created = result.stdout.trim();
expect(created.startsWith(template.replace('XXXXXX', ''))).toBe(true);
@@ -1990,6 +1991,7 @@ describe('Codex generation (--host codex)', () => {
cwd: ROOT,
stdout: 'pipe',
stderr: 'pipe',
timeout: 120_000,
});
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
@@ -2005,11 +2007,13 @@ describe('Codex generation (--host codex)', () => {
cwd: ROOT,
stdout: 'pipe',
stderr: 'pipe',
timeout: 120_000,
});
const agentsResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'agents', '--dry-run', '--out-dir', EXTERNAL_OUT], {
cwd: ROOT,
stdout: 'pipe',
stderr: 'pipe',
timeout: 120_000,
});
expect(codexResult.exitCode).toBe(0);
expect(agentsResult.exitCode).toBe(0);
@@ -2224,6 +2228,7 @@ describe('Codex generation (--host codex)', () => {
cwd: ROOT,
stdout: 'pipe',
stderr: 'pipe',
timeout: 120_000,
});
expect(override.exitCode).toBe(0);
const content = fs.readFileSync(path.join(overrideOut, '.agents', 'skills', 'gstack-ship', 'SKILL.md'), 'utf-8');
@@ -2342,10 +2347,10 @@ describe('Factory generation (--host factory)', () => {
test('--host droid alias works', () => {
const factoryResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], {
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
});
const droidResult = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'droid', '--dry-run', '--out-dir', EXTERNAL_OUT], {
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
});
expect(factoryResult.exitCode).toBe(0);
expect(droidResult.exitCode).toBe(0);
@@ -2354,7 +2359,7 @@ describe('Factory generation (--host factory)', () => {
test('--host factory --dry-run freshness', () => {
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'factory', '--dry-run', '--out-dir', EXTERNAL_OUT], {
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
});
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
@@ -2432,7 +2437,7 @@ describe('Parameterized host smoke tests', () => {
test('--dry-run freshness check passes', () => {
const result = Bun.spawnSync(
['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', hostConfig.name, '--dry-run', '--out-dir', EXTERNAL_OUT],
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe' }
{ cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000 }
);
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
@@ -2457,7 +2462,7 @@ describe('--host all', () => {
// claude host plus every external host regenerate deterministically.
test('--host all generates for all registered hosts', () => {
const result = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'all', '--dry-run', '--out-dir', EXTERNAL_OUT], {
cwd: ROOT, stdout: 'pipe', stderr: 'pipe',
cwd: ROOT, stdout: 'pipe', stderr: 'pipe', timeout: 120_000,
});
expect(result.exitCode).toBe(0);
const output = result.stdout.toString();
test/git-ref-fixture-tripwire.test.ts
+64
View File
@@ -0,0 +1,64 @@
/**
* Git-ref fixture tripwire: no test or helper may pin repo content to a raw
* commit SHA (the `git show <sha>:path` fixture pattern).
*
* The class: test/helpers/auq-sdk-capture.ts defaulted verboseSkill() to
* `git show ab66193e^:plan-ceo-review/SKILL.md` — a BRANCH-LOCAL ref. That
* fixture dies the day the branch is pruned, and already failed on shallow
* clones (CI executors fetch-depth-0 exists precisely because self-derived
* selection crashed on shallow checkouts). The v1.75 precedent is to VENDOR
* the frozen content under test/fixtures/ instead — content-addressed by the
* repo itself, immune to ref pruning and clone depth.
*
* Scans test trees + helpers for two shapes:
* - a quoted `<hex>{7,40}[^]?:` rev-path (the `git show SHA:path` form)
* - a gitRef-style default parameter carrying a raw hex SHA
*/
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
const ROOT = path.resolve(import.meta.dir, '..');
const SCAN_ROOTS = ['test', 'browse/test', 'design/test', 'make-pdf/test'];
const SELF = path.join('test', 'git-ref-fixture-tripwire.test.ts');
// Quoted `SHA:` rev-path (7-40 hex chars, optional ^/~ suffix, then colon) —
// requires >= 2 digits among the hex so ordinary words ('deadbeef' aside)
// and pure-alpha identifiers don't false-positive.
const REV_PATH = /['"`]([0-9a-f]{7,40})[\^~]?:/g;
const GIT_REF_DEFAULT = /gitRef\s*=\s*['"`][0-9a-f]{7,40}/;
const looksLikeSha = (s: string): boolean => /[0-9]/.test(s) && /[a-f]/.test(s);
describe('git-ref fixture tripwire', () => {
test('no raw-SHA fixture refs in the test trees (vendor the content instead)', () => {
const hits: string[] = [];
for (const root of SCAN_ROOTS) {
const abs = path.join(ROOT, root);
if (!fs.existsSync(abs)) continue;
const stack = [abs];
while (stack.length > 0) {
const dir = stack.pop()!;
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
const full = path.join(dir, entry.name);
if (entry.isDirectory()) { stack.push(full); continue; }
if (!/\.(?:[cm]?[jt]s|tsx)$/.test(entry.name)) continue;
const rel = path.relative(ROOT, full);
if (rel === SELF) continue;
const src = fs.readFileSync(full, 'utf-8');
src.split('\n').forEach((line, i) => {
for (const m of line.matchAll(REV_PATH)) {
if (looksLikeSha(m[1])) hits.push(`${rel}:${i + 1} ${line.trim().slice(0, 100)}`);
}
if (GIT_REF_DEFAULT.test(line)) hits.push(`${rel}:${i + 1} ${line.trim().slice(0, 100)}`);
});
}
}
}
expect(
hits,
`raw-SHA fixture reference(s) — these die on branch prune and fail on shallow clones. `
+ `Vendor the frozen content under test/fixtures/ instead (v1.75 precedent):\n ${hits.join('\n ')}`,
).toEqual([]);
});
});
test/global-discover.test.ts
+2 -1
View File
@@ -180,10 +180,11 @@ describe("gstack-global-discover", () => {
// Create a git repo as the session target
const repoDir = join(tmpDir, "fake-repo");
mkdirSync(repoDir);
spawnSync("git", ["init"], { cwd: repoDir, stdio: "pipe" });
spawnSync("git", ["init"], { cwd: repoDir, stdio: "pipe", timeout: 30_000 });
spawnSync("git", ["commit", "--allow-empty", "-m", "init"], {
cwd: repoDir,
stdio: "pipe",
timeout: 30_000,
});
// Write a session with a 20KB first line (simulates Codex v0.117+)
test/gstack-artifacts-init.test.ts
+9 -8
View File
@@ -119,7 +119,7 @@ exit 0
* test focused on artifacts-init's branching logic, not git plumbing.
*/
function makeFakeGit() {
const realGit = spawnSync('which', ['git'], { encoding: 'utf-8' }).stdout.trim();
const realGit = spawnSync('which', ['git'], { encoding: 'utf-8', timeout: 30_000 }).stdout.trim();
const script = `#!/bin/bash
# Walk argv past leading -C <dir> and similar flags to find the real subcommand.
args=("$@")
@@ -157,6 +157,7 @@ function run(argv: string[], opts: { env?: Record<string, string>; input?: strin
encoding: 'utf-8',
input: opts.input,
cwd: ROOT,
timeout: 30_000,
});
return {
stdout: res.stdout || '',
@@ -176,7 +177,7 @@ beforeEach(() => {
fakeBinDir = fs.mkdtempSync(path.join(os.tmpdir(), 'artifacts-fake-bin-'));
ghCallLog = path.join(fakeBinDir, 'gh-calls.log');
glabCallLog = path.join(fakeBinDir, 'glab-calls.log');
spawnSync('git', ['init', '--bare', '-q', '-b', 'main', bareRemote]);
spawnSync('git', ['init', '--bare', '-q', '-b', 'main', bareRemote], { timeout: 30_000 });
makeFakeGit();
});
@@ -277,7 +278,7 @@ describe('gstack-artifacts-init canonical URL storage (codex Finding #10)', () =
makeFakeGh({ webUrl: 'https://github.com/testuser/gstack-artifacts-testuser' });
const r = run(['--host', 'github']);
expect(r.status).toBe(0);
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' });
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 });
expect(remote.stdout.trim()).toBe('https://github.com/testuser/gstack-artifacts-testuser');
});
@@ -288,7 +289,7 @@ describe('gstack-artifacts-init canonical URL storage (codex Finding #10)', () =
});
const r = run(['--host', 'github']);
expect(r.status).toBe(0);
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' });
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 });
expect(remote.stdout.trim()).toBe('git@github.com:testuser/gstack-artifacts-testuser.git');
});
@@ -296,7 +297,7 @@ describe('gstack-artifacts-init canonical URL storage (codex Finding #10)', () =
makeFakeGh({ gitProtocol: 'unset' });
const r = run(['--host', 'github']);
expect(r.status).toBe(0);
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' });
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 });
expect(remote.stdout.trim()).toBe('https://github.com/testuser/gstack-artifacts-testuser');
});
@@ -304,7 +305,7 @@ describe('gstack-artifacts-init canonical URL storage (codex Finding #10)', () =
makeFakeGlab({ gitProtocol: 'ssh' });
const r = run(['--host', 'gitlab']);
expect(r.status).toBe(0);
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' });
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 });
expect(remote.stdout.trim()).toBe('git@gitlab.com:testuser/gstack-artifacts-testuser.git');
});
});
@@ -359,7 +360,7 @@ describe('gstack-artifacts-init idempotency', () => {
makeFakeGh({ gitProtocol: 'ssh' });
const r = run(['--remote', 'https://github.com/testuser/gstack-artifacts-testuser']);
expect(r.status).toBe(0);
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' });
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 });
expect(remote.stdout.trim()).toBe('https://github.com/testuser/gstack-artifacts-testuser');
});
@@ -372,7 +373,7 @@ describe('gstack-artifacts-init idempotency', () => {
'ssh',
]);
expect(r.status).toBe(0);
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8' });
const remote = spawnSync('git', ['-C', tmpHome, 'remote', 'get-url', 'origin'], { encoding: 'utf-8', timeout: 30_000 });
expect(remote.stdout.trim()).toBe('git@github.com:testuser/gstack-artifacts-testuser.git');
});
test/gstack-artifacts-url.test.ts
+1 -1
View File
@@ -13,7 +13,7 @@ const ROOT = path.resolve(import.meta.dir, '..');
const URL_BIN = path.join(ROOT, 'bin', 'gstack-artifacts-url');
function run(args: string[]): { code: number; stdout: string; stderr: string } {
const r = spawnSync(URL_BIN, args, { encoding: 'utf-8' });
const r = spawnSync(URL_BIN, args, { encoding: 'utf-8', timeout: 30_000 });
return {
code: r.status ?? -1,
stdout: (r.stdout || '').trim(),
test/gstack-codex-session-import.test.ts
+2 -2
View File
@@ -68,7 +68,7 @@ function runImport(sessionPath: string): { stdout: string; stderr: string; statu
env.GSTACK_STATE_ROOT = stateRoot;
env.GSTACK_QUESTION_LOG_NO_DERIVE = '1';
delete env.GSTACK_HOME;
const res = spawnSync(BIN, [sessionPath], { env, encoding: 'utf-8', cwd: ROOT });
const res = spawnSync(BIN, [sessionPath], { env, encoding: 'utf-8', cwd: ROOT, timeout: 30_000 });
return {
stdout: res.stdout ?? '',
stderr: res.stderr ?? '',
@@ -196,7 +196,7 @@ describe('default mode (no args → latest)', () => {
}
env.GSTACK_STATE_ROOT = stateRoot;
env.CODEX_SESSIONS_ROOT = emptyDir;
const res = spawnSync(BIN, [], { env, encoding: 'utf-8', cwd: ROOT });
const res = spawnSync(BIN, [], { env, encoding: 'utf-8', cwd: ROOT, timeout: 30_000 });
expect(res.status).toBe(0);
expect(res.stdout).toMatch(/NO_SESSIONS/);
} finally {
test/gstack-config-defaults.test.ts
+1
View File
@@ -45,6 +45,7 @@ const STATE = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-config-test-'));
function get(key: string): { out: string; code: number } {
const r = spawnSync('bash', [CONFIG_BIN, 'get', key], {
encoding: 'utf-8',
timeout: 30_000,
env: { ...process.env, GSTACK_STATE_ROOT: STATE },
});
return { out: r.stdout ?? '', code: r.status ?? -1 };
test/gstack-config-key-locale.test.ts
+1
View File
@@ -28,6 +28,7 @@ function run(args: string[]) {
// live tree (observed in the free-tests CI job). Relink behavior itself is
// covered in isolation by test/relink.test.ts's mock install.
env: { ...process.env, GSTACK_STATE_ROOT: stateRoot, GSTACK_SETUP_RUNNING: "1" },
timeout: 30_000,
});
return {
test/gstack-config-redact-keys.test.ts
+1
View File
@@ -16,6 +16,7 @@ function cfg(args: string[]): { code: number; out: string; err: string } {
const r = spawnSync(CONFIG, args, {
encoding: "utf8",
env: { ...process.env, GSTACK_HOME: home },
timeout: 30_000,
});
return { code: r.status ?? 0, out: r.stdout ?? "", err: r.stderr ?? "" };
}
test/gstack-decision-bins.test.ts
+3 -3
View File
@@ -20,18 +20,18 @@ function opts(): ExecSyncOptionsWithStringEncoding {
}
function log(arg: string, expectFail = false): { out: string; code: number } {
try {
return { out: execSync(`${LOG} '${arg.replace(/'/g, "'\\''")}'`, opts()).trim(), code: 0 };
return { out: execSync(`${LOG} '${arg.replace(/'/g, "'\\''")}'`, opts()).trim(), code: 0 }; // timeout via opts()
} catch (e: any) {
if (expectFail) return { out: (e.stderr?.toString() || "").trim(), code: e.status || 1 };
throw e;
}
}
function logFlag(flag: string): string {
return execSync(`${LOG} ${flag}`, opts()).trim();
return execSync(`${LOG} ${flag}`, opts()).trim(); // timeout via opts()
}
function search(args = ""): string {
try {
return execSync(`${SEARCH} ${args}`, opts()).trim();
return execSync(`${SEARCH} ${args}`, opts()).trim(); // timeout via opts()
} catch {
return "";
}
test/gstack-detach.test.ts
+56 -3
View File
@@ -15,13 +15,13 @@ const ROOT = path.resolve(import.meta.dir, '..');
const DETACH = path.join(ROOT, 'bin', 'gstack-detach');
function ownPgid(): string {
return (spawnSync('ps', ['-o', 'pgid=', '-p', String(process.pid)], { encoding: 'utf-8' }).stdout || '').trim();
return (spawnSync('ps', ['-o', 'pgid=', '-p', String(process.pid)], { encoding: 'utf-8', timeout: 30_000 }).stdout || '').trim();
}
function waitFor(pred: () => boolean, ms: number): boolean {
const end = Date.now() + ms;
while (Date.now() < end) {
if (pred()) return true;
spawnSync('sleep', ['0.2']);
spawnSync('sleep', ['0.2'], { timeout: 30_000 });
}
return pred();
}
@@ -69,6 +69,59 @@ describe('gstack-detach', () => {
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
}, 16000);
test('watchdog group-SIGKILLs TERM-immune grandchildren (no orphan survives)', () => {
// Regression pin for the 2026-08 escalation change: the watchdog used to
// follow its killpg(SIGTERM) + 5s grace with a DIRECT proc.kill() — a
// grandchild that ignores TERM survived and burned cores/API for hours
// (the observed 15-hour-orphan class). Now the grace escalates to
// killpg(SIGKILL). The child here traps TERM and spawns a TERM-immune
// grandchild; only a GROUP SIGKILL clears both. Markers are per-run
// unique (pid) so concurrent worktree suites can't cross-kill.
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gd-'));
const log = path.join(dir, 'run.log');
const g1 = `6091.${process.pid}`;
const g2 = `6092.${process.pid}`;
const alive = (m: string) => spawnSync('pgrep', ['-f', `sleep ${m.replace('.', '\\.')}`], { stdio: 'pipe', timeout: 5_000 }).status === 0;
try {
spawnSync(DETACH, ['--log', log, '--timeout', '1', '--', 'bash', '-c',
`trap '' TERM; (trap '' TERM; sleep ${g1}) & exec sleep ${g2}`],
{ encoding: 'utf-8', timeout: 10000 });
expect(waitFor(() => logHas(log, '### gstack-detach EXIT=timeout ###'), 15000)).toBe(true);
// Grace is 5s after the TERM that both processes ignore — the SIGKILL
// escalation must clear the whole group shortly after the sentinel.
expect(waitFor(() => !alive(g1) && !alive(g2), 10000),
'TERM-immune child/grandchild survived the watchdog — killpg(SIGKILL) escalation regressed').toBe(true);
} finally {
spawnSync('pkill', ['-9', '-f', `sleep 609[12]\\.${process.pid}`], { stdio: 'ignore', timeout: 5_000 });
fs.rmSync(dir, { recursive: true, force: true });
}
}, 30000);
test('watchdog kills the grandchild even when the LEADER dies on the SIGTERM', () => {
// The pgid-after-grace bug: killpg(getpgid(proc.pid), SIGKILL) raised
// ESRCH once the leader had honored the TERM, and the except fell back
// to proc.kill() on a corpse — the TERM-immune grandchild lived forever.
// The fix captures the pgid AT SPAWN. This variant is the one the
// TERM-immune-leader test above cannot see.
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gd-'));
const log = path.join(dir, 'run.log');
const g = `6093.${process.pid}`;
const alive = () => spawnSync('pgrep', ['-f', `sleep ${g.replace('.', '\\.')}`], { stdio: 'pipe', timeout: 5_000 }).status === 0;
try {
// Leader: no trap — dies on the watchdog's SIGTERM. Grandchild:
// TERM-immune, same group — only a saved-pgid SIGKILL reaches it.
spawnSync(DETACH, ['--log', log, '--timeout', '1', '--', 'bash', '-c',
`(trap '' TERM; sleep ${g}) & sleep 60`],
{ encoding: 'utf-8', timeout: 10000 });
expect(waitFor(() => logHas(log, '### gstack-detach EXIT=timeout ###'), 15000)).toBe(true);
expect(waitFor(() => !alive(), 10000),
'grandchild survived a dead leader — the pgid must be captured at spawn, not resolved after the grace').toBe(true);
} finally {
spawnSync('pkill', ['-9', '-f', `sleep 6093\\.${process.pid}`], { stdio: 'ignore', timeout: 5_000 });
fs.rmSync(dir, { recursive: true, force: true });
}
}, 30000);
test('machine --lock serializes concurrent runs (second WAITS for the first)', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gd-'));
const lock = `gstack-detach-test-${process.pid}`;
@@ -90,7 +143,7 @@ describe('gstack-detach', () => {
}, 20000);
test('rejects missing command (exit 2)', () => {
const r = spawnSync(DETACH, ['--label', 'x'], { encoding: 'utf-8' });
const r = spawnSync(DETACH, ['--label', 'x'], { encoding: 'utf-8', timeout: 30_000 });
expect(r.status).toBe(2);
});
});
test/gstack-developer-profile.test.ts
+2
View File
@@ -36,6 +36,7 @@ function runDev(...args: string[]): { stdout: string; stderr: string; status: nu
env: { ...process.env, GSTACK_HOME: tmpHome },
encoding: 'utf-8',
cwd: ROOT,
timeout: 30_000,
});
return {
stdout: res.stdout ?? '',
@@ -49,6 +50,7 @@ function logQuestion(payload: Record<string, unknown>): number {
env: { ...process.env, GSTACK_HOME: tmpHome },
encoding: 'utf-8',
cwd: ROOT,
timeout: 30_000,
});
return res.status ?? -1;
}
test/gstack-egress-cli.test.ts
+3
View File
@@ -36,6 +36,7 @@ function run(args: string[]) {
const result = spawnSync(BIN, args, {
encoding: 'utf-8',
env: { ...process.env, GSTACK_HOME: home },
timeout: 30_000,
});
return { code: result.status ?? -1, stdout: result.stdout || '', stderr: result.stderr || '' };
}
@@ -122,11 +123,13 @@ describe('gstack-egress grants', () => {
const config = spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'telemetry', 'community'], {
encoding: 'utf-8',
env: { ...process.env, GSTACK_HOME: home },
timeout: 30_000,
});
expect(config.status).toBe(0);
spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'artifacts_sync_mode', 'full'], {
encoding: 'utf-8',
env: { ...process.env, GSTACK_HOME: home },
timeout: 30_000,
});
const r = run(['grants', '--json']);
expect(r.code).toBe(0);

Some files were not shown because too many files have changed in this diff Show More