gstack/ship/sections/tests.md.tmpl

## Step 4: Test Framework Bootstrap

{{TEST_BOOTSTRAP}}

---

## Step 5: Run tests (on merged code)

**Run it:** `~/.claude/skills/gstack/bin/gstack-ship-preflight --tests-only --base <base> --lane <label>=<command>...`
(or the Step 1 call with lanes). It runs the free pre-gate first, before any lane, eval or
remote gate (`gstack-pregate --allow-repo-commands`: stage 1 regen, secrets, strays, literals;
stage 2 the selected lanes under pipefail), then the lanes inside the evidence ledger (one at
a time on 4 or fewer cores unless `--lanes serial|parallel`), and prints `PREGATE:`, `LANES:` and
one `LANE: <label> <state> exit= tests_ran= log=` line each.
`fail` or `incomplete` blocks: fix, rerun. `requires-remote` (other platforms; the remote gate under 8 cores) needs
`gstack-pregate clear` or, under a user cap, `--remote --spend-cap <usd>`. Rerun
after the final stamp (`gstack-ship-release finish` does); Step 18 refuses a stale one.

Use the project's test commands discovered in Step 4 or documented in CLAUDE.md/AGENTS.md. Run every applicable suite; do not assume Rails or Vitest. Pass each as `--lane <label>=<command>`; use the same lane labels and exact commands again in Step 16.

<a id="ci-carry"></a>
**CI is the one full test run.** When `origin/<base>`'s `.gstack/pregate.json` maps a lane's exact
command in `ship_lanes_from` (gstack maps `bun run test`), that lane is carried from a verified CI run
at this exact, clean tree instead of running here: `LANE: <label> carried run=<id>/<attempt> tests_ran=<n>`.
Before the first push run only targeted tests and the repro (the full suite too when the diff touches
merge-check code: `lib/ship-*`, `lib/pregate/*`, the ship and pregate bins,
`scripts/lib/shard-engine.ts`). Push early so CI runs beside the review: commit, `gstack-ship-release push
--branch <branch> --early` (sealed and secrets scans first), open a draft PR, then `gstack-ci-carry wait` in
the background; once it prints `CI_WAIT: green`, rerun `--tests-only` and the mapped lane is carried. A red
run is a failed lane: triage it below. Batch review fixes into one commit and one push. `CI_CARRY: none
label=<l> (<reason>)` names why a mapped lane ran here (a dirty tree, no green run at HEAD yet, a fork or stale
run, a newer red run, a tree.json that misses an inventory file). Any other lane that already passed at this
tree, command and runtime is reused (`reused=<time>`), never run twice. In Step 16, `finish` defers a mapped
lane it cannot carry (`EVIDENCE: DEFERRED`); after the push, `gstack-ship-release accept --base <base>
--lane …` carries it from the green run at the final head and reruns the pregate there.

**If no applicable test suite exists:** Name the untested scope. AskUserQuestion:
A) Add tests (recommended), B) Ship with this named testing
gap, or C) Stop. Reuse an actual prior B answer only for the same scope and
content; declining bootstrap alone is not that approval. B continues with the
gap recorded, not passing tests. Independent build, eval, review and QA gates
still apply. A declared but unavailable suite is a blocker, not an absent suite.
A runs Step 4 with this new bootstrap choice, then returns here to run the tests.
C stops this attempt.

**For Rails projects using `bin/test-lane`, do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
`db:test:prepare` internally, which loads the schema into the correct lane database.
Running bare test migrations without INSTANCE hits an orphan DB and corrupts structure.sql.

Each lane runs as `gstack-evidence run --label <label> -- '<command>'`: transparent (exit code passes
through) and recorded as `{command, exit, working-tree fingerprint, log path}` in
`$GSTACK_STATE_ROOT/projects/<slug>/<branch>-evidence.jsonl`, which Step 16 cites instead of
re-running when the content hasn't changed. Each lane has its own log (no shared /tmp
collisions between concurrent ships); read it for failure detail.

**A lane passes on its runner summary line, not on exit 0 alone.** Read each
lane's `tests_ran` from its `LANE:` line (or from the runner's own summary in the log)
and keep the three states apart; never merge them:

1. `tests_ran: N` (N > 0) with exit 0 — the lane passed; record N for the PR body.
2. `tests_ran: 0` — **ZERO-RUN**: bun `0 pass 0 fail`, jest "No tests found", pytest
   "no tests ran", rspec "0 examples", `go test` with only `[no test files]`, or an
   empty selection after a filter. Exit 0 here is not a pass, so triage a ZERO-RUN
   lane below as a failed lane: a bad selector, a missing file or a skipped `describe`
   is an in-branch failure until proven otherwise.
3. `tests_ran: unknown` — the runner's summary was not recognised. Keep today's
   exit-0 pass and record `count unavailable` for that lane in the PR body.

**If any test fails:** Do NOT immediately stop. When a free-suite shard failed,
first Read `~/.claude/skills/gstack/ship/sections/measure.md` and rerun that
shard's file list as it says (diagnostic, not a verdict). Then apply the Test
Failure Ownership Triage:

{{TEST_FAILURE_TRIAGE}}

**Red base (before T3).** When T1 calls every failure likely pre-existing,
prove it before asking: write `.gstack/tmp/base-red.json` with your file-write tool
(`{"lane":"<label>","total":<failures the lane reported>,"failures":[{"id":"<test
name>","file":"<test file>","signature":"<first error line>"}]}`), then run

```bash
~/.claude/skills/gstack/bin/gstack-base-red classify --failures .gstack/tmp/base-red.json --base <base> --pause
```

It reads `base_red` from the base branch's ship policy and reruns those files at
the base head and at HEAD in scratch worktrees. Exit 0 with `BASE_RED_VERDICT:
base-red` and `PAUSE: <file>`: skip T3, record the pause file and its cases in
the invocation record, and continue to Step 6; Step 16 treats exactly these cases
as this run's waived failures (any other failure is new), and the receipt write
adds `--base-red <pause file>`, so Step 22 ends `SHIP_PAUSED` with
`status=incomplete` instead of READY. Exit 1 (`IN_BRANCH:` lines,
`BASE_RED_UNCONFIGURED`, `BASE_RED_UNNAMED`, `BASE_RED_REPRO_ERROR`): continue T3
as written.

**Resume a red-base pause.** A parent waits with `gstack-base-red wait --pause
<file>` (a background operation; `BASE_GREEN:` once a new base head passes the
cases), then runs `/ship --resume <record>`: if HEAD is still the paused head,
rerun this Step 5 on HEAD merged with the new base in a scratch worktree (never a
pushed base merge); all green rewrites the receipt without `--base-red` and runs
Step 22. Any failure is triaged here as new; a moved HEAD starts /ship fresh.

**After triage:** If any in-branch failures remain unfixed, **STOP**. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, skipped, or base-red and paused), continue to Step 6.

**If all pass:** Report each lane's pass count (or `count unavailable`) in one line and continue to Step 6.

---

## Step 6: Eval Suites (conditional)

Evals are mandatory when prompt-related files change. Select from the full diff,
including uncommitted changes, before deciding whether to skip.

**1. Select affected suites using the project's contract.**

**Project-native path:** Read CLAUDE.md/AGENTS.md, package scripts and the eval
dependency map. Include changed prompts, skill templates, judges and harness
code. Use the documented selector and pre-merge command. If it reports no
affected suites, record that result and continue to Step 7. If prompt-related
files changed but selection or the command is unknown, report the validation
gap and ask before shipping. A missing Rails-pattern match is not a skip signal
for another stack.

**Rails example only — when this repository provides `bin/test-lane` and
`test/evals/*_eval_runner.rb`:**

- Match the diff against the project's documented prompt paths, such as
  `app/services/*_prompt_builder.rb`, generation/writer/designer services,
  evaluator/scorer/classifier/analyzer services, voice/writing/prompt/token
  concerns, chat tools, `config/system_prompts/*.txt` and `test/evals/**/*`.
- Match changed files to each runner's `PROMPT_SOURCE_FILES`; follow shared
  judge/support/fixture imports to all affected suites. A runner such as
  `post_generation_eval_runner.rb` maps to `post_generation_eval_test.rb`.
- Use the project's full pre-merge tier (`EVAL_JUDGE_TIER=full` for this runner).
  Do not substitute a cheaper development tier. If selection remains uncertain,
  include every plausibly affected suite.

**2. Run the selected command and preserve its exit status.**

For the Rails example:

```bash
set -o pipefail
EVAL_JUDGE_TIER=full EVAL_VERBOSE=1 bin/test-lane --eval test/evals/<suite>_eval_test.rb 2>&1 | tee /tmp/ship_evals.txt
```

Use the native command for other stacks. Respect the project's concurrency and
retry policy. Rails suites sharing a test lane run sequentially; stop on the
first failure before starting another paid suite.

**Long eval suites (30+ min): launch detached so a turn boundary can't kill them.**
Use the detached runner and eval lock; set its outer timeout to cover the
project's declared suite duration and retries. Do not change individual eval
limits. For a suite whose full bound fits 5400 seconds:

```bash
~/.claude/skills/gstack/bin/gstack-detach --label ship-evals --lock gstack-evals --timeout 5400 -- <project eval command>
```

Poll the printed log for `### gstack-detach EXIT=<code> ###`. Silence is not
success. Retain every configured attempt; skipped or unstarted cases do not
satisfy coverage.

**3. Check results and save evidence for Step 19.**

- **If any eval fails:** Do not rerun the full gate. Show failures and available
  costs, then Read `~/.claude/skills/gstack/ship/sections/measure.md` and run its
  loop for each red case: classify, measure alone, fix at the cause, re-measure,
  then gate once. **STOP** only at its named-red stop.
- **If all selected evals pass:** Record actual counts, any reused evidence and
  its source, and available costs. Continue to Step 7.

---