gstack/ship/sections/tests.md.tmpl

## Step 4: Test Framework Bootstrap

{{TEST_BOOTSTRAP}}

---

## Step 5: Run tests (on merged code)

Use the project's test commands discovered in Step 4 or documented in CLAUDE.md/AGENTS.md. Run every applicable suite; do not assume Rails or Vitest. The commands below are examples only for repositories that actually provide them. Use the same lane labels and exact commands again in Step 16.

**For Rails projects using `bin/test-lane`, do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
`db:test:prepare` internally, which loads the schema into the correct lane database.
Running bare test migrations without INSTANCE hits an orphan DB and corrupts structure.sql.

Run independent test suites in parallel, each wrapped in the evidence ledger. The
wrapper is transparent (streams output live, exit code passes through) and
records `{command, exit, working-tree fingerprint, log path}` to
`~/.gstack/projects/<slug>/<branch>-evidence.jsonl` — Step 16 cites this
record instead of re-running when the content hasn't changed:

```bash
~/.claude/skills/gstack/bin/gstack-evidence run --label tests -- 'bin/test-lane 2>&1' &
~/.claude/skills/gstack/bin/gstack-evidence run --label vitest -- 'npm run test 2>&1' &
wait
```

After all suites complete, check the `gstack-evidence: recorded label=... exit=...
log=...` summary lines — each carries the lane's exit code and a per-run log
file (no shared /tmp collisions between concurrent ships). Read the log files
for failure detail.

**If any test fails:** Do NOT immediately stop. Apply the Test Failure Ownership Triage:

{{TEST_FAILURE_TRIAGE}}

**After triage:** If any in-branch failures remain unfixed, **STOP**. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, or skipped), continue to Step 6.

**If all pass:** Continue silently — just note the counts briefly.

---

## Step 6: Eval Suites (conditional)

Evals are mandatory when prompt-related files change. Skip this step entirely if no prompt files are in the diff.

Use the project's documented eval selection and pre-merge command first (including changed skill templates and judge/harness code). The Rails patterns and commands below apply only when that runner exists. For other stacks, use their native eval scripts and dependency map. If prompts changed but no eval command is documented, report the missing validation and ask before shipping; never silently treat that as no affected prompts.

**1. Check if the diff touches prompt-related files:**

```bash
git diff origin/<base> --name-only
```

Match against these patterns (from CLAUDE.md):
- `app/services/*_prompt_builder.rb`
- `app/services/*_generation_service.rb`, `*_writer_service.rb`, `*_designer_service.rb`
- `app/services/*_evaluator.rb`, `*_scorer.rb`, `*_classifier_service.rb`, `*_analyzer.rb`
- `app/services/concerns/*voice*.rb`, `*writing*.rb`, `*prompt*.rb`, `*token*.rb`
- `app/services/chat_tools/*.rb`, `app/services/x_thread_tools/*.rb`
- `config/system_prompts/*.txt`
- `test/evals/**/*` (eval infrastructure changes affect all suites)

**If no matches:** Print "No prompt-related files changed — skipping evals." and continue to Step 7.

**2. Identify affected eval suites:**

Each eval runner (`test/evals/*_eval_runner.rb`) declares `PROMPT_SOURCE_FILES` listing which source files affect it. Grep these to find which suites match the changed files:

```bash
grep -l "changed_file_basename" test/evals/*_eval_runner.rb
```

Map runner → test file: `post_generation_eval_runner.rb` → `post_generation_eval_test.rb`.

**Special cases:**
- Changes to `test/evals/judges/*.rb`, `test/evals/support/*.rb`, or `test/evals/fixtures/` affect ALL suites that use those judges/support files. Check imports in the eval test files to determine which.
- Changes to `config/system_prompts/*.txt` — grep eval runners for the prompt filename to find affected suites.
- If unsure which suites are affected, run ALL suites that could plausibly be impacted. Over-testing is better than missing a regression.

**3. Run affected suites at `EVAL_JUDGE_TIER=full`:**

`/ship` is a pre-merge gate, so always use full tier (Sonnet structural + Opus persona judges).

```bash
EVAL_JUDGE_TIER=full EVAL_VERBOSE=1 bin/test-lane --eval test/evals/<suite>_eval_test.rb 2>&1 | tee /tmp/ship_evals.txt
```

If multiple suites need to run, run them sequentially (each needs a test lane). If the first suite fails, stop immediately — don't burn API cost on remaining suites.

**Long eval suites (30+ min): launch detached so a turn boundary can't kill them.**
A plain backgrounded eval lives in the harness's process group and dies to a
SIGTERM ("polite quit") on a turn boundary, a stopped monitor, or an interruption
(observed mid-`/ship`: `script terminated by signal SIGTERM`). Run it through
`~/.claude/skills/gstack/bin/gstack-detach` instead — it survives in its own
session, serializes against other worktrees via a machine lock (no API
saturation), and writes a guaranteed `### gstack-detach EXIT=<code> ###` sentinel:

```bash
~/.claude/skills/gstack/bin/gstack-detach --label ship-evals --lock gstack-evals --timeout 5400 -- <project eval command>
```

Then poll the printed log path; break on the `EXIT=` sentinel (covers both pass
and crash — silence is never success). The detached run survives even if your
poller is reaped.

**4. Check results:**

- **If any eval fails:** Show the failures, the cost dashboard, and **STOP**. Do not proceed.
- **If all pass:** Note pass counts and cost. Continue to Step 7.

**5. Save eval output** — include eval results and cost dashboard in the PR body (Step 19).

**Tier reference (for context — /ship always uses `full`):**
| Tier | When | Speed (cached) | Cost |
|------|------|----------------|------|
| `fast` (Haiku) | Dev iteration, smoke tests | ~5s (14x faster) | ~$0.07/run |
| `standard` (Sonnet) | Default dev, `bin/test-lane --eval` | ~17s (4x faster) | ~$0.37/run |
| `full` (Opus persona) | **`/ship` and pre-merge** | ~72s (baseline) | ~$1.27/run |

---