Files
gstack/ship/sections/tests.md.tmpl
T
Garry Tanandcapy-ai[bot] 79f6012bbc v1.99.5.0 Give the head a pull request merges a full review when its removed lines changed or it was only re-checked (GST-2706) (#4365)
* Count review rounds per pull request across /ship runs (GST-2673)

/ship's two-round review cap was counted inside each run, so a pull request
shipped many times was reviewed again at every head. Step 1 now reads the
rounds the PR body's receipt history records, passes them to the review plan,
and sends later heads to the scoped re-check, with a lane or Captain round
grant for an extra round. Adds the no-loss replay preregistration.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Cap full review rounds per pull request and keep fix-checks to each run

The plan counts a PR's earlier full rounds (--prior-full) against the cap of
two, as the approved plan says, and prints every earlier round as pr_rounds=.
Renames the troubleshooting anchor to pr-round-cap, lists <SESSION_ID> as a
reviewed placeholder, and leaves ship/DRIVER.md at its size cap.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Point the ship driver's review pass at Step 9's plan flags, which carry the PR round count

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Give a pull request's final head a full review and make the diff-only re-check advice only (GST-2673)

A scoped re-check that reads only what changed since the last reviewed head
caught 22 of 24 planted faults but none of 8 real faults that full rounds had
caught, so it no longer carries a /ship receipt. A later head whose own lines
changed now gets a fresh full review, the PR's last before merge; a PR already
at two full rounds gets one read-only full review with no fix round, whose
blocker goes to the lane's Captain. carry prints RECHECK: advisory and refuses
--recheck. The drift threshold, the re-check blocker count and the
design-change grant kind are removed with the route they fed.

Repins lib/review-depth.ts and ship/sections/review-army.md under replay
GST-2673 (docs/evals/pr-round-count-results.md; deterministic grid against
main: 196,608 of 196,608 plans identical with no prior rounds).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Keep a review round grant usable through its fix round and after a read-only review

A grant now holds after the fix round's commits move HEAD past it, counts only the rounds earlier heads spent, and a receipt written without a run keeps other runs' stubs.

Review-Finding: p1/specialist-batched#1 reader=state-machines
Review-Finding: p1/red-team#2 reader=red-team
Review-Finding: p1/red-team#3 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Count every review round at a granted head again, so a grant can't reset the count

Reverts the history change that skipped rounds recorded at a round grant's head: the run that hit the cap is recorded at that same head, so skipping it wiped the count. A grant recorded after a read-only review still doesn't buy a fix round (fails closed).

Review-Finding: p2/adversarial-native#1 reader=adversarial-native
Review-Finding: p2/outside-voice#1 reader=outside-voice
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Give a merge head a full review when its removed lines changed or its receipt came from a re-check

A head whose removed lines changed with the base no longer carries on one scoped reader, a receipt an older gstack carried through a scoped re-check no longer carries on or lands, and a /ship body post keeps the run history. The read-only stop stops offering a round grant it can't honor, and the stamp-only-carry no-stop class is removed because a refused carry never stopped. (GST-2706)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Keep the run history inside Agent notes when the PR text quotes a receipt

Review-Finding: p1/red-team#1 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Let a receipt write skip a retired no-stop entry left in an older branch ledger

Review-Finding: p3/outside-voice#1 reader=outside-voice
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Say in troubleshooting that the queue refuses a receipt carried through a re-check

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-17 12:01:00 +00:00

175 lines
10 KiB
Cheetah

## Step 4: Test Framework Bootstrap
{{TEST_BOOTSTRAP}}
---
## Step 5: Run tests (on merged code)
**Run it:** `~/.claude/skills/gstack/bin/gstack-ship-preflight --tests-only --base <base> --lane <label>=<command>...`
(or the Step 1 call with lanes). It runs the free pre-gate first, before any lane, eval or
remote gate (`gstack-pregate --allow-repo-commands`: stage 1 regen, secrets, strays, literals;
stage 2 the selected lanes under pipefail), then the lanes inside the evidence ledger, and prints `PREGATE:`, `LANES:` and
one `LANE: <label> <state> exit= tests_ran= log=` line each.
`fail` or `incomplete` blocks: fix, rerun. **No-stop** (docs/review-policy.md#no-stop): a pregate flake rerun (one per branch), a
timeout-only fix and two other routine stops go through `gstack-no-stop record` (`--help`); `recorded` goes on,
`refused` stops. `requires-remote` (other platforms; the remote gate under 8 cores) needs
`gstack-pregate clear` or, under a user cap, `--remote --spend-cap <usd>`. Rerun
after the final stamp (`gstack-ship-release finish` does); Step 18 refuses a stale one.
Use the project's test commands discovered in Step 4 or documented in CLAUDE.md/AGENTS.md. Run every applicable suite; do not assume Rails or Vitest. Pass each as `--lane <label>=<command>`; use the same lane labels and exact commands again in Step 16.
<a id="ci-carry"></a>
**CI is the one full test run.** When `origin/<base>`'s `.gstack/pregate.json` maps a lane's exact
command in `ship_lanes_from` (gstack maps `bun run test`), that lane is carried from a verified CI run
at this exact, clean tree instead of running here: `LANE: <label> carried run=<id>/<attempt> tests_ran=<n>`.
Before the first push that lane runs only targeted tests and the repro (the full suite too when the diff touches
merge-check code: `lib/ship-*`, `lib/pregate/*`, the ship and pregate bins,
`scripts/lib/shard-engine.ts`). Push early so CI runs beside the review: commit, `gstack-ship-release push
--branch <branch> --early` (sealed, secrets and guards first; no consent: Step 6.5 asks only for Greptile), open a draft PR
(`gstack-post pr-create --base <base> --draft --title-file <f> --body-file <f>`), then `gstack-ci-carry wait` in
the background; once it prints `CI_WAIT: green`, rerun `--tests-only` and the mapped lane is carried. A red
run is a failed lane: triage it below. Batch review fixes into one commit and one push. `CI_CARRY: none
label=<l> (<reason>)` names why a mapped lane ran here. A docs-only change ([docs/pregate.md#docs-only](../../docs/pregate.md#docs-only)) owes no test lane: pregate runs
the docs lane, and a mapped lane prints `LANE: <label> not owed: docs-only change …` from CI's skipped run. Any other lane that already passed at this
tree, command and runtime is reused (`reused=<time>`). In Step 16, `finish` defers a mapped
lane it cannot carry (`EVIDENCE: DEFERRED`); after Step 17's push, `gstack-ship-release accept --base <base>
--lane …` carries it from the green run at the final head and reruns the pregate there.
**If no applicable test suite exists:** Name the untested scope. AskUserQuestion:
A) Add tests (recommended), B) Ship with this named testing
gap, or C) Stop. Reuse an actual prior B answer only for the same scope and
content; declining bootstrap alone is not that approval. B continues with the
gap recorded, not passing tests. Independent build, eval, review and QA gates
still apply. A declared but unavailable suite is a blocker, not an absent suite.
A runs Step 4 with this new bootstrap choice, then returns here to run the tests.
C stops this attempt.
**For Rails projects using `bin/test-lane`, do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
`db:test:prepare` internally, which loads the schema into the correct lane database.
Running bare test migrations without INSTANCE hits an orphan DB and corrupts structure.sql.
Each lane runs as `gstack-evidence run --label <label> -- '<command>'`: transparent (exit code passes
through) and recorded as `{command, exit, working-tree fingerprint, log path}` in
`$GSTACK_STATE_ROOT/projects/<slug>/<branch>-evidence.jsonl`, which Step 16 cites instead of
re-running when the content hasn't changed. Each lane has its own log (no shared /tmp
collisions between concurrent ships); read it for failure detail.
**A lane passes on its runner summary line, not on exit 0 alone.** Read each
lane's `tests_ran` from its `LANE:` line (or from the runner's own summary in the log)
and keep the three states apart; never merge them:
1. `tests_ran: N` (N > 0) with exit 0 — the lane passed; record N for the PR body.
2. `tests_ran: 0` — **ZERO-RUN**: bun `0 pass 0 fail`, jest "No tests found", pytest
"no tests ran", rspec "0 examples", `go test` with only `[no test files]`, or an
empty selection after a filter. Exit 0 here is not a pass, so triage a ZERO-RUN
lane below as a failed lane: a bad selector, a missing file or a skipped `describe`
is an in-branch failure until proven otherwise.
3. `tests_ran: unknown` — the runner's summary was not recognised. Keep today's
exit-0 pass and record `count unavailable` for that lane in the PR body.
**If any test fails:** Do NOT immediately stop. When a free-suite shard failed,
first Read `~/.claude/skills/gstack/ship/sections/measure.md` and rerun that
shard's file list as it says (diagnostic, not a verdict). Then apply the Test
Failure Ownership Triage:
{{TEST_FAILURE_TRIAGE}}
**Red base (before T3).** When T1 calls every failure likely pre-existing,
prove it before asking: write `.gstack/tmp/base-red.json` with your file-write tool
(`{"lane":"<label>","total":<failures the lane reported>,"failures":[{"id":"<test
name>","file":"<test file>","signature":"<first error line>"}]}`), then run
```bash
~/.claude/skills/gstack/bin/gstack-base-red classify --failures .gstack/tmp/base-red.json --base <base> --pause
```
It reads `base_red` from the base branch's ship policy and reruns those files at
the base head and at HEAD in scratch worktrees. Exit 0 with `BASE_RED_VERDICT:
base-red` and `PAUSE: <file>`: skip T3, record the pause file and its cases in
the invocation record, and continue to Step 6; Step 16 treats exactly these cases
as this run's waived failures (any other failure is new), and the receipt write
adds `--base-red <pause file>`, so Step 22 ends `SHIP_PAUSED` with
`status=incomplete` instead of READY. Exit 1 (`IN_BRANCH:` lines,
`BASE_RED_UNCONFIGURED`, `BASE_RED_UNNAMED`, `BASE_RED_REPRO_ERROR`): continue T3
as written.
**Resume a red-base pause.** A parent waits with `gstack-base-red wait --pause
<file>` (a background operation; `BASE_GREEN:` once a new base head passes the
cases), then runs `/ship --resume <record>`: if HEAD is still the paused head,
rerun this Step 5 on HEAD merged with the new base in a scratch worktree (never a
pushed base merge); all green rewrites the receipt without `--base-red` and runs
Step 22. Any failure is triaged here as new; a moved HEAD starts /ship fresh.
**After triage:** If any in-branch failures remain unfixed, **STOP**. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, skipped, or base-red and paused), continue to Step 6.
**If all pass:** Report each lane's pass count (or `count unavailable`) in one line and continue to Step 6.
---
## Step 6: Eval Suites (conditional)
Evals are mandatory when prompt-related files change. Select from the full diff,
including uncommitted changes, before deciding whether to skip.
**1. Select affected suites using the project's contract.**
**Project-native path:** Read CLAUDE.md/AGENTS.md, package scripts and the eval
dependency map. Include changed prompts, skill templates, judges and harness
code. Use the documented selector and pre-merge command. If it reports no
affected suites, record that result and continue to Step 7. If prompt-related
files changed but selection or the command is unknown, report the validation
gap and ask before shipping. A missing Rails-pattern match is not a skip signal
for another stack.
**Rails example only — when this repository provides `bin/test-lane` and
`test/evals/*_eval_runner.rb`:**
- Match the diff against the project's documented prompt paths, such as
`app/services/*_prompt_builder.rb`, generation/writer/designer services,
evaluator/scorer/classifier/analyzer services, voice/writing/prompt/token
concerns, chat tools, `config/system_prompts/*.txt` and `test/evals/**/*`.
- Match changed files to each runner's `PROMPT_SOURCE_FILES`; follow shared
judge/support/fixture imports to all affected suites. A runner such as
`post_generation_eval_runner.rb` maps to `post_generation_eval_test.rb`.
- Use the project's full pre-merge tier (`EVAL_JUDGE_TIER=full` for this runner).
Do not substitute a cheaper development tier. If selection remains uncertain,
include every plausibly affected suite.
**2. Run the selected command and preserve its exit status.**
For the Rails example:
```bash
set -o pipefail
EVAL_JUDGE_TIER=full EVAL_VERBOSE=1 bin/test-lane --eval test/evals/<suite>_eval_test.rb 2>&1 | tee /tmp/ship_evals.txt
```
Use the native command for other stacks. Respect the project's concurrency and
retry policy. Rails suites sharing a test lane run sequentially; stop on the
first failure before starting another paid suite.
**Long eval suites (30+ min): launch detached so a turn boundary can't kill them.**
Use the detached runner and eval lock; set its outer timeout to cover the
project's declared suite duration and retries. Do not change individual eval
limits. For a suite whose full bound fits 5400 seconds:
```bash
~/.claude/skills/gstack/bin/gstack-detach --label ship-evals --lock gstack-evals --timeout 5400 -- <project eval command>
```
Poll the printed log for `### gstack-detach EXIT=<code> ###`. Silence is not
success. Retain every configured attempt; skipped or unstarted cases do not
satisfy coverage.
**3. Check results and save evidence for Step 19.**
- **If any eval fails:** Do not rerun the full gate. Show failures and available
costs, then Read `~/.claude/skills/gstack/ship/sections/measure.md` and run its
loop for each red case: classify, measure alone, fix at the cause, re-measure,
then gate once. **STOP** only at its named-red stop.
- **If all selected evals pass:** Record actual counts, any reused evidence and
its source, and available costs. Continue to Step 7.
---