mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-17 21:54:30 +01:00
* Count review rounds per pull request across /ship runs (GST-2673) /ship's two-round review cap was counted inside each run, so a pull request shipped many times was reviewed again at every head. Step 1 now reads the rounds the PR body's receipt history records, passes them to the review plan, and sends later heads to the scoped re-check, with a lane or Captain round grant for an extra round. Adds the no-loss replay preregistration. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Cap full review rounds per pull request and keep fix-checks to each run The plan counts a PR's earlier full rounds (--prior-full) against the cap of two, as the approved plan says, and prints every earlier round as pr_rounds=. Renames the troubleshooting anchor to pr-round-cap, lists <SESSION_ID> as a reviewed placeholder, and leaves ship/DRIVER.md at its size cap. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Point the ship driver's review pass at Step 9's plan flags, which carry the PR round count Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Give a pull request's final head a full review and make the diff-only re-check advice only (GST-2673) A scoped re-check that reads only what changed since the last reviewed head caught 22 of 24 planted faults but none of 8 real faults that full rounds had caught, so it no longer carries a /ship receipt. A later head whose own lines changed now gets a fresh full review, the PR's last before merge; a PR already at two full rounds gets one read-only full review with no fix round, whose blocker goes to the lane's Captain. carry prints RECHECK: advisory and refuses --recheck. The drift threshold, the re-check blocker count and the design-change grant kind are removed with the route they fed. Repins lib/review-depth.ts and ship/sections/review-army.md under replay GST-2673 (docs/evals/pr-round-count-results.md; deterministic grid against main: 196,608 of 196,608 plans identical with no prior rounds). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Keep a review round grant usable through its fix round and after a read-only review A grant now holds after the fix round's commits move HEAD past it, counts only the rounds earlier heads spent, and a receipt written without a run keeps other runs' stubs. Review-Finding: p1/specialist-batched#1 reader=state-machines Review-Finding: p1/red-team#2 reader=red-team Review-Finding: p1/red-team#3 reader=red-team Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Count every review round at a granted head again, so a grant can't reset the count Reverts the history change that skipped rounds recorded at a round grant's head: the run that hit the cap is recorded at that same head, so skipping it wiped the count. A grant recorded after a read-only review still doesn't buy a fix round (fails closed). Review-Finding: p2/adversarial-native#1 reader=adversarial-native Review-Finding: p2/outside-voice#1 reader=outside-voice Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Give a merge head a full review when its removed lines changed or its receipt came from a re-check A head whose removed lines changed with the base no longer carries on one scoped reader, a receipt an older gstack carried through a scoped re-check no longer carries on or lands, and a /ship body post keeps the run history. The read-only stop stops offering a round grant it can't honor, and the stamp-only-carry no-stop class is removed because a refused carry never stopped. (GST-2706) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Keep the run history inside Agent notes when the PR text quotes a receipt Review-Finding: p1/red-team#1 reader=red-team Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Let a receipt write skip a retired no-stop entry left in an older branch ledger Review-Finding: p3/outside-voice#1 reader=outside-voice Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Say in troubleshooting that the queue refuses a receipt carried through a re-check Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
175 lines
10 KiB
Cheetah
175 lines
10 KiB
Cheetah
## Step 4: Test Framework Bootstrap
|
|
|
|
{{TEST_BOOTSTRAP}}
|
|
|
|
---
|
|
|
|
## Step 5: Run tests (on merged code)
|
|
|
|
**Run it:** `~/.claude/skills/gstack/bin/gstack-ship-preflight --tests-only --base <base> --lane <label>=<command>...`
|
|
(or the Step 1 call with lanes). It runs the free pre-gate first, before any lane, eval or
|
|
remote gate (`gstack-pregate --allow-repo-commands`: stage 1 regen, secrets, strays, literals;
|
|
stage 2 the selected lanes under pipefail), then the lanes inside the evidence ledger, and prints `PREGATE:`, `LANES:` and
|
|
one `LANE: <label> <state> exit= tests_ran= log=` line each.
|
|
`fail` or `incomplete` blocks: fix, rerun. **No-stop** (docs/review-policy.md#no-stop): a pregate flake rerun (one per branch), a
|
|
timeout-only fix and two other routine stops go through `gstack-no-stop record` (`--help`); `recorded` goes on,
|
|
`refused` stops. `requires-remote` (other platforms; the remote gate under 8 cores) needs
|
|
`gstack-pregate clear` or, under a user cap, `--remote --spend-cap <usd>`. Rerun
|
|
after the final stamp (`gstack-ship-release finish` does); Step 18 refuses a stale one.
|
|
|
|
Use the project's test commands discovered in Step 4 or documented in CLAUDE.md/AGENTS.md. Run every applicable suite; do not assume Rails or Vitest. Pass each as `--lane <label>=<command>`; use the same lane labels and exact commands again in Step 16.
|
|
|
|
<a id="ci-carry"></a>
|
|
**CI is the one full test run.** When `origin/<base>`'s `.gstack/pregate.json` maps a lane's exact
|
|
command in `ship_lanes_from` (gstack maps `bun run test`), that lane is carried from a verified CI run
|
|
at this exact, clean tree instead of running here: `LANE: <label> carried run=<id>/<attempt> tests_ran=<n>`.
|
|
Before the first push that lane runs only targeted tests and the repro (the full suite too when the diff touches
|
|
merge-check code: `lib/ship-*`, `lib/pregate/*`, the ship and pregate bins,
|
|
`scripts/lib/shard-engine.ts`). Push early so CI runs beside the review: commit, `gstack-ship-release push
|
|
--branch <branch> --early` (sealed, secrets and guards first; no consent: Step 6.5 asks only for Greptile), open a draft PR
|
|
(`gstack-post pr-create --base <base> --draft --title-file <f> --body-file <f>`), then `gstack-ci-carry wait` in
|
|
the background; once it prints `CI_WAIT: green`, rerun `--tests-only` and the mapped lane is carried. A red
|
|
run is a failed lane: triage it below. Batch review fixes into one commit and one push. `CI_CARRY: none
|
|
label=<l> (<reason>)` names why a mapped lane ran here. A docs-only change ([docs/pregate.md#docs-only](../../docs/pregate.md#docs-only)) owes no test lane: pregate runs
|
|
the docs lane, and a mapped lane prints `LANE: <label> not owed: docs-only change …` from CI's skipped run. Any other lane that already passed at this
|
|
tree, command and runtime is reused (`reused=<time>`). In Step 16, `finish` defers a mapped
|
|
lane it cannot carry (`EVIDENCE: DEFERRED`); after Step 17's push, `gstack-ship-release accept --base <base>
|
|
--lane …` carries it from the green run at the final head and reruns the pregate there.
|
|
|
|
**If no applicable test suite exists:** Name the untested scope. AskUserQuestion:
|
|
A) Add tests (recommended), B) Ship with this named testing
|
|
gap, or C) Stop. Reuse an actual prior B answer only for the same scope and
|
|
content; declining bootstrap alone is not that approval. B continues with the
|
|
gap recorded, not passing tests. Independent build, eval, review and QA gates
|
|
still apply. A declared but unavailable suite is a blocker, not an absent suite.
|
|
A runs Step 4 with this new bootstrap choice, then returns here to run the tests.
|
|
C stops this attempt.
|
|
|
|
**For Rails projects using `bin/test-lane`, do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
|
|
`db:test:prepare` internally, which loads the schema into the correct lane database.
|
|
Running bare test migrations without INSTANCE hits an orphan DB and corrupts structure.sql.
|
|
|
|
Each lane runs as `gstack-evidence run --label <label> -- '<command>'`: transparent (exit code passes
|
|
through) and recorded as `{command, exit, working-tree fingerprint, log path}` in
|
|
`$GSTACK_STATE_ROOT/projects/<slug>/<branch>-evidence.jsonl`, which Step 16 cites instead of
|
|
re-running when the content hasn't changed. Each lane has its own log (no shared /tmp
|
|
collisions between concurrent ships); read it for failure detail.
|
|
|
|
**A lane passes on its runner summary line, not on exit 0 alone.** Read each
|
|
lane's `tests_ran` from its `LANE:` line (or from the runner's own summary in the log)
|
|
and keep the three states apart; never merge them:
|
|
|
|
1. `tests_ran: N` (N > 0) with exit 0 — the lane passed; record N for the PR body.
|
|
2. `tests_ran: 0` — **ZERO-RUN**: bun `0 pass 0 fail`, jest "No tests found", pytest
|
|
"no tests ran", rspec "0 examples", `go test` with only `[no test files]`, or an
|
|
empty selection after a filter. Exit 0 here is not a pass, so triage a ZERO-RUN
|
|
lane below as a failed lane: a bad selector, a missing file or a skipped `describe`
|
|
is an in-branch failure until proven otherwise.
|
|
3. `tests_ran: unknown` — the runner's summary was not recognised. Keep today's
|
|
exit-0 pass and record `count unavailable` for that lane in the PR body.
|
|
|
|
**If any test fails:** Do NOT immediately stop. When a free-suite shard failed,
|
|
first Read `~/.claude/skills/gstack/ship/sections/measure.md` and rerun that
|
|
shard's file list as it says (diagnostic, not a verdict). Then apply the Test
|
|
Failure Ownership Triage:
|
|
|
|
{{TEST_FAILURE_TRIAGE}}
|
|
|
|
**Red base (before T3).** When T1 calls every failure likely pre-existing,
|
|
prove it before asking: write `.gstack/tmp/base-red.json` with your file-write tool
|
|
(`{"lane":"<label>","total":<failures the lane reported>,"failures":[{"id":"<test
|
|
name>","file":"<test file>","signature":"<first error line>"}]}`), then run
|
|
|
|
```bash
|
|
~/.claude/skills/gstack/bin/gstack-base-red classify --failures .gstack/tmp/base-red.json --base <base> --pause
|
|
```
|
|
|
|
It reads `base_red` from the base branch's ship policy and reruns those files at
|
|
the base head and at HEAD in scratch worktrees. Exit 0 with `BASE_RED_VERDICT:
|
|
base-red` and `PAUSE: <file>`: skip T3, record the pause file and its cases in
|
|
the invocation record, and continue to Step 6; Step 16 treats exactly these cases
|
|
as this run's waived failures (any other failure is new), and the receipt write
|
|
adds `--base-red <pause file>`, so Step 22 ends `SHIP_PAUSED` with
|
|
`status=incomplete` instead of READY. Exit 1 (`IN_BRANCH:` lines,
|
|
`BASE_RED_UNCONFIGURED`, `BASE_RED_UNNAMED`, `BASE_RED_REPRO_ERROR`): continue T3
|
|
as written.
|
|
|
|
**Resume a red-base pause.** A parent waits with `gstack-base-red wait --pause
|
|
<file>` (a background operation; `BASE_GREEN:` once a new base head passes the
|
|
cases), then runs `/ship --resume <record>`: if HEAD is still the paused head,
|
|
rerun this Step 5 on HEAD merged with the new base in a scratch worktree (never a
|
|
pushed base merge); all green rewrites the receipt without `--base-red` and runs
|
|
Step 22. Any failure is triaged here as new; a moved HEAD starts /ship fresh.
|
|
|
|
**After triage:** If any in-branch failures remain unfixed, **STOP**. Do not proceed. If all failures were pre-existing and handled (fixed, TODOed, assigned, skipped, or base-red and paused), continue to Step 6.
|
|
|
|
**If all pass:** Report each lane's pass count (or `count unavailable`) in one line and continue to Step 6.
|
|
|
|
---
|
|
|
|
## Step 6: Eval Suites (conditional)
|
|
|
|
Evals are mandatory when prompt-related files change. Select from the full diff,
|
|
including uncommitted changes, before deciding whether to skip.
|
|
|
|
**1. Select affected suites using the project's contract.**
|
|
|
|
**Project-native path:** Read CLAUDE.md/AGENTS.md, package scripts and the eval
|
|
dependency map. Include changed prompts, skill templates, judges and harness
|
|
code. Use the documented selector and pre-merge command. If it reports no
|
|
affected suites, record that result and continue to Step 7. If prompt-related
|
|
files changed but selection or the command is unknown, report the validation
|
|
gap and ask before shipping. A missing Rails-pattern match is not a skip signal
|
|
for another stack.
|
|
|
|
**Rails example only — when this repository provides `bin/test-lane` and
|
|
`test/evals/*_eval_runner.rb`:**
|
|
|
|
- Match the diff against the project's documented prompt paths, such as
|
|
`app/services/*_prompt_builder.rb`, generation/writer/designer services,
|
|
evaluator/scorer/classifier/analyzer services, voice/writing/prompt/token
|
|
concerns, chat tools, `config/system_prompts/*.txt` and `test/evals/**/*`.
|
|
- Match changed files to each runner's `PROMPT_SOURCE_FILES`; follow shared
|
|
judge/support/fixture imports to all affected suites. A runner such as
|
|
`post_generation_eval_runner.rb` maps to `post_generation_eval_test.rb`.
|
|
- Use the project's full pre-merge tier (`EVAL_JUDGE_TIER=full` for this runner).
|
|
Do not substitute a cheaper development tier. If selection remains uncertain,
|
|
include every plausibly affected suite.
|
|
|
|
**2. Run the selected command and preserve its exit status.**
|
|
|
|
For the Rails example:
|
|
|
|
```bash
|
|
set -o pipefail
|
|
EVAL_JUDGE_TIER=full EVAL_VERBOSE=1 bin/test-lane --eval test/evals/<suite>_eval_test.rb 2>&1 | tee /tmp/ship_evals.txt
|
|
```
|
|
|
|
Use the native command for other stacks. Respect the project's concurrency and
|
|
retry policy. Rails suites sharing a test lane run sequentially; stop on the
|
|
first failure before starting another paid suite.
|
|
|
|
**Long eval suites (30+ min): launch detached so a turn boundary can't kill them.**
|
|
Use the detached runner and eval lock; set its outer timeout to cover the
|
|
project's declared suite duration and retries. Do not change individual eval
|
|
limits. For a suite whose full bound fits 5400 seconds:
|
|
|
|
```bash
|
|
~/.claude/skills/gstack/bin/gstack-detach --label ship-evals --lock gstack-evals --timeout 5400 -- <project eval command>
|
|
```
|
|
|
|
Poll the printed log for `### gstack-detach EXIT=<code> ###`. Silence is not
|
|
success. Retain every configured attempt; skipped or unstarted cases do not
|
|
satisfy coverage.
|
|
|
|
**3. Check results and save evidence for Step 19.**
|
|
|
|
- **If any eval fails:** Do not rerun the full gate. Show failures and available
|
|
costs, then Read `~/.claude/skills/gstack/ship/sections/measure.md` and run its
|
|
loop for each red case: classify, measure alone, fix at the cause, re-measure,
|
|
then gate once. **STOP** only at its named-red stop.
|
|
- **If all selected evals pass:** Record actual counts, any reused evidence and
|
|
its source, and available costs. Continue to Step 7.
|
|
|
|
---
|