feat: test bootstrap, regression tests, coverage audit, retro test health

- Add {{TEST_BOOTSTRAP}} resolver to gen-skill-docs.ts
- Add Phase 8e.5 regression test generation to /qa and /qa-design-review
- Add Step 3.4 test coverage audit with quality scoring to /ship
- Add test health tracking to /retro
- Add 2 E2E evals (bootstrap + coverage audit)
- Add 26 validation tests
- Update ARCHITECTURE.md placeholder table
- Add 2 P3 TODOs (CI/CD non-GitHub, auto-upgrade weak tests)
This commit is contained in:
Garry Tan
2026-03-17 10:41:40 -07:00
parent 73b00b4e29
commit 04eb7824bd
16 changed files with 1512 additions and 6 deletions
+101
View File
@@ -11,6 +11,7 @@ allowed-tools:
- Grep
- Glob
- AskUserQuestion
- WebSearch
---
{{PREAMBLE}}
@@ -39,6 +40,7 @@ You are running the `/ship` workflow. This is a **non-interactive, fully automat
- Multi-file changesets (auto-split into bisectable commits)
- TODOS.md completed-item detection (auto-mark)
- Auto-fixable review findings (dead code, N+1, stale comments — fixed automatically)
- Test coverage gaps (auto-generate and commit, or flag in PR body)
---
@@ -75,6 +77,12 @@ git fetch origin <base> && git merge origin/<base> --no-edit
---
## Step 2.5: Test Framework Bootstrap
{{TEST_BOOTSTRAP}}
---
## Step 3: Run tests (on merged code)
**Do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
@@ -159,6 +167,94 @@ If multiple suites need to run, run them sequentially (each needs a test lane).
---
## Step 3.4: Test Coverage Audit
100% coverage is the goal — every untested path is a path where bugs hide and vibe coding becomes yolo coding. Evaluate what was ACTUALLY coded (from the diff), not what was planned.
**0. Before/after test count:**
```bash
# Count test files before any generation
find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' | grep -v node_modules | wc -l
```
Store this number for the PR body.
**1. Build the code path map** from `git diff origin/<base>...HEAD`:
Extract all new or modified:
- Functions/methods (def, function, const, class methods)
- Conditional branches (if/else, switch/case, ternary, guard clauses, early returns)
- API routes/endpoints (route definitions, controller actions)
- Components (new files or new exports)
- Error handlers (try/catch, rescue, error boundaries, fallback paths)
**2. Search for corresponding tests and score quality:**
For each code path, search for a test exercising it:
- `src/services/billing.ts:processPayment` → `billing.test.ts`, `billing.spec.ts`
- `app/controllers/payments_controller.rb#create` `test/controllers/payments_controller_test.rb`
- New if/else → tests for BOTH paths (not just happy path)
- New error handler → test triggering the error condition
Quality scoring rubric:
- ★★★ Tests behavior with edge cases AND error paths
- ★★ Tests correct behavior, happy path only
- ★ Smoke test / existence check / trivial assertion (e.g., "it renders", "it doesn't throw")
**3. Output ASCII coverage diagram:**
```
NEW CODE PATH COVERAGE MAP
===========================
[+] src/services/billing.ts
├── processPayment()
│ ├── [★★★ TESTED] Happy path + card declined + timeout — billing.test.ts:42
│ ├── [GAP] Network timeout — NO TEST
│ └── [GAP] Invalid currency — NO TEST
└── refundPayment()
├── [★★ TESTED] Full refund — billing.test.ts:89
└── [★ TESTED] Partial refund (checks non-throw only) — billing.test.ts:101
─────────────────────────────────
COVERAGE: 3/5 new paths tested (60%)
QUALITY: ★★★: 1 ★★: 1 ★: 1 (avg: ★★)
GAPS: 2 paths need tests
─────────────────────────────────
```
**Fast path:** All paths covered → "Step 3.4: All new code paths have test coverage ✓" Continue.
**4. Generate tests for uncovered paths:**
If test framework detected (or bootstrapped in Step 2.5):
- Prioritize error handlers and edge cases first (happy paths are more likely already tested)
- Read 2-3 existing test files to match conventions exactly
- Generate unit tests. Mock all external dependencies (DB, API, Redis).
- Write tests that exercise the specific uncovered path with real assertions
- Run each test. Passes → commit as `test: coverage for {feature}`
- Fails → fix once. Still fails → revert, note gap in diagram.
Caps: 30 code paths max, 10 tests generated max, 2-min per-test exploration cap.
If no test framework AND user declined bootstrap → diagram only, no generation. Note: "Test generation skipped — no test framework configured."
**Diff is test-only changes:** Skip Step 3.4 entirely: "No new application code paths to audit."
**5. After-count and coverage summary:**
```bash
# Count test files after generation
find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' | grep -v node_modules | wc -l
```
For PR body: `Tests: {before} → {after} (+{delta} new)`
Coverage line: `Test Coverage Audit: N new code paths. M covered (X%). K tests generated, J committed.`
---
## Step 3.5: Pre-Landing Review
Review the diff for structural issues that tests don't catch.
@@ -387,6 +483,10 @@ gh pr create --base <base> --title "<type>: <summary>" --body "$(cat <<'EOF'
## Summary
<bullet points from CHANGELOG>
## Test Coverage
<coverage diagram from Step 3.4, or "All new code paths have test coverage.">
<If Step 3.4 ran: "Tests: {before} → {after} (+{delta} new)">
## Pre-Landing Review
<findings from Step 3.5, or "No issues found.">
@@ -428,4 +528,5 @@ EOF
- **Split commits for bisectability** — each commit = one logical change.
- **TODOS.md completion detection must be conservative.** Only mark items as completed when the diff clearly shows the work is done.
- **Use Greptile reply templates from greptile-triage.md.** Every reply includes evidence (inline diff, code references, re-rank suggestion). Never post vague replies.
- **Step 3.4 generates coverage tests.** They must pass before committing. Never commit failing tests.
- **The goal is: user says `/ship`, next thing they see is the review + PR URL.**