feat: Test Bootstrap + Regression Tests + Coverage Audit (v0.6.0) (#136)

* feat: test bootstrap, regression tests, coverage audit, retro test health

- Add {{TEST_BOOTSTRAP}} resolver to gen-skill-docs.ts
- Add Phase 8e.5 regression test generation to /qa and /qa-design-review
- Add Step 3.4 test coverage audit with quality scoring to /ship
- Add test health tracking to /retro
- Add 2 E2E evals (bootstrap + coverage audit)
- Add 26 validation tests
- Update ARCHITECTURE.md placeholder table
- Add 2 P3 TODOs (CI/CD non-GitHub, auto-upgrade weak tests)

* chore: bump version and changelog (v0.6.0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: make coverage audit trace actual codepaths, not just syntax patterns

Step 3.4 now instructs Claude to read full files, trace data flow through
every branch, diagram the execution, and check each branch against tests.
Phase 8e.5 regression tests now trace the bug's codepath before writing
the test, catching adjacent edge cases.

* feat: coverage audit now maps user flows, interactions, and error states

Step 3.4 now covers the full picture: code branches AND user-facing behavior.
Maps user flows (complete journey through the feature), interaction edge cases
(double-click, back button, stale state, slow connection), error states
(what does the user actually see?), and boundary states (zero results,
10k results, max-length input). Coverage diagram splits into Code Path
Coverage and User Flow Coverage sections with separate percentages.

* fix: raise test gen cap to 20, add validation tests for user flow coverage

- Raise Step 3.4 test generation cap from 10 to 20 (code + user flow combined)
- Add 3 validation tests: codepath tracing, user flow mapping, diagram sections

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-03-17 13:05:18 -05:00
committed by GitHub
co-authored by Claude Opus 4.6
parent b65a464d37
commit a2d756f945
18 changed files with 1672 additions and 7 deletions
+28 -1
View File
@@ -164,6 +164,15 @@ cat ~/.gstack/greptile-history.md 2>/dev/null || true
# 9. TODOS.md backlog (if available)
cat TODOS.md 2>/dev/null || true
# 10. Test file count
find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' 2>/dev/null | grep -v node_modules | wc -l
# 11. Regression test commits in window
git log origin/<default> --since="<window>" --oneline --grep="test(qa):" --grep="test(design):" --grep="test: coverage"
# 12. Test files changed in window
git log origin/<default> --since="<window>" --format="" --name-only | grep -E '\.(test|spec)\.' | sort -u | wc -l
```
### Step 2: Compute Metrics
@@ -185,6 +194,7 @@ Calculate and present these metrics in a summary table:
| Detected sessions | N |
| Avg LOC/session-hour | N |
| Greptile signal | N% (Y catches, Z FPs) |
| Test Health | N total tests · M added this period · K regression tests |
Then show a **per-author leaderboard** immediately below:
@@ -408,7 +418,17 @@ Use the Write tool to save the JSON file with this schema:
}
```
**Note:** Only include the `greptile` field if `~/.gstack/greptile-history.md` exists and has entries within the time window. Only include the `backlog` field if `TODOS.md` exists. If either has no data, omit the field entirely.
**Note:** Only include the `greptile` field if `~/.gstack/greptile-history.md` exists and has entries within the time window. Only include the `backlog` field if `TODOS.md` exists. Only include the `test_health` field if test files were found (command 10 returns > 0). If any has no data, omit the field entirely.
Include test health data in the JSON when test files exist:
```json
"test_health": {
"total_test_files": 47,
"tests_added_this_period": 5,
"regression_test_commits": 3,
"test_files_changed": 8
}
```
Include backlog data in the JSON when TODOS.md exists:
```json
@@ -464,6 +484,13 @@ Narrative covering:
- Any XL PRs that should have been split
- Greptile signal ratio and trend (if history exists): "Greptile: X% signal (Y valid catches, Z false positives)"
### Test Health
- Total test files: N (from command 10)
- Tests added this period: M (from command 12 — test files changed)
- Regression test commits: list `test(qa):` and `test(design):` and `test: coverage` commits from command 11
- If prior retro exists and has `test_health`: show delta "Test count: {last} → {now} (+{delta})"
- If test ratio < 20%: flag as growth area — "100% test coverage is the goal. Tests make vibe coding safe."
### Focus & Highlights
(from Step 8)
- Focus score with interpretation
+28 -1
View File
@@ -99,6 +99,15 @@ cat ~/.gstack/greptile-history.md 2>/dev/null || true
# 9. TODOS.md backlog (if available)
cat TODOS.md 2>/dev/null || true
# 10. Test file count
find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' 2>/dev/null | grep -v node_modules | wc -l
# 11. Regression test commits in window
git log origin/<default> --since="<window>" --oneline --grep="test(qa):" --grep="test(design):" --grep="test: coverage"
# 12. Test files changed in window
git log origin/<default> --since="<window>" --format="" --name-only | grep -E '\.(test|spec)\.' | sort -u | wc -l
```
### Step 2: Compute Metrics
@@ -120,6 +129,7 @@ Calculate and present these metrics in a summary table:
| Detected sessions | N |
| Avg LOC/session-hour | N |
| Greptile signal | N% (Y catches, Z FPs) |
| Test Health | N total tests · M added this period · K regression tests |
Then show a **per-author leaderboard** immediately below:
@@ -343,7 +353,17 @@ Use the Write tool to save the JSON file with this schema:
}
```
**Note:** Only include the `greptile` field if `~/.gstack/greptile-history.md` exists and has entries within the time window. Only include the `backlog` field if `TODOS.md` exists. If either has no data, omit the field entirely.
**Note:** Only include the `greptile` field if `~/.gstack/greptile-history.md` exists and has entries within the time window. Only include the `backlog` field if `TODOS.md` exists. Only include the `test_health` field if test files were found (command 10 returns > 0). If any has no data, omit the field entirely.
Include test health data in the JSON when test files exist:
```json
"test_health": {
"total_test_files": 47,
"tests_added_this_period": 5,
"regression_test_commits": 3,
"test_files_changed": 8
}
```
Include backlog data in the JSON when TODOS.md exists:
```json
@@ -399,6 +419,13 @@ Narrative covering:
- Any XL PRs that should have been split
- Greptile signal ratio and trend (if history exists): "Greptile: X% signal (Y valid catches, Z false positives)"
### Test Health
- Total test files: N (from command 10)
- Tests added this period: M (from command 12 — test files changed)
- Regression test commits: list `test(qa):` and `test(design):` and `test: coverage` commits from command 11
- If prior retro exists and has `test_health`: show delta "Test count: {last} → {now} (+{delta})"
- If test ratio < 20%: flag as growth area — "100% test coverage is the goal. Tests make vibe coding safe."
### Focus & Highlights
(from Step 8)
- Focus score with interpretation