mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-14 00:49:00 +02:00
fix: repair frontier eval budgets and workflow instructions
Preserve frontier models and quality thresholds while fixing truncated judge output, ordered section expansion, consent checks, QA scoring, and ship audit gates. Add regression coverage and refresh generated docs. Co-Authored-By: OpenAI Codex <noreply@openai.com>
This commit is contained in:
Garry Tan
co-authored by
OpenAI Codex
OpenAI Codex
parent
0cbccc1c6c
commit
a9f9ec5f08
@@ -21,9 +21,9 @@
|
||||
|
||||
Parse the LAST line as JSON.
|
||||
|
||||
If `total` is 0, skip this step silently. Continue to Step 12.
|
||||
If `total` is 0, skip this step silently. Continue to Step 11.
|
||||
|
||||
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never lands mid-ship):** print `Greptile triage did not complete — review the PR comments manually` and continue to Step 12, recording the triage as UNAVAILABLE — not as zero comments — in the PR body: add the literal line `Greptile triage: UNAVAILABLE (dispatch failed)` to the review-results section Step 19 assembles (an unavailable triage must not read as a clean one; Step 20's metrics schema carries no triage field, so the PR body is the record). Do not block /ship on the triage subagent.
|
||||
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never lands mid-ship):** print `Greptile triage did not complete — review the PR comments manually` and continue to Step 11, recording the triage as UNAVAILABLE — not as zero comments — in the PR body: add the literal line `Greptile triage: UNAVAILABLE (dispatch failed)` to the review-results section Step 19 assembles (an unavailable triage must not read as a clean one; Step 20's metrics schema carries no triage field, so the PR body is the record). Do not block /ship on the triage subagent.
|
||||
|
||||
Otherwise, print: `+ {total} Greptile comments ({valid_actionable} valid, {already_fixed} already fixed, {false_positive} FP)`.
|
||||
|
||||
@@ -50,6 +50,6 @@ For each comment in `comments`:
|
||||
|
||||
**SUPPRESSED:** Skip silently — these are known false positives from previous triage.
|
||||
|
||||
**After all comments are resolved:** If any fixes were applied, the tests from Step 5 are now stale. **Re-run tests** (Step 5) before continuing to Step 12. If no fixes were applied, continue to Step 12.
|
||||
**After all comments are resolved:** If any fixes were applied, the tests from Step 5 are now stale. **Re-run tests** (Step 5) before continuing to Step 11. If no fixes were applied, continue to Step 11.
|
||||
|
||||
---
|
||||
|
||||
@@ -19,9 +19,9 @@
|
||||
|
||||
Parse the LAST line as JSON.
|
||||
|
||||
If `total` is 0, skip this step silently. Continue to Step 12.
|
||||
If `total` is 0, skip this step silently. Continue to Step 11.
|
||||
|
||||
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never lands mid-ship):** print `Greptile triage did not complete — review the PR comments manually` and continue to Step 12, recording the triage as UNAVAILABLE — not as zero comments — in the PR body: add the literal line `Greptile triage: UNAVAILABLE (dispatch failed)` to the review-results section Step 19 assembles (an unavailable triage must not read as a clean one; Step 20's metrics schema carries no triage field, so the PR body is the record). Do not block /ship on the triage subagent.
|
||||
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never lands mid-ship):** print `Greptile triage did not complete — review the PR comments manually` and continue to Step 11, recording the triage as UNAVAILABLE — not as zero comments — in the PR body: add the literal line `Greptile triage: UNAVAILABLE (dispatch failed)` to the review-results section Step 19 assembles (an unavailable triage must not read as a clean one; Step 20's metrics schema carries no triage field, so the PR body is the record). Do not block /ship on the triage subagent.
|
||||
|
||||
Otherwise, print: `+ {total} Greptile comments ({valid_actionable} valid, {already_fixed} already fixed, {false_positive} FP)`.
|
||||
|
||||
@@ -48,6 +48,6 @@ For each comment in `comments`:
|
||||
|
||||
**SUPPRESSED:** Skip silently — these are known false positives from previous triage.
|
||||
|
||||
**After all comments are resolved:** If any fixes were applied, the tests from Step 5 are now stale. **Re-run tests** (Step 5) before continuing to Step 12. If no fixes were applied, continue to Step 12.
|
||||
**After all comments are resolved:** If any fixes were applied, the tests from Step 5 are now stale. **Re-run tests** (Step 5) before continuing to Step 11. If no fixes were applied, continue to Step 11.
|
||||
|
||||
---
|
||||
|
||||
@@ -8,9 +8,10 @@
|
||||
|
||||
**Subagent prompt:** Pass these instructions to the subagent:
|
||||
|
||||
> You are running a ship-workflow plan completion audit. The base branch is `<base>`. Use `git diff <base>...HEAD` to see what shipped. Do not commit or push — report only.
|
||||
>
|
||||
> ### Plan File Discovery
|
||||
````text
|
||||
You are running a ship-workflow plan completion audit. The base branch is `<base>`. Use `git diff <base>...HEAD` to see what shipped. Do not commit or push. Report only: classify every item, but do not execute Gate Logic, ask the user, or advance the workflow. The parent applies those gates to your report.
|
||||
|
||||
### Plan File Discovery
|
||||
|
||||
1. **Conversation context (primary):** Check if there is an active plan file in this conversation. The host agent's system messages include plan file paths when in plan mode. If found, use it directly — this is the most reliable signal.
|
||||
|
||||
@@ -130,13 +131,30 @@ Plan: {plan file path}
|
||||
[UNVERIFIABLE] Supabase auth allowlist contains user email — external system, confirm in Supabase dashboard
|
||||
|
||||
─────────────────────────────────
|
||||
COMPLETION: 5/9 DONE, 1 PARTIAL, 1 NOT DONE, 1 CHANGED, 2 UNVERIFIABLE
|
||||
COMPLETION: 4/10 DONE, 1 PARTIAL, 2 NOT DONE, 1 CHANGED, 2 UNVERIFIABLE
|
||||
─────────────────────────────────
|
||||
```
|
||||
|
||||
After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
|
||||
{"total_items":N,"done":N,"changed":N,"partial":N,"not_done":N,"unverifiable":N,"summary":"<markdown checklist for PR body>"}
|
||||
Counts map one-to-one to the classifications above and sum to total_items. No plan or no actionable items means all counts are zero with the skip reason in summary. Do not classify work as deferred; only the parent can record a user-approved deferral.
|
||||
````
|
||||
|
||||
**Parent processing:**
|
||||
|
||||
1. Parse the LAST line of the subagent's output as JSON.
|
||||
2. Store the counts for Step 20 metrics; use `summary` in PR body.
|
||||
3. Apply Gate Logic below to `not_done` and `unverifiable` before continuing. Track user-approved deferrals separately; `partial` items receive a PR note, not the NOT DONE gate.
|
||||
4. Embed `summary` in PR body's `## Plan Completion` section (Step 19). For the UNVERIFIABLE gate, also embed `## Plan Completion — Manual Verifications` with each Y response's evidence and each D response's dropped item.
|
||||
|
||||
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never races the fallback):** Fall back to running the audit inline (parent processes the same plan-extraction + classification logic). If the inline fallback also fails (e.g., plan file unreadable, parser error), do NOT silently pass — surface the failure as an explicit AskUserQuestion: "Plan Completion audit could not run ({reason}). Options: (A) Skip audit and ship anyway — record that the audit was skipped in PR body and Step 20 metrics; (B) Stop and fix the audit." Default and recommended option is (B). Silent fail-open is the failure shape that VAS-449 surfaced.
|
||||
|
||||
---
|
||||
|
||||
|
||||
### Gate Logic
|
||||
|
||||
After producing the completion checklist, evaluate in priority order:
|
||||
The parent evaluates the completion checklist in priority order, including after an inline fallback:
|
||||
|
||||
1. **Any NOT DONE items** (highest priority — known missing work). Use AskUserQuestion:
|
||||
- Show the completion checklist above
|
||||
@@ -144,10 +162,10 @@ After producing the completion checklist, evaluate in priority order:
|
||||
- RECOMMENDATION: depends on item count and severity. If 1-2 minor items (docs, config), recommend B. If core functionality is missing, recommend A.
|
||||
- Options:
|
||||
A) Stop — implement the missing items before shipping
|
||||
B) Ship anyway — defer these to a follow-up (will create P1 TODOs in Step 5.5)
|
||||
B) Ship anyway — defer these to a follow-up (will create P1 TODOs in Step 14)
|
||||
C) These items were intentionally dropped — remove from scope
|
||||
- If A: STOP. List the missing items for the user to implement.
|
||||
- If B: Continue. For each NOT DONE item, create a P1 TODO in Step 5.5 with "Deferred from plan: {plan file path}".
|
||||
- If B: Continue. For each NOT DONE item, create a P1 TODO in Step 14 with "Deferred from plan: {plan file path}".
|
||||
- If C: Continue. Note in PR body: "Plan items intentionally dropped: {list}."
|
||||
|
||||
2. **Any UNVERIFIABLE items** (silent gaps — the diff cannot prove them either way). Only fires after NOT DONE is resolved or absent.
|
||||
@@ -174,21 +192,7 @@ After producing the completion checklist, evaluate in priority order:
|
||||
|
||||
**No plan file found:** Skip entirely. "No plan file detected — skipping plan completion audit."
|
||||
|
||||
**Include in PR body (Step 8):** Add a `## Plan Completion` section with the checklist summary.
|
||||
>
|
||||
> After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
|
||||
> `{"total_items":N,"done":N,"changed":N,"deferred":N,"unverifiable":N,"summary":"<markdown checklist for PR body>"}`
|
||||
|
||||
**Parent processing:**
|
||||
|
||||
1. Parse the LAST line of the subagent's output as JSON.
|
||||
2. Store `done`, `deferred`, `unverifiable` for Step 20 metrics; use `summary` in PR body.
|
||||
3. If `deferred > 0` or `unverifiable > 0` and no user override, present the items via the appropriate AskUserQuestion (see Gate Logic priority order above) before continuing.
|
||||
4. Embed `summary` in PR body's `## Plan Completion` section (Step 19). If `unverifiable > 0` and the user picked option A in the UNVERIFIABLE gate, also embed `## Plan Completion — Manual Verifications` listing each user-confirmed item.
|
||||
|
||||
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never races the fallback):** Fall back to running the audit inline (parent processes the same plan-extraction + classification logic). If the inline fallback also fails (e.g., plan file unreadable, parser error), do NOT silently pass — surface the failure as an explicit AskUserQuestion: "Plan Completion audit could not run ({reason}). Options: (A) Skip audit and ship anyway — record that the audit was skipped in PR body and Step 20 metrics; (B) Stop and fix the audit." Default and recommended option is (B). Silent fail-open is the failure shape that VAS-449 surfaced.
|
||||
|
||||
---
|
||||
**Include in PR body (Step 19):** Add a `## Plan Completion` section with the checklist summary.
|
||||
|
||||
## Step 8.1: Plan Verification
|
||||
|
||||
|
||||
@@ -6,24 +6,29 @@
|
||||
|
||||
**Subagent prompt:** Pass these instructions to the subagent:
|
||||
|
||||
> You are running a ship-workflow plan completion audit. The base branch is `<base>`. Use `git diff <base>...HEAD` to see what shipped. Do not commit or push — report only.
|
||||
>
|
||||
> {{PLAN_COMPLETION_AUDIT_SHIP}}
|
||||
>
|
||||
> After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
|
||||
> `{"total_items":N,"done":N,"changed":N,"deferred":N,"unverifiable":N,"summary":"<markdown checklist for PR body>"}`
|
||||
````text
|
||||
You are running a ship-workflow plan completion audit. The base branch is `<base>`. Use `git diff <base>...HEAD` to see what shipped. Do not commit or push. Report only: classify every item, but do not execute Gate Logic, ask the user, or advance the workflow. The parent applies those gates to your report.
|
||||
|
||||
{{PLAN_COMPLETION_AUDIT_SHIP}}
|
||||
|
||||
After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
|
||||
{"total_items":N,"done":N,"changed":N,"partial":N,"not_done":N,"unverifiable":N,"summary":"<markdown checklist for PR body>"}
|
||||
Counts map one-to-one to the classifications above and sum to total_items. No plan or no actionable items means all counts are zero with the skip reason in summary. Do not classify work as deferred; only the parent can record a user-approved deferral.
|
||||
````
|
||||
|
||||
**Parent processing:**
|
||||
|
||||
1. Parse the LAST line of the subagent's output as JSON.
|
||||
2. Store `done`, `deferred`, `unverifiable` for Step 20 metrics; use `summary` in PR body.
|
||||
3. If `deferred > 0` or `unverifiable > 0` and no user override, present the items via the appropriate AskUserQuestion (see Gate Logic priority order above) before continuing.
|
||||
4. Embed `summary` in PR body's `## Plan Completion` section (Step 19). If `unverifiable > 0` and the user picked option A in the UNVERIFIABLE gate, also embed `## Plan Completion — Manual Verifications` listing each user-confirmed item.
|
||||
2. Store the counts for Step 20 metrics; use `summary` in PR body.
|
||||
3. Apply Gate Logic below to `not_done` and `unverifiable` before continuing. Track user-approved deferrals separately; `partial` items receive a PR note, not the NOT DONE gate.
|
||||
4. Embed `summary` in PR body's `## Plan Completion` section (Step 19). For the UNVERIFIABLE gate, also embed `## Plan Completion — Manual Verifications` with each Y response's evidence and each D response's dropped item.
|
||||
|
||||
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never races the fallback):** Fall back to running the audit inline (parent processes the same plan-extraction + classification logic). If the inline fallback also fails (e.g., plan file unreadable, parser error), do NOT silently pass — surface the failure as an explicit AskUserQuestion: "Plan Completion audit could not run ({reason}). Options: (A) Skip audit and ship anyway — record that the audit was skipped in PR body and Step 20 metrics; (B) Stop and fix the audit." Default and recommended option is (B). Silent fail-open is the failure shape that VAS-449 surfaced.
|
||||
|
||||
---
|
||||
|
||||
{{PLAN_COMPLETION_GATE_SHIP}}
|
||||
|
||||
{{PLAN_VERIFICATION_EXEC}}
|
||||
|
||||
{{LEARNINGS_SEARCH:query=release ship version changelog merge pr}}
|
||||
|
||||
@@ -161,8 +161,8 @@ you missed it.>
|
||||
<If Step 18 returned `documentation_section: null` (no docs updated), omit this section entirely.>
|
||||
|
||||
## Test plan
|
||||
- [x] All Rails tests pass (N runs, 0 failures)
|
||||
- [x] All Vitest tests pass (N tests)
|
||||
- [x] <Actual project test command>: <observed passing summary>
|
||||
- [x] <Other executed test lane, if any>: <observed passing summary>
|
||||
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||||
```
|
||||
@@ -194,7 +194,7 @@ printf '%s' "v$NEW_VERSION <type>: <summary>" | ~/.claude/skills/gstack/bin/gsta
|
||||
```
|
||||
|
||||
HIGH blocks (exit 3, no skip). MEDIUM → AskUserQuestion (PII subset offers
|
||||
`--auto-redact`). Same scan runs before the `gh pr edit --body` path (Step 17).
|
||||
`--auto-redact`). Same scan runs before the `gh pr edit --body` path (Step 19).
|
||||
|
||||
**If GitHub:** create from the SCANNED file (exact bytes scanned = bytes sent).
|
||||
`$PR_BODY_FILE` comes from the scan block above — restate it in this shell if
|
||||
|
||||
@@ -159,8 +159,8 @@ you missed it.>
|
||||
<If Step 18 returned `documentation_section: null` (no docs updated), omit this section entirely.>
|
||||
|
||||
## Test plan
|
||||
- [x] All Rails tests pass (N runs, 0 failures)
|
||||
- [x] All Vitest tests pass (N tests)
|
||||
- [x] <Actual project test command>: <observed passing summary>
|
||||
- [x] <Other executed test lane, if any>: <observed passing summary>
|
||||
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||||
```
|
||||
@@ -192,7 +192,7 @@ printf '%s' "v$NEW_VERSION <type>: <summary>" | ~/.claude/skills/gstack/bin/gsta
|
||||
```
|
||||
|
||||
HIGH blocks (exit 3, no skip). MEDIUM → AskUserQuestion (PII subset offers
|
||||
`--auto-redact`). Same scan runs before the `gh pr edit --body` path (Step 17).
|
||||
`--auto-redact`). Same scan runs before the `gh pr edit --body` path (Step 19).
|
||||
|
||||
**If GitHub:** create from the SCANNED file (exact bytes scanned = bytes sent).
|
||||
`$PR_BODY_FILE` comes from the scan block above — restate it in this shell if
|
||||
|
||||
@@ -329,7 +329,7 @@ Logging simplification's advisories as `findings: 0` would auto-gate the
|
||||
lens into permanent silence after 10 dispatches.
|
||||
|
||||
Include the Design specialist even though it uses `design-checklist.md` instead of the specialist schema files.
|
||||
Remember these stats — you will need them for the review-log entry in Step 5.8.
|
||||
Remember these stats — you will need them for the review-log persist.
|
||||
|
||||
---
|
||||
|
||||
@@ -391,6 +391,8 @@ If no prior reviews exist or none have a `findings` array, skip this step silent
|
||||
|
||||
Output a summary header: `Pre-Landing Review: N issues (X critical, Y informational)`
|
||||
|
||||
**Resume the Step 9 checklist at item 4 below.** The intervening Step 9.x specialist phases augment items 1-3; they do not replace the Fix-First processing and persistence that follow.
|
||||
|
||||
4. **Classify each finding from both the checklist pass and specialist review (Step 9.1-Step 9.2) as AUTO-FIX or ASK** per the Fix-First Heuristic in
|
||||
checklist.md. Critical findings lean toward ASK; informational lean toward AUTO-FIX.
|
||||
|
||||
@@ -404,9 +406,9 @@ Output a summary header: `Pre-Landing Review: N issues (X critical, Y informatio
|
||||
- If 3 or fewer ASK items, you may use individual AskUserQuestion calls instead
|
||||
|
||||
7. **After all fixes (auto + user-approved):**
|
||||
- If ANY fixes were applied: commit fixed files by name (`git add <fixed-files> && git commit -m "fix: pre-landing review fixes"`), then **stay in this invocation and loop**: re-run the test suite (Step 5) on the fixed code, then re-run this review (Step 9 items 2-6) against the updated diff. Repeat until one full pass applies ZERO fixes — tests green and review clean — then continue to Step 12. NEVER stop to tell the user to run `/ship` again; a fix-and-rerun cycle has no user decision in it, and stopping there breaks the fully-automated contract (#2391).
|
||||
- If ANY fixes were applied: commit fixed files by name (`git add <fixed-files> && git commit -m "fix: pre-landing review fixes"`), then **stay in this invocation and loop**: re-run the test suite (Step 5) on the fixed code, then re-run this review (Step 9 items 2-6) against the updated diff. Repeat until one full pass applies ZERO fixes — tests green and review clean — then continue to Step 10. NEVER stop to tell the user to run `/ship` again; a fix-and-rerun cycle has no user decision in it, and stopping there breaks the fully-automated contract (#2391).
|
||||
- **Bound: 3 fix cycles.** If the 3rd cycle still applies fixes, STOP and report which findings keep reappearing — a review that won't converge is a genuine blocker worth human eyes, not a re-run request.
|
||||
- If no fixes applied (all ASK items skipped, or no issues found): continue to Step 12.
|
||||
- If no fixes applied (all ASK items skipped, or no issues found): continue to Step 10.
|
||||
|
||||
8. Output summary: `Pre-Landing Review: N issues — M auto-fixed, K asked (J fixed, L skipped)`
|
||||
|
||||
|
||||
@@ -20,6 +20,8 @@ Review the diff for structural issues that tests don't catch.
|
||||
|
||||
{{CROSS_REVIEW_DEDUP}}
|
||||
|
||||
**Resume the Step 9 checklist at item 4 below.** The intervening Step 9.x specialist phases augment items 1-3; they do not replace the Fix-First processing and persistence that follow.
|
||||
|
||||
4. **Classify each finding from both the checklist pass and specialist review (Step 9.1-Step 9.2) as AUTO-FIX or ASK** per the Fix-First Heuristic in
|
||||
checklist.md. Critical findings lean toward ASK; informational lean toward AUTO-FIX.
|
||||
|
||||
@@ -33,9 +35,9 @@ Review the diff for structural issues that tests don't catch.
|
||||
- If 3 or fewer ASK items, you may use individual AskUserQuestion calls instead
|
||||
|
||||
7. **After all fixes (auto + user-approved):**
|
||||
- If ANY fixes were applied: commit fixed files by name (`git add <fixed-files> && git commit -m "fix: pre-landing review fixes"`), then **stay in this invocation and loop**: re-run the test suite (Step 5) on the fixed code, then re-run this review (Step 9 items 2-6) against the updated diff. Repeat until one full pass applies ZERO fixes — tests green and review clean — then continue to Step 12. NEVER stop to tell the user to run `/ship` again; a fix-and-rerun cycle has no user decision in it, and stopping there breaks the fully-automated contract (#2391).
|
||||
- If ANY fixes were applied: commit fixed files by name (`git add <fixed-files> && git commit -m "fix: pre-landing review fixes"`), then **stay in this invocation and loop**: re-run the test suite (Step 5) on the fixed code, then re-run this review (Step 9 items 2-6) against the updated diff. Repeat until one full pass applies ZERO fixes — tests green and review clean — then continue to Step 10. NEVER stop to tell the user to run `/ship` again; a fix-and-rerun cycle has no user decision in it, and stopping there breaks the fully-automated contract (#2391).
|
||||
- **Bound: 3 fix cycles.** If the 3rd cycle still applies fixes, STOP and report which findings keep reappearing — a review that won't converge is a genuine blocker worth human eyes, not a re-run request.
|
||||
- If no fixes applied (all ASK items skipped, or no issues found): continue to Step 12.
|
||||
- If no fixes applied (all ASK items skipped, or no issues found): continue to Step 10.
|
||||
|
||||
8. Output summary: `Pre-Landing Review: N issues — M auto-fixed, K asked (J fixed, L skipped)`
|
||||
|
||||
|
||||
@@ -8,9 +8,10 @@
|
||||
|
||||
**Subagent prompt:** Pass the following instructions to the subagent, with `<base>` substituted with the base branch:
|
||||
|
||||
> You are running a ship-workflow test coverage audit. Run `git diff <base>...HEAD` as needed. Do not commit or push — report only.
|
||||
>
|
||||
> 100% coverage is the goal — every untested path is a path where bugs hide and vibe coding becomes yolo coding. Evaluate what was ACTUALLY coded (from the diff), not what was planned.
|
||||
````text
|
||||
You are running a ship-workflow test coverage audit. Run `git diff <base>...HEAD` as needed. Do not commit or push. Perform only this audit; return unresolved user decisions to the parent instead of asking or advancing to another workflow step.
|
||||
|
||||
100% coverage is the goal — every untested path is a path where bugs hide and vibe coding becomes yolo coding. Evaluate what was ACTUALLY coded (from the diff), not what was planned.
|
||||
|
||||
### Test Framework Detection
|
||||
|
||||
@@ -37,7 +38,7 @@ ls jest.config.* vitest.config.* playwright.config.* cypress.config.* .rspec pyt
|
||||
git ls-files | grep -cE '(^|/)(tests?|spec|__tests__)/|(^|/)tests?\.py$|(^|/)test_[^/]+\.py$|_test\.(go|py|rb|ts|js|exs)$|\.(test|spec)\.[jt]sx?$|_spec\.rb$|Test\.(java|kt)$' | sed 's/^/TESTFILES:/'
|
||||
```
|
||||
|
||||
3. **If no framework detected:** falls through to the Test Framework Bootstrap step (Step 4) which handles full setup.
|
||||
3. **If no framework detected:** use the bootstrap decision already made in Step 4; report diagram-only coverage if setup was declined. Do not restart bootstrap from this audit.
|
||||
|
||||
**0. Before/after test count:**
|
||||
|
||||
@@ -131,8 +132,6 @@ A regression is when:
|
||||
|
||||
When uncertain whether a change is a regression, err on the side of writing the test.
|
||||
|
||||
Format: commit as `test: regression test for {what broke}`
|
||||
|
||||
**4. Output ASCII coverage diagram:**
|
||||
|
||||
Include BOTH code paths and user flows in the same diagram. Mark E2E-worthy and eval-worthy paths:
|
||||
@@ -168,14 +167,14 @@ If test framework detected (or bootstrapped in Step 4):
|
||||
- For paths marked [→E2E]: generate integration/E2E tests using the project's E2E framework (Playwright, Cypress, Capybara, etc.)
|
||||
- For paths marked [→EVAL]: generate eval tests using the project's eval framework, or flag for manual eval if none exists
|
||||
- Write tests that exercise the specific uncovered path with real assertions
|
||||
- Run each test. Passes → commit as `test: coverage for {feature}`
|
||||
- Run each test. Passes → keep the change and report its path; the parent commits in Step 15.
|
||||
- Fails → fix once. Still fails → revert, note gap in diagram.
|
||||
|
||||
Caps: 30 code paths max, 20 tests generated max (code + user flow combined), 2-min per-test exploration cap.
|
||||
|
||||
If no test framework AND user declined bootstrap → diagram only, no generation. Note: "Test generation skipped — no test framework configured."
|
||||
|
||||
**Diff is test-only changes:** Skip Step 7 entirely: "No new application code paths to audit."
|
||||
**Diff is test-only changes:** Return a skipped audit with null coverage, zero gaps, and "No new application code paths to audit."
|
||||
|
||||
**6. After-count and coverage summary:**
|
||||
|
||||
@@ -185,40 +184,7 @@ git ls-files 2>/dev/null | grep -E '(\.test\.|\.spec\.|_test\.|_spec\.)' | wc -l
|
||||
```
|
||||
|
||||
For PR body: `Tests: {before} → {after} (+{delta} new)`
|
||||
Coverage line: `Test Coverage Audit: N new code paths. M covered (X%). K tests generated, J committed.`
|
||||
|
||||
**7. Coverage gate:**
|
||||
|
||||
Before proceeding, check CLAUDE.md for a `## Test Coverage` section with `Minimum:` and `Target:` fields. If found, use those percentages. Otherwise use defaults: Minimum = 60%, Target = 80%.
|
||||
|
||||
Using the coverage percentage from the diagram in substep 4 (the `COVERAGE: X/Y (Z%)` line):
|
||||
|
||||
- **>= target:** Pass. "Coverage gate: PASS ({X}%)." Continue.
|
||||
- **>= minimum, < target:** Use AskUserQuestion:
|
||||
- "AI-assessed coverage is {X}%. {N} code paths are untested. Target is {target}%."
|
||||
- RECOMMENDATION: Choose A because untested code paths are where production bugs hide.
|
||||
- Options:
|
||||
A) Generate more tests for remaining gaps (recommended)
|
||||
B) Ship anyway — I accept the coverage risk
|
||||
C) These paths don't need tests — mark as intentionally uncovered
|
||||
- If A: Loop back to substep 5 (generate tests) targeting the remaining gaps. After second pass, if still below target, present AskUserQuestion again with updated numbers. Maximum 2 generation passes total.
|
||||
- If B: Continue. Include in PR body: "Coverage gate: {X}% — user accepted risk."
|
||||
- If C: Continue. Include in PR body: "Coverage gate: {X}% — {N} paths intentionally uncovered."
|
||||
|
||||
- **< minimum:** Use AskUserQuestion:
|
||||
- "AI-assessed coverage is critically low ({X}%). {N} of {M} code paths have no tests. Minimum threshold is {minimum}%."
|
||||
- RECOMMENDATION: Choose A because less than {minimum}% means more code is untested than tested.
|
||||
- Options:
|
||||
A) Generate tests for remaining gaps (recommended)
|
||||
B) Override — ship with low coverage (I understand the risk)
|
||||
- If A: Loop back to substep 5. Maximum 2 passes. If still below minimum after 2 passes, present the override choice again.
|
||||
- If B: Continue. Include in PR body: "Coverage gate: OVERRIDDEN at {X}%."
|
||||
|
||||
**Coverage percentage undetermined:** If the coverage diagram doesn't produce a clear numeric percentage (ambiguous output, parse error), **skip the gate** with: "Coverage gate: could not determine percentage — skipping." Do not default to 0% or block.
|
||||
|
||||
**Test-only diffs:** Skip the gate (same as the existing fast-path).
|
||||
|
||||
**100% coverage:** "Coverage gate: PASS (100%)." Continue.
|
||||
Coverage line: `Test Coverage Audit: N new code paths. M covered (X%). K tests generated, awaiting parent commit.`
|
||||
|
||||
### Test Plan Artifact
|
||||
|
||||
@@ -250,9 +216,11 @@ Repo: {owner/repo}
|
||||
## Critical Paths
|
||||
- {end-to-end flow that must work}
|
||||
```
|
||||
>
|
||||
> After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
|
||||
> `{"coverage_pct":N,"gaps":N,"diagram":"<full markdown coverage diagram for PR body>","tests_added":["path",...]}`
|
||||
|
||||
After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
|
||||
{"coverage_pct":N,"gaps":N,"diagram":"<full markdown coverage diagram for PR body>","tests_added":["path",...]}
|
||||
Use null for an undetermined or skipped coverage percentage, not zero. Include every remaining gap in the diagram so the parent can target a second pass.
|
||||
````
|
||||
|
||||
**Parent processing:**
|
||||
|
||||
@@ -263,4 +231,40 @@ Repo: {owner/repo}
|
||||
|
||||
**If the subagent fails, times out, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never races the fallback):** Fall back to running the audit inline in the parent. Do not block /ship on subagent failure — partial results are better than none.
|
||||
|
||||
|
||||
**7. Coverage gate:**
|
||||
|
||||
The parent owns this gate after receiving the audit result, including after an inline fallback. Generated tests stay uncommitted until Step 15. Any further generation uses the same audit prompt with the remaining gaps and pass count supplied.
|
||||
|
||||
Before proceeding, check CLAUDE.md for a `## Test Coverage` section with `Minimum:` and `Target:` fields. If found, use those percentages. Otherwise use defaults: Minimum = 60%, Target = 80%.
|
||||
|
||||
Using the coverage percentage from the diagram in substep 4 (the `COVERAGE: X/Y (Z%)` line):
|
||||
|
||||
- **>= target:** Pass. "Coverage gate: PASS ({X}%)." Continue.
|
||||
- **>= minimum, < target:** Use AskUserQuestion:
|
||||
- "AI-assessed coverage is {X}%. {N} code paths are untested. Target is {target}%."
|
||||
- RECOMMENDATION: Choose A because untested code paths are where production bugs hide.
|
||||
- Options:
|
||||
A) Generate more tests for remaining gaps (recommended)
|
||||
B) Ship anyway — I accept the coverage risk
|
||||
C) These paths don't need tests — mark as intentionally uncovered
|
||||
- If A: Dispatch one more generation pass targeting remaining gaps, then re-evaluate the result here. Maximum 2 generation passes total. At the cap, offer only B/C or stop; do not offer another generation pass.
|
||||
- If B: Continue. Include in PR body: "Coverage gate: {X}% — user accepted risk."
|
||||
- If C: Continue. Include in PR body: "Coverage gate: {X}% — {N} paths intentionally uncovered."
|
||||
|
||||
- **< minimum:** Use AskUserQuestion:
|
||||
- "AI-assessed coverage is critically low ({X}%). {N} of {M} code paths have no tests. Minimum threshold is {minimum}%."
|
||||
- RECOMMENDATION: Choose A because less than {minimum}% means more code is untested than tested.
|
||||
- Options:
|
||||
A) Generate tests for remaining gaps (recommended)
|
||||
B) Override — ship with low coverage (I understand the risk)
|
||||
- If A: Dispatch one more generation pass. Maximum 2 passes total. At the cap, offer only B or stop; do not offer another generation pass.
|
||||
- If B: Continue. Include in PR body: "Coverage gate: OVERRIDDEN at {X}%."
|
||||
|
||||
**Coverage percentage undetermined:** If the coverage diagram doesn't produce a clear numeric percentage (ambiguous output, parse error), **skip the gate** with: "Coverage gate: could not determine percentage — skipping." Do not default to 0% or block.
|
||||
|
||||
**Test-only diffs:** Skip the gate (same as the existing fast-path).
|
||||
|
||||
**100% coverage:** "Coverage gate: PASS (100%)." Continue.
|
||||
|
||||
---
|
||||
|
||||
@@ -6,12 +6,15 @@
|
||||
|
||||
**Subagent prompt:** Pass the following instructions to the subagent, with `<base>` substituted with the base branch:
|
||||
|
||||
> You are running a ship-workflow test coverage audit. Run `git diff <base>...HEAD` as needed. Do not commit or push — report only.
|
||||
>
|
||||
> {{TEST_COVERAGE_AUDIT_SHIP}}
|
||||
>
|
||||
> After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
|
||||
> `{"coverage_pct":N,"gaps":N,"diagram":"<full markdown coverage diagram for PR body>","tests_added":["path",...]}`
|
||||
````text
|
||||
You are running a ship-workflow test coverage audit. Run `git diff <base>...HEAD` as needed. Do not commit or push. Perform only this audit; return unresolved user decisions to the parent instead of asking or advancing to another workflow step.
|
||||
|
||||
{{TEST_COVERAGE_AUDIT_SHIP}}
|
||||
|
||||
After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
|
||||
{"coverage_pct":N,"gaps":N,"diagram":"<full markdown coverage diagram for PR body>","tests_added":["path",...]}
|
||||
Use null for an undetermined or skipped coverage percentage, not zero. Include every remaining gap in the diagram so the parent can target a second pass.
|
||||
````
|
||||
|
||||
**Parent processing:**
|
||||
|
||||
@@ -22,4 +25,6 @@
|
||||
|
||||
**If the subagent fails, times out, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never races the fallback):** Fall back to running the audit inline in the parent. Do not block /ship on subagent failure — partial results are better than none.
|
||||
|
||||
{{TEST_COVERAGE_GATE_SHIP}}
|
||||
|
||||
---
|
||||
|
||||
@@ -194,11 +194,13 @@ Only commit if there are changes. Stage all bootstrap files (config, test direct
|
||||
|
||||
## Step 5: Run tests (on merged code)
|
||||
|
||||
**Do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
|
||||
Use the project's test commands discovered in Step 4 or documented in CLAUDE.md/AGENTS.md. Run every applicable suite; do not assume Rails or Vitest. The commands below are examples only for repositories that actually provide them. Use the same lane labels and exact commands again in Step 16.
|
||||
|
||||
**For Rails projects using `bin/test-lane`, do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
|
||||
`db:test:prepare` internally, which loads the schema into the correct lane database.
|
||||
Running bare test migrations without INSTANCE hits an orphan DB and corrupts structure.sql.
|
||||
|
||||
Run both test suites in parallel, each wrapped in the evidence ledger. The
|
||||
Run independent test suites in parallel, each wrapped in the evidence ledger. The
|
||||
wrapper is transparent (streams output live, exit code passes through) and
|
||||
records `{command, exit, working-tree fingerprint, log path}` to
|
||||
`~/.gstack/projects/<slug>/<branch>-evidence.jsonl` — Step 16 cites this
|
||||
@@ -210,7 +212,7 @@ record instead of re-running when the content hasn't changed:
|
||||
wait
|
||||
```
|
||||
|
||||
After both complete, check the `gstack-evidence: recorded label=... exit=...
|
||||
After all suites complete, check the `gstack-evidence: recorded label=... exit=...
|
||||
log=...` summary lines — each carries the lane's exit code and a per-run log
|
||||
file (no shared /tmp collisions between concurrent ships). Read the log files
|
||||
for failure detail.
|
||||
@@ -331,6 +333,8 @@ Use AskUserQuestion:
|
||||
|
||||
Evals are mandatory when prompt-related files change. Skip this step entirely if no prompt files are in the diff.
|
||||
|
||||
Use the project's documented eval selection and pre-merge command first (including changed skill templates and judge/harness code). The Rails patterns and commands below apply only when that runner exists. For other stacks, use their native eval scripts and dependency map. If prompts changed but no eval command is documented, report the missing validation and ask before shipping; never silently treat that as no affected prompts.
|
||||
|
||||
**1. Check if the diff touches prompt-related files:**
|
||||
|
||||
```bash
|
||||
@@ -346,7 +350,7 @@ Match against these patterns (from CLAUDE.md):
|
||||
- `config/system_prompts/*.txt`
|
||||
- `test/evals/**/*` (eval infrastructure changes affect all suites)
|
||||
|
||||
**If no matches:** Print "No prompt-related files changed — skipping evals." and continue to Step 9.
|
||||
**If no matches:** Print "No prompt-related files changed — skipping evals." and continue to Step 7.
|
||||
|
||||
**2. Identify affected eval suites:**
|
||||
|
||||
@@ -392,7 +396,7 @@ poller is reaped.
|
||||
**4. Check results:**
|
||||
|
||||
- **If any eval fails:** Show the failures, the cost dashboard, and **STOP**. Do not proceed.
|
||||
- **If all pass:** Note pass counts and cost. Continue to Step 9.
|
||||
- **If all pass:** Note pass counts and cost. Continue to Step 7.
|
||||
|
||||
**5. Save eval output** — include eval results and cost dashboard in the PR body (Step 19).
|
||||
|
||||
|
||||
@@ -6,11 +6,13 @@
|
||||
|
||||
## Step 5: Run tests (on merged code)
|
||||
|
||||
**Do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
|
||||
Use the project's test commands discovered in Step 4 or documented in CLAUDE.md/AGENTS.md. Run every applicable suite; do not assume Rails or Vitest. The commands below are examples only for repositories that actually provide them. Use the same lane labels and exact commands again in Step 16.
|
||||
|
||||
**For Rails projects using `bin/test-lane`, do NOT run `RAILS_ENV=test bin/rails db:migrate`** — `bin/test-lane` already calls
|
||||
`db:test:prepare` internally, which loads the schema into the correct lane database.
|
||||
Running bare test migrations without INSTANCE hits an orphan DB and corrupts structure.sql.
|
||||
|
||||
Run both test suites in parallel, each wrapped in the evidence ledger. The
|
||||
Run independent test suites in parallel, each wrapped in the evidence ledger. The
|
||||
wrapper is transparent (streams output live, exit code passes through) and
|
||||
records `{command, exit, working-tree fingerprint, log path}` to
|
||||
`~/.gstack/projects/<slug>/<branch>-evidence.jsonl` — Step 16 cites this
|
||||
@@ -22,7 +24,7 @@ record instead of re-running when the content hasn't changed:
|
||||
wait
|
||||
```
|
||||
|
||||
After both complete, check the `gstack-evidence: recorded label=... exit=...
|
||||
After all suites complete, check the `gstack-evidence: recorded label=... exit=...
|
||||
log=...` summary lines — each carries the lane's exit code and a per-run log
|
||||
file (no shared /tmp collisions between concurrent ships). Read the log files
|
||||
for failure detail.
|
||||
@@ -41,6 +43,8 @@ for failure detail.
|
||||
|
||||
Evals are mandatory when prompt-related files change. Skip this step entirely if no prompt files are in the diff.
|
||||
|
||||
Use the project's documented eval selection and pre-merge command first (including changed skill templates and judge/harness code). The Rails patterns and commands below apply only when that runner exists. For other stacks, use their native eval scripts and dependency map. If prompts changed but no eval command is documented, report the missing validation and ask before shipping; never silently treat that as no affected prompts.
|
||||
|
||||
**1. Check if the diff touches prompt-related files:**
|
||||
|
||||
```bash
|
||||
@@ -56,7 +60,7 @@ Match against these patterns (from CLAUDE.md):
|
||||
- `config/system_prompts/*.txt`
|
||||
- `test/evals/**/*` (eval infrastructure changes affect all suites)
|
||||
|
||||
**If no matches:** Print "No prompt-related files changed — skipping evals." and continue to Step 9.
|
||||
**If no matches:** Print "No prompt-related files changed — skipping evals." and continue to Step 7.
|
||||
|
||||
**2. Identify affected eval suites:**
|
||||
|
||||
@@ -102,7 +106,7 @@ poller is reaped.
|
||||
**4. Check results:**
|
||||
|
||||
- **If any eval fails:** Show the failures, the cost dashboard, and **STOP**. Do not proceed.
|
||||
- **If all pass:** Note pass counts and cost. Continue to Step 9.
|
||||
- **If all pass:** Note pass counts and cost. Continue to Step 7.
|
||||
|
||||
**5. Save eval output** — include eval results and cost dashboard in the PR body (Step 19).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user