CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-15 01:15:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: repair frontier eval budgets and workflow instructions

Browse Source 
Preserve frontier models and quality thresholds while fixing truncated judge output, ordered section expansion, consent checks, QA scoring, and ship audit gates. Add regression coverage and refresh generated docs.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
This commit is contained in:
Garry Tan
2026-09-09 04:11:19 +00:00
co-authored by OpenAI Codex
parent 0cbccc1c6c
commit a9f9ec5f08
43 changed files with 916 additions and 518 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ship/sections/plan-completion.md
+26 -22
View File
@@ -8,9 +8,10 @@
**Subagent prompt:** Pass these instructions to the subagent:
> You are running a ship-workflow plan completion audit. The base branch is `<base>`. Use `git diff <base>...HEAD` to see what shipped. Do not commit or push — report only.
>
> ### Plan File Discovery
````text
You are running a ship-workflow plan completion audit. The base branch is `<base>`. Use `git diff <base>...HEAD` to see what shipped. Do not commit or push. Report only: classify every item, but do not execute Gate Logic, ask the user, or advance the workflow. The parent applies those gates to your report.
### Plan File Discovery
1. **Conversation context (primary):** Check if there is an active plan file in this conversation. The host agent's system messages include plan file paths when in plan mode. If found, use it directly — this is the most reliable signal.
@@ -130,13 +131,30 @@ Plan: {plan file path}
[UNVERIFIABLE] Supabase auth allowlist contains user email — external system, confirm in Supabase dashboard
─────────────────────────────────
COMPLETION: 5/9 DONE, 1 PARTIAL, 1 NOT DONE, 1 CHANGED, 2 UNVERIFIABLE
COMPLETION: 4/10 DONE, 1 PARTIAL, 2 NOT DONE, 1 CHANGED, 2 UNVERIFIABLE
─────────────────────────────────
```
After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
{"total_items":N,"done":N,"changed":N,"partial":N,"not_done":N,"unverifiable":N,"summary":"<markdown checklist for PR body>"}
Counts map one-to-one to the classifications above and sum to total_items. No plan or no actionable items means all counts are zero with the skip reason in summary. Do not classify work as deferred; only the parent can record a user-approved deferral.
````
**Parent processing:**
1. Parse the LAST line of the subagent's output as JSON.
2. Store the counts for Step 20 metrics; use `summary` in PR body.
3. Apply Gate Logic below to `not_done` and `unverifiable` before continuing. Track user-approved deferrals separately; `partial` items receive a PR note, not the NOT DONE gate.
4. Embed `summary` in PR body's `## Plan Completion` section (Step 19). For the UNVERIFIABLE gate, also embed `## Plan Completion — Manual Verifications` with each Y response's evidence and each D response's dropped item.
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never races the fallback):** Fall back to running the audit inline (parent processes the same plan-extraction + classification logic). If the inline fallback also fails (e.g., plan file unreadable, parser error), do NOT silently pass — surface the failure as an explicit AskUserQuestion: "Plan Completion audit could not run ({reason}). Options: (A) Skip audit and ship anyway — record that the audit was skipped in PR body and Step 20 metrics; (B) Stop and fix the audit." Default and recommended option is (B). Silent fail-open is the failure shape that VAS-449 surfaced.
---
### Gate Logic
After producing the completion checklist, evaluate in priority order:
The parent evaluates the completion checklist in priority order, including after an inline fallback:
1. **Any NOT DONE items** (highest priority — known missing work). Use AskUserQuestion:
- Show the completion checklist above
@@ -144,10 +162,10 @@ After producing the completion checklist, evaluate in priority order:
- RECOMMENDATION: depends on item count and severity. If 1-2 minor items (docs, config), recommend B. If core functionality is missing, recommend A.
- Options:
A) Stop — implement the missing items before shipping
B) Ship anyway — defer these to a follow-up (will create P1 TODOs in Step 5.5)
B) Ship anyway — defer these to a follow-up (will create P1 TODOs in Step 14)
C) These items were intentionally dropped — remove from scope
- If A: STOP. List the missing items for the user to implement.
- If B: Continue. For each NOT DONE item, create a P1 TODO in Step 5.5 with "Deferred from plan: {plan file path}".
- If B: Continue. For each NOT DONE item, create a P1 TODO in Step 14 with "Deferred from plan: {plan file path}".
- If C: Continue. Note in PR body: "Plan items intentionally dropped: {list}."
2. **Any UNVERIFIABLE items** (silent gaps — the diff cannot prove them either way). Only fires after NOT DONE is resolved or absent.
@@ -174,21 +192,7 @@ After producing the completion checklist, evaluate in priority order:
**No plan file found:** Skip entirely. "No plan file detected — skipping plan completion audit."
**Include in PR body (Step 8):** Add a `## Plan Completion` section with the checklist summary.
>
> After your analysis, output a single JSON object on the LAST LINE of your response (no other text after it):
> `{"total_items":N,"done":N,"changed":N,"deferred":N,"unverifiable":N,"summary":"<markdown checklist for PR body>"}`
**Parent processing:**
1. Parse the LAST line of the subagent's output as JSON.
2. Store `done`, `deferred`, `unverifiable` for Step 20 metrics; use `summary` in PR body.
3. If `deferred > 0` or `unverifiable > 0` and no user override, present the items via the appropriate AskUserQuestion (see Gate Logic priority order above) before continuing.
4. Embed `summary` in PR body's `## Plan Completion` section (Step 19). If `unverifiable > 0` and the user picked option A in the UNVERIFIABLE gate, also embed `## Plan Completion — Manual Verifications` listing each user-confirmed item.
**If the subagent fails, returns invalid JSON, or never completes (backgrounded despite the flag, or no final output after ~10 minutes — stop waiting; if a backgrounded task is still running, stop it first so a late result never races the fallback):** Fall back to running the audit inline (parent processes the same plan-extraction + classification logic). If the inline fallback also fails (e.g., plan file unreadable, parser error), do NOT silently pass — surface the failure as an explicit AskUserQuestion: "Plan Completion audit could not run ({reason}). Options: (A) Skip audit and ship anyway — record that the audit was skipped in PR body and Step 20 metrics; (B) Stop and fix the audit." Default and recommended option is (B). Silent fail-open is the failure shape that VAS-449 surfaced.
---
**Include in PR body (Step 19):** Add a `## Plan Completion` section with the checklist summary.
## Step 8.1: Plan Verification