mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-20 11:52:20 +02:00
fix: resolve workflow gaps exposed by frontier evals
Clarify plan-review ordering and fallback modes, preserve deploy readiness gates, honor configured merge methods, correct benchmark and canary contracts, and restore vendored installs on setup failure. Cover recovery with real-shell regressions. Co-Authored-By: OpenAI Codex <noreply@openai.com>
This commit is contained in:
Garry Tan
co-authored by
OpenAI Codex
OpenAI Codex
parent
a9f9ec5f08
commit
1f678a5b81
@@ -671,7 +671,7 @@ If none was produced (user may have cancelled), proceed with standard review.
|
||||
Before reviewing anything, answer these questions:
|
||||
1. **What existing code already partially or fully solves each sub-problem?** Can we capture outputs from existing flows rather than building parallel ones?
|
||||
2. **What is the minimum set of changes that achieves the stated goal?** Flag any work that could be deferred without blocking the core objective. Be ruthless about scope creep.
|
||||
3. **Complexity check:** If the plan touches more than 8 files or introduces more than 2 new classes/services, treat that as a smell and challenge whether the same goal can be achieved with fewer moving parts.
|
||||
3. **Complexity check:** If the plan touches 8+ files or introduces 2+ new classes/services, treat that as a smell and challenge whether the same goal can be achieved with fewer moving parts.
|
||||
4. **Search check:** For each architectural pattern, infrastructure component, or concurrency approach the plan introduces, research through Aside (Web research runs in Aside, above), one read-only request per pattern:
|
||||
- Does the runtime/framework have a built-in? Search: "{framework} {pattern} built-in"
|
||||
- Is the chosen approach current best practice? Search: "{pattern} best practice {current year}"
|
||||
@@ -687,9 +687,9 @@ Before reviewing anything, answer these questions:
|
||||
If the plan rolls a custom solution where a built-in exists, flag it as a scope reduction opportunity. Annotate recommendations with **[Layer 1]**, **[Layer 2]**, **[Layer 3]**, or **[EUREKA]** (see preamble's Search Before Building section). If you find a eureka moment — a reason the standard approach is wrong for this case — present it as an architectural insight.
|
||||
5. **TODOS cross-reference:** Read `TODOS.md` if it exists. Are any deferred items blocking this plan? Can any deferred items be bundled into this PR without expanding scope? Does this plan create new work that should be captured as a TODO?
|
||||
|
||||
5. **Completeness check:** Is the plan doing the complete version or a shortcut? With AI-assisted coding, the cost of completeness (100% test coverage, full edge case handling, complete error paths) is 10-100x cheaper than with a human team. If the plan proposes a shortcut that saves human-hours but only saves minutes with CC+gstack, recommend the complete version. Boil the ocean.
|
||||
6. **Completeness check:** Is the plan doing the complete version or a shortcut? With AI-assisted coding, the cost of completeness (100% test coverage, full edge case handling, complete error paths) is 10-100x cheaper than with a human team. If the plan proposes a shortcut that saves human-hours but only saves minutes with CC+gstack, recommend the complete version. Boil the ocean.
|
||||
|
||||
6. **Distribution check:** If the plan introduces a new artifact type (CLI binary, library package, container image, mobile app), does it include the build/publish pipeline? Code without distribution is code nobody can use. Check:
|
||||
7. **Distribution check:** If the plan introduces a new artifact type (CLI binary, library package, container image, mobile app), does it include the build/publish pipeline? Code without distribution is code nobody can use. Check:
|
||||
- Is there a CI/CD workflow for building and publishing the artifact?
|
||||
- Are target platforms defined (linux/darwin/windows, amd64/arm64)?
|
||||
- How will users download or install it (GitHub Releases, package manager, container registry)?
|
||||
@@ -699,7 +699,7 @@ If the complexity check triggers (8+ files or 2+ new classes/services), STOP bef
|
||||
|
||||
**STOP.** Do NOT proceed to Section 1 (Architecture review), edit the plan file with a proposed scope reduction, or call ExitPlanMode until the user responds. Naming the 80% solution in chat prose and continuing — or loading the AskUserQuestion schema via ToolSearch and then never invoking it — is the failure mode this gate exists to prevent.
|
||||
|
||||
If the complexity check does not trigger, present your Step 0 findings and proceed directly to Section 1.
|
||||
If the complexity check does not trigger, present your Step 0 findings and enter Review Sections: run Prior Learnings and Confidence Calibration, then Section 1.
|
||||
|
||||
Always work through the full interactive review: one section at a time (Architecture → Code Quality → Tests → Performance) with at most 8 top issues per section.
|
||||
|
||||
|
||||
@@ -126,7 +126,7 @@ If a design doc exists, read it. Use it as the source of truth for the problem s
|
||||
Before reviewing anything, answer these questions:
|
||||
1. **What existing code already partially or fully solves each sub-problem?** Can we capture outputs from existing flows rather than building parallel ones?
|
||||
2. **What is the minimum set of changes that achieves the stated goal?** Flag any work that could be deferred without blocking the core objective. Be ruthless about scope creep.
|
||||
3. **Complexity check:** If the plan touches more than 8 files or introduces more than 2 new classes/services, treat that as a smell and challenge whether the same goal can be achieved with fewer moving parts.
|
||||
3. **Complexity check:** If the plan touches 8+ files or introduces 2+ new classes/services, treat that as a smell and challenge whether the same goal can be achieved with fewer moving parts.
|
||||
4. **Search check:** For each architectural pattern, infrastructure component, or concurrency approach the plan introduces, research through Aside (Web research runs in Aside, above), one read-only request per pattern:
|
||||
- Does the runtime/framework have a built-in? Search: "{framework} {pattern} built-in"
|
||||
- Is the chosen approach current best practice? Search: "{pattern} best practice {current year}"
|
||||
@@ -142,9 +142,9 @@ Before reviewing anything, answer these questions:
|
||||
If the plan rolls a custom solution where a built-in exists, flag it as a scope reduction opportunity. Annotate recommendations with **[Layer 1]**, **[Layer 2]**, **[Layer 3]**, or **[EUREKA]** (see preamble's Search Before Building section). If you find a eureka moment — a reason the standard approach is wrong for this case — present it as an architectural insight.
|
||||
5. **TODOS cross-reference:** Read `TODOS.md` if it exists. Are any deferred items blocking this plan? Can any deferred items be bundled into this PR without expanding scope? Does this plan create new work that should be captured as a TODO?
|
||||
|
||||
5. **Completeness check:** Is the plan doing the complete version or a shortcut? With AI-assisted coding, the cost of completeness (100% test coverage, full edge case handling, complete error paths) is 10-100x cheaper than with a human team. If the plan proposes a shortcut that saves human-hours but only saves minutes with CC+gstack, recommend the complete version. Boil the ocean.
|
||||
6. **Completeness check:** Is the plan doing the complete version or a shortcut? With AI-assisted coding, the cost of completeness (100% test coverage, full edge case handling, complete error paths) is 10-100x cheaper than with a human team. If the plan proposes a shortcut that saves human-hours but only saves minutes with CC+gstack, recommend the complete version. Boil the ocean.
|
||||
|
||||
6. **Distribution check:** If the plan introduces a new artifact type (CLI binary, library package, container image, mobile app), does it include the build/publish pipeline? Code without distribution is code nobody can use. Check:
|
||||
7. **Distribution check:** If the plan introduces a new artifact type (CLI binary, library package, container image, mobile app), does it include the build/publish pipeline? Code without distribution is code nobody can use. Check:
|
||||
- Is there a CI/CD workflow for building and publishing the artifact?
|
||||
- Are target platforms defined (linux/darwin/windows, amd64/arm64)?
|
||||
- How will users download or install it (GitHub Releases, package manager, container registry)?
|
||||
@@ -154,7 +154,7 @@ If the complexity check triggers (8+ files or 2+ new classes/services), STOP bef
|
||||
|
||||
**STOP.** Do NOT proceed to Section 1 (Architecture review), edit the plan file with a proposed scope reduction, or call ExitPlanMode until the user responds. Naming the 80% solution in chat prose and continuing — or loading the AskUserQuestion schema via ToolSearch and then never invoking it — is the failure mode this gate exists to prevent.
|
||||
|
||||
If the complexity check does not trigger, present your Step 0 findings and proceed directly to Section 1.
|
||||
If the complexity check does not trigger, present your Step 0 findings and enter Review Sections: run Prior Learnings and Confidence Calibration, then Section 1.
|
||||
|
||||
Always work through the full interactive review: one section at a time (Architecture → Code Quality → Tests → Performance) with at most 8 top issues per section.
|
||||
|
||||
|
||||
@@ -44,20 +44,7 @@ matches a past learning, display:
|
||||
This makes the compounding visible. The user should see that gstack is getting
|
||||
smarter on their codebase over time.
|
||||
|
||||
### 1. Architecture review
|
||||
Evaluate:
|
||||
* Overall system design and component boundaries.
|
||||
* Dependency graph and coupling concerns.
|
||||
* Data flow patterns and potential bottlenecks.
|
||||
* Scaling characteristics and single points of failure.
|
||||
* Security architecture (auth, data access, API boundaries).
|
||||
* Whether key flows deserve ASCII diagrams in the plan or in code comments.
|
||||
* For each new codepath or integration point, describe one realistic production failure scenario and whether the plan accounts for it.
|
||||
* **Distribution architecture:** If this introduces a new artifact (binary, package, container), how does it get built, published, and updated? Is the CI/CD pipeline part of the plan or deferred?
|
||||
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly.
|
||||
|
||||
**STOP.** Do NOT proceed to the next review section, edit the plan file with the proposed fix, or call ExitPlanMode until the user responds. An issue with an "obvious fix" is still an issue and still needs explicit user approval before it lands in the plan. Loading the AskUserQuestion schema via ToolSearch and then writing the recommendation as chat prose is the failure mode this gate exists to prevent.
|
||||
**Plan-review evidence:** Apply the calibration gate below before Section 1. For proposed work, quote the motivating plan requirement (plan file:line); verify it against existing interfaces where applicable. Do not require nonexistent future code or describe a proposed regression as an observed one. Code-specific examples apply when critiquing existing code. Put suppressed findings in a `Suppressed findings` appendix to the review report.
|
||||
|
||||
## Confidence Calibration
|
||||
|
||||
@@ -121,6 +108,21 @@ confirms it IS a real issue, that is a calibration event. Your initial confidenc
|
||||
too low. Log the corrected pattern as a learning so future reviews catch it with
|
||||
higher confidence.
|
||||
|
||||
### 1. Architecture review
|
||||
Evaluate:
|
||||
* Overall system design and component boundaries.
|
||||
* Dependency graph and coupling concerns.
|
||||
* Data flow patterns and potential bottlenecks.
|
||||
* Scaling characteristics and single points of failure.
|
||||
* Security architecture (auth, data access, API boundaries).
|
||||
* Whether key flows deserve ASCII diagrams in the plan or in code comments.
|
||||
* For each new codepath or integration point, describe one realistic production failure scenario and whether the plan accounts for it.
|
||||
* **Distribution architecture:** If this introduces a new artifact (binary, package, container), how does it get built, published, and updated? Is the CI/CD pipeline part of the plan or deferred?
|
||||
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly.
|
||||
|
||||
**STOP.** Do NOT proceed to the next review section, edit the plan file with the proposed fix, or call ExitPlanMode until the user responds. An issue with an "obvious fix" is still an issue and still needs explicit user approval before it lands in the plan. Loading the AskUserQuestion schema via ToolSearch and then writing the recommendation as chat prose is the failure mode this gate exists to prevent.
|
||||
|
||||
### 2. Code quality review
|
||||
Evaluate:
|
||||
* Code organization and module structure.
|
||||
@@ -387,12 +389,14 @@ Branch on the echoed `CODEX_MODE`:
|
||||
- **`model_unusable`** — authed but the account cannot use gstack's selected Codex model (#2477: HTTP 400 on every call). Relay the probe's HINT lines, tell the user the one-line fix (set `GSTACK_CODEX_MODEL=<supported-model>` or pass an explicit `-c model=...` override), and fall back to the Claude subagent path. The ~10s round trip is cached for 1h; timeouts fail open to `ready`.
|
||||
- **`ready`** — run the Codex pass below.
|
||||
|
||||
When the mode is `ready`, `not_installed`, or `not_authed`, print one line so the off-switch
|
||||
On `under_codex`, no in-host substitute is defined here: skip this outside-voice section and continue to the required outputs. Do not invoke Codex again or label a self-review as independent.
|
||||
|
||||
For all other non-disabled modes (`ready`, `not_installed`, `not_authed`, `broken_install`, `model_unusable`), print one line so the off-switch
|
||||
stays discoverable: "Running the outside voice automatically (standard step). Disable: `gstack-config set codex_reviews disabled`."
|
||||
|
||||
**Construct the plan review prompt** (for `ready`, `not_installed`, and `not_authed` — skip only on `disabled`).
|
||||
**Construct the plan review prompt** for every remaining mode, including all Claude fallback modes (skip on `disabled` or `under_codex`).
|
||||
Read the plan file being reviewed (the file the user pointed this review at, or the branch
|
||||
diff scope). If a CEO plan document was written in Step 0D-POST, read that too — it contains
|
||||
diff scope). If a CEO plan document from an earlier `/plan-ceo-review` Step 0D-POST is available, read that too — it contains
|
||||
the scope decisions and vision.
|
||||
|
||||
Construct this prompt (substitute the actual plan content — if plan content exceeds 30KB,
|
||||
@@ -438,7 +442,7 @@ CODEX SAYS (plan review — outside voice):
|
||||
- Timeout: "Codex timed out after 5 minutes." Fall back to the Claude subagent below.
|
||||
- Empty response: "Codex returned no response." Fall back to the Claude subagent below.
|
||||
|
||||
**If `CODEX_MODE: not_installed` or `not_authed` (or Codex errored at runtime):**
|
||||
**If `CODEX_MODE: not_installed`, `not_authed`, `broken_install`, or `model_unusable` (or Codex errored at runtime):**
|
||||
|
||||
Dispatch via the Agent tool with `run_in_background: false` (subagents default to background since Claude Code v2.1.198; the findings must land before the workflow continues). The subagent has fresh context and no conversation bias — but it is the SAME model family, not an outside model; weigh its agreement accordingly.
|
||||
Bound it the same way as Codex: cap the dispatch at a 5-minute timeout so "never blocking"
|
||||
@@ -947,4 +951,3 @@ Use AskUserQuestion with only the applicable options:
|
||||
|
||||
## Unresolved decisions
|
||||
If the user does not respond to an AskUserQuestion or interrupts to move on, note which decisions were left unresolved. At the end of the review, list these as "Unresolved decisions that may bite you later" — never silently default to an option.
|
||||
|
||||
|
||||
@@ -6,6 +6,10 @@
|
||||
|
||||
{{LEARNINGS_SEARCH}}
|
||||
|
||||
**Plan-review evidence:** Apply the calibration gate below before Section 1. For proposed work, quote the motivating plan requirement (plan file:line); verify it against existing interfaces where applicable. Do not require nonexistent future code or describe a proposed regression as an observed one. Code-specific examples apply when critiquing existing code. Put suppressed findings in a `Suppressed findings` appendix to the review report.
|
||||
|
||||
{{CONFIDENCE_CALIBRATION}}
|
||||
|
||||
### 1. Architecture review
|
||||
Evaluate:
|
||||
* Overall system design and component boundaries.
|
||||
@@ -21,8 +25,6 @@ For each issue found in this section, call AskUserQuestion individually. One iss
|
||||
|
||||
**STOP.** Do NOT proceed to the next review section, edit the plan file with the proposed fix, or call ExitPlanMode until the user responds. An issue with an "obvious fix" is still an issue and still needs explicit user approval before it lands in the plan. Loading the AskUserQuestion schema via ToolSearch and then writing the recommendation as chat prose is the failure mode this gate exists to prevent.
|
||||
|
||||
{{CONFIDENCE_CALIBRATION}}
|
||||
|
||||
### 2. Code quality review
|
||||
Evaluate:
|
||||
* Code organization and module structure.
|
||||
@@ -222,4 +224,3 @@ Use AskUserQuestion with only the applicable options:
|
||||
|
||||
## Unresolved decisions
|
||||
If the user does not respond to an AskUserQuestion or interrupts to move on, note which decisions were left unresolved. At the end of the review, list these as "Unresolved decisions that may bite you later" — never silently default to an option.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user