mirror of
https://github.com/garrytan/gstack.git
synced 2026-06-17 15:20:11 +02:00
fix: delete AskUserQuestion fallback (root cause of forever war)
The /plan-eng-review skill failed to fire AskUserQuestion on a real plan review and surfaced 4 calibration decisions via prose instead. Investigation traced this to a "fallback when neither variant is callable" clause in the preamble that the model rationalizes around as a general escape hatch from "fanning out round-trip AUQs," even when an AUQ variant IS callable. Codex review confirmed the fallback exists in 8 inline sites with 2 surviving escape hatches the original narrowing missed (a "genuinely trivial" exception duplicated across all 4 plan-* templates, and a "outside plan mode, output as prose and stop" branch in the preamble itself). Net deletion in skill text. Closes both branches of the deleted fallback (plan-file write AND prose-and-stop) and the trivial-fix exception with a single hard rule: If no AskUserQuestion variant appears in your tool list, this skill is BLOCKED. Stop, report `BLOCKED — AskUserQuestion unavailable`, and wait for the user. Honest about being a model directive, not a runtime guard — none of the PTY harness helpers enforce BLOCKED today. The architectural improvement is that the model has fewer alternatives to obey it against. Runtime enforcement is a follow-up TODO. Sources changed: - scripts/resolvers/preamble/generate-ask-user-format.ts: delete both fallback branches; replace with 1-line BLOCKED rule - scripts/resolvers/preamble/generate-completion-status.ts: delete fallback in generatePlanModeInfo - plan-eng-review/SKILL.md.tmpl: delete fallback at Step 0 + Sections 1-4 (5 instances) + delete trivial-fix exception - office-hours/SKILL.md.tmpl: delete fallback in approach-selection - plan-ceo-review/SKILL.md.tmpl: delete trivial-fix exception - plan-design-review/SKILL.md.tmpl: delete trivial-fix exception - plan-devex-review/SKILL.md.tmpl: delete trivial-fix exception Generated SKILL.md regen lands in a follow-up commit per the bisect convention (template changes separate from regenerated output). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -440,7 +440,7 @@ Rules:
|
||||
|
||||
**RECOMMENDATION:** Choose [X] because [one-line reason mapped to the founder's stated goal].
|
||||
|
||||
Emit ONE AskUserQuestion that lists every alternative (A/B and optionally C) as numbered options, using the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — write the question text and call the tool. If no AskUserQuestion variant is callable in this session, follow the preamble's "Tool resolution" fallback: in plan mode, write `## Decisions to confirm` into the plan file and ExitPlanMode; outside plan mode, output the decision brief as prose and stop. Never silently auto-decide.
|
||||
Emit ONE AskUserQuestion that lists every alternative (A/B and optionally C) as numbered options, using the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — write the question text and call the tool.
|
||||
|
||||
**STOP.** Do NOT proceed to Phase 4.5 (Founder Signal Synthesis), Phase 5 (Design Doc), Phase 6 (Closing), or any design-doc generation until the user responds. A "clearly winning approach" is still an approach decision and still needs explicit user approval before it lands in the design doc. Writing the recommendation in chat prose and continuing forward is the failure mode this gate exists to prevent.
|
||||
|
||||
|
||||
@@ -681,7 +681,7 @@ Follow the AskUserQuestion format from the Preamble above. Additional rules for
|
||||
* For each option: effort, risk, and maintenance burden in one line.
|
||||
* **Map the reasoning to my engineering preferences above.** One sentence connecting your recommendation to a specific preference.
|
||||
* Label with issue NUMBER + option LETTER (e.g., "3A", "3B").
|
||||
* **Escape hatch (tightened):** If a section has zero findings, state "No issues, moving on" and proceed. If it has findings, use AskUserQuestion for each — a finding with an "obvious fix" is still a finding and still needs user approval before any change lands in the plan. Only skip AskUserQuestion when the decision is genuinely trivial (e.g., a typo fix) AND there are no meaningful alternatives. When in doubt, ask.
|
||||
* **Zero findings:** if a section has zero findings, state "No issues, moving on" and proceed. Otherwise, use AskUserQuestion for each finding — a finding with an "obvious fix" is still a finding and still needs user approval before any change lands in the plan.
|
||||
|
||||
## Required Outputs
|
||||
|
||||
|
||||
@@ -348,7 +348,7 @@ Follow the AskUserQuestion format from the Preamble above. Additional rules for
|
||||
* Present 2-3 options. For each: effort to specify now, risk if deferred.
|
||||
* **Map to Design Principles above.** One sentence connecting your recommendation to a specific principle.
|
||||
* Label with issue NUMBER + option LETTER (e.g., "3A", "3B").
|
||||
* **Escape hatch (tightened):** If a section has zero findings, state "No issues, moving on" and proceed. If it has findings, use AskUserQuestion for each — a gap with an "obvious fix" is still a gap and still needs user approval before any change lands in the plan. Only skip AskUserQuestion when the fix is genuinely trivial AND there are no meaningful design alternatives. When in doubt, ask.
|
||||
* **Zero findings:** if a section has zero findings, state "No issues, moving on" and proceed. Otherwise, use AskUserQuestion for each gap — a gap with an "obvious fix" is still a gap and still needs user approval before any change lands in the plan.
|
||||
* **NEVER use AskUserQuestion to ask which variant the user prefers.** Always create a comparison board first (`$D compare --serve`) and open it in the browser. The board has rating controls, comments, remix/regenerate buttons, and structured feedback output. Use AskUserQuestion ONLY to notify the user the board is open and wait for them to finish — not to present variants inline and ask "which do you prefer?" That is a degraded experience.
|
||||
|
||||
## Required Outputs
|
||||
|
||||
@@ -669,11 +669,10 @@ DX reviews:
|
||||
* **Map to DX First Principles above.** One sentence connecting your recommendation
|
||||
to a specific principle (e.g., "This violates 'zero friction at T0' because
|
||||
[persona] needs 3 extra config steps before their first API call").
|
||||
* **Escape hatch (tightened):** If a section has zero findings, state "No issues,
|
||||
moving on" and proceed. If it has findings, use AskUserQuestion for each — a
|
||||
gap with an "obvious fix" is still a gap and still needs user approval before
|
||||
any change lands in the plan. Only skip AskUserQuestion when the fix is
|
||||
genuinely trivial AND there are no meaningful DX alternatives. When in doubt, ask.
|
||||
* **Zero findings:** if a section has zero findings, state "No issues, moving on"
|
||||
and proceed. Otherwise, use AskUserQuestion for each gap — a gap with an
|
||||
"obvious fix" is still a gap and still needs user approval before any change
|
||||
lands in the plan.
|
||||
* Assume the user hasn't looked at this window in 20 minutes. Re-ground every question.
|
||||
|
||||
## Required Outputs
|
||||
|
||||
@@ -113,7 +113,7 @@ Before reviewing anything, answer these questions:
|
||||
- How will users download or install it (GitHub Releases, package manager, container registry)?
|
||||
If the plan defers distribution, flag it explicitly in the "NOT in scope" section — don't let it silently drop.
|
||||
|
||||
If the complexity check triggers (8+ files or 2+ new classes/services), STOP before any review-section work. Call AskUserQuestion: name what's overbuilt, propose a minimal version that achieves the core goal, ask whether to reduce or proceed as-is. The AskUserQuestion call is a tool_use, not prose — call the tool directly. If no AskUserQuestion variant is callable, follow the preamble's "Tool resolution" fallback: in plan mode, write `## Decisions to confirm` into the plan file and ExitPlanMode; outside plan mode, output the decision brief as prose and stop. Never silently auto-decide.
|
||||
If the complexity check triggers (8+ files or 2+ new classes/services), STOP before any review-section work. Call AskUserQuestion: name what's overbuilt, propose a minimal version that achieves the core goal, ask whether to reduce or proceed as-is. The AskUserQuestion call is a tool_use, not prose — call the tool directly.
|
||||
|
||||
**STOP.** Do NOT proceed to Section 1 (Architecture review), edit the plan file with a proposed scope reduction, or call ExitPlanMode until the user responds. Naming the 80% solution in chat prose and continuing — or loading the AskUserQuestion schema via ToolSearch and then never invoking it — is the failure mode this gate exists to prevent.
|
||||
|
||||
@@ -142,7 +142,7 @@ Evaluate:
|
||||
* For each new codepath or integration point, describe one realistic production failure scenario and whether the plan accounts for it.
|
||||
* **Distribution architecture:** If this introduces a new artifact (binary, package, container), how does it get built, published, and updated? Is the CI/CD pipeline part of the plan or deferred?
|
||||
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly. If no AskUserQuestion variant is callable in this session, follow the preamble's "Tool resolution" fallback: in plan mode, write `## Decisions to confirm` into the plan file and ExitPlanMode; outside plan mode, output the decision brief as prose and stop. Never silently auto-decide.
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly.
|
||||
|
||||
**STOP.** Do NOT proceed to the next review section, edit the plan file with the proposed fix, or call ExitPlanMode until the user responds. An issue with an "obvious fix" is still an issue and still needs explicit user approval before it lands in the plan. Loading the AskUserQuestion schema via ToolSearch and then writing the recommendation as chat prose is the failure mode this gate exists to prevent.
|
||||
|
||||
@@ -157,7 +157,7 @@ Evaluate:
|
||||
* Areas that are over-engineered or under-engineered relative to my preferences.
|
||||
* Existing ASCII diagrams in touched files — are they still accurate after this change?
|
||||
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly. If no AskUserQuestion variant is callable in this session, follow the preamble's "Tool resolution" fallback: in plan mode, write `## Decisions to confirm` into the plan file and ExitPlanMode; outside plan mode, output the decision brief as prose and stop. Never silently auto-decide.
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly.
|
||||
|
||||
**STOP.** Do NOT proceed to the next review section, edit the plan file with the proposed fix, or call ExitPlanMode until the user responds. An issue with an "obvious fix" is still an issue and still needs explicit user approval before it lands in the plan. Loading the AskUserQuestion schema via ToolSearch and then writing the recommendation as chat prose is the failure mode this gate exists to prevent.
|
||||
|
||||
@@ -167,7 +167,7 @@ For each issue found in this section, call AskUserQuestion individually. One iss
|
||||
|
||||
For LLM/prompt changes: check the "Prompt/LLM changes" file patterns listed in CLAUDE.md. If this plan touches ANY of those patterns, state which eval suites must be run, which cases should be added, and what baselines to compare against. Then use AskUserQuestion to confirm the eval scope with the user.
|
||||
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly. If no AskUserQuestion variant is callable in this session, follow the preamble's "Tool resolution" fallback: in plan mode, write `## Decisions to confirm` into the plan file and ExitPlanMode; outside plan mode, output the decision brief as prose and stop. Never silently auto-decide.
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly.
|
||||
|
||||
**STOP.** Do NOT proceed to the next review section, edit the plan file with the proposed fix, or call ExitPlanMode until the user responds. An issue with an "obvious fix" is still an issue and still needs explicit user approval before it lands in the plan. Loading the AskUserQuestion schema via ToolSearch and then writing the recommendation as chat prose is the failure mode this gate exists to prevent.
|
||||
|
||||
@@ -178,7 +178,7 @@ Evaluate:
|
||||
* Caching opportunities.
|
||||
* Slow or high-complexity code paths.
|
||||
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly. If no AskUserQuestion variant is callable in this session, follow the preamble's "Tool resolution" fallback: in plan mode, write `## Decisions to confirm` into the plan file and ExitPlanMode; outside plan mode, output the decision brief as prose and stop. Never silently auto-decide.
|
||||
For each issue found in this section, call AskUserQuestion individually. One issue per call. Present options, state your recommendation, explain WHY. Do NOT batch multiple issues into one AskUserQuestion. Use the preamble's AskUserQuestion Format section. The AskUserQuestion call is a tool_use, not prose — call the tool directly.
|
||||
|
||||
**STOP.** Do NOT proceed to the next review section, edit the plan file with the proposed fix, or call ExitPlanMode until the user responds. An issue with an "obvious fix" is still an issue and still needs explicit user approval before it lands in the plan. Loading the AskUserQuestion schema via ToolSearch and then writing the recommendation as chat prose is the failure mode this gate exists to prevent.
|
||||
|
||||
@@ -201,7 +201,7 @@ Follow the AskUserQuestion format from the Preamble above. Additional rules for
|
||||
* **Map the reasoning to my engineering preferences above.** One sentence connecting your recommendation to a specific preference (DRY, explicit > clever, minimal diff, etc.).
|
||||
* Label with issue NUMBER + option LETTER (e.g., "3A", "3B").
|
||||
* **Coverage vs kind:** for every per-issue AskUserQuestion you raise in this review, decide whether the options differ in coverage or in kind. If coverage (e.g., more tests vs fewer, complete error handling vs happy-path-only, full edge-case coverage vs shortcut), include `Completeness: N/10` on each option. If kind (e.g., architectural choice between two different systems, posture-over-posture, A/B/C where each is a different kind of thing), skip the score and add one line: `Note: options differ in kind, not coverage — no completeness score.` Do NOT fabricate scores on kind-differentiated questions — filler scores are worse than no score.
|
||||
* **Escape hatch (tightened):** If a section has zero findings, state "No issues, moving on" and proceed. If it has findings, use AskUserQuestion for each — a finding with an "obvious fix" is still a finding and still needs user approval before any change lands in the plan. Only skip AskUserQuestion when the decision is genuinely trivial (e.g., a typo fix) AND there are no meaningful alternatives. When in doubt, ask.
|
||||
* **Zero findings:** if a section has zero findings, state "No issues, moving on" and proceed. Otherwise, use AskUserQuestion for each finding — a finding with an "obvious fix" is still a finding and still needs user approval before any change lands in the plan.
|
||||
|
||||
## Required outputs
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ export function generateAskUserFormat(_ctx: TemplateContext): string {
|
||||
|
||||
**Rule:** if any \`mcp__*__AskUserQuestion\` variant is in your tool list, prefer it. Hosts may disable native AUQ via \`--disallowedTools AskUserQuestion\` (Conductor does, by default) and route through their MCP variant; calling native there silently fails. Same questions/options shape; same decision-brief format applies.
|
||||
|
||||
**Fallback when neither variant is callable:** in plan mode, write the decision brief into the plan file as a \`## Decisions to confirm\` section + ExitPlanMode (the native "Ready to execute?" surfaces it). Outside plan mode, output the brief as prose and stop. **Never silently auto-decide** — only \`/plan-tune\` AUTO_DECIDE opt-ins authorize auto-picking.
|
||||
**If no AskUserQuestion variant appears in your tool list, this skill is BLOCKED.** Stop, report \`BLOCKED — AskUserQuestion unavailable\`, and wait for the user. Do not write decisions to the plan file as a substitute, do not emit them as prose and stop, and do not silently auto-decide (only \`/plan-tune\` AUTO_DECIDE opt-ins authorize auto-picking).
|
||||
|
||||
### Format
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@ In plan mode, allowed because they inform the plan: \`$B\`, \`$D\`, \`codex exec
|
||||
|
||||
## Skill Invocation During Plan Mode
|
||||
|
||||
If the user invokes a skill in plan mode, the skill takes precedence over generic plan mode behavior. **Treat the skill file as executable instructions, not reference.** Follow it step by step starting from Step 0; the first AskUserQuestion is the workflow entering plan mode, not a violation of it. AskUserQuestion (any variant — \`mcp__*__AskUserQuestion\` or native; see "AskUserQuestion Format → Tool resolution") satisfies plan mode's end-of-turn requirement. If no variant is callable, fall back to writing the decision brief into the plan file as a \`## Decisions to confirm\` section + ExitPlanMode — never silently auto-decide. At a STOP point, stop immediately. Do not continue the workflow or call ExitPlanMode there. Commands marked "PLAN MODE EXCEPTION — ALWAYS RUN" execute. Call ExitPlanMode only after the skill workflow completes, or if the user tells you to cancel the skill or leave plan mode.`;
|
||||
If the user invokes a skill in plan mode, the skill takes precedence over generic plan mode behavior. **Treat the skill file as executable instructions, not reference.** Follow it step by step starting from Step 0; the first AskUserQuestion is the workflow entering plan mode, not a violation of it. AskUserQuestion (any variant — \`mcp__*__AskUserQuestion\` or native; see "AskUserQuestion Format → Tool resolution") satisfies plan mode's end-of-turn requirement. If no variant is callable, the skill is BLOCKED — stop and report \`BLOCKED — AskUserQuestion unavailable\` per the AskUserQuestion Format rule. At a STOP point, stop immediately. Do not continue the workflow or call ExitPlanMode there. Commands marked "PLAN MODE EXCEPTION — ALWAYS RUN" execute. Call ExitPlanMode only after the skill workflow completes, or if the user tells you to cancel the skill or leave plan mode.`;
|
||||
}
|
||||
|
||||
export function generateCompletionStatus(ctx: TemplateContext): string {
|
||||
|
||||
Reference in New Issue
Block a user