fix: repair native workflow evaluation and clarify review steps

This commit is contained in:
Garry Tan committed 2026-09-09 23:27:32 +00:00
1 parent e8470a7e44
commit ca16058341
66 files changed
+4175 -398

No files matched your search

+17 -16
View File
@@ -661,12 +661,12 @@ If ${outsideVoiceFor(ctx).label} is available, use AskUserQuestion:
> A) Yes — get outside design voices
> B) No — proceed without
If user chooses A, launch both voices simultaneously:
If user chooses A, run both independent voices below and wait for both results before synthesis. They may overlap when the host supports parallel tool calls; the native subagent call remains blocking.
1. **${outsideVoiceFor(ctx).label}** (via Bash, \`model_reasoning_effort="medium"\`):
Prompt: "For this product approach, provide: a visual thesis (one sentence — mood, material, energy), a content plan (hero → support → detail → CTA), and 2 interaction ideas that change page feel. Apply beautiful defaults: composition-first, brand-first, cardless, poster not document. Be opinionated." Include the approved product approach and wireframe source in the prepared prompt.
${outsideVoiceInvocation(ctx, { timeoutMs: 300000, reasoningEffort: 'medium' })}
${outsideVoiceInvocation(ctx, { timeoutMs: 300000, reasoningEffort: 'medium', purpose: 'design-direction' })}
${outsideVoiceProvenance(ctx, 'design-sketch')}
@@ -772,7 +772,7 @@ Be bold. Be specific. No hedging.`;
const optInSection = isAutomatic ? `
**Automatic:** Outside voices run automatically when ${outsideVoiceFor(ctx).label} is available. No opt-in needed.` : `
Use AskUserQuestion:
> "Want outside design voices${isPlanDesignReview ? ' before the detailed review' : ''}? ${outsideVoiceFor(ctx).label} evaluates against OpenAI's design hard rules + litmus checks; ${outsideVoiceFor(ctx).nativeLabel} subagent does an independent ${isDesignConsultation ? 'design direction proposal' : 'completeness review'}."
> "Want outside design voices${isPlanDesignReview ? ' before the detailed review' : ''}? ${outsideVoiceFor(ctx).label} ${isDesignConsultation ? 'proposes an independent design direction' : "evaluates against OpenAI's design hard rules + litmus checks"}; ${outsideVoiceFor(ctx).nativeLabel} subagent does an independent ${isDesignConsultation ? 'design direction proposal' : 'completeness review'}."
>
> A) Yes — run outside design voices
> B) No — proceed without
@@ -808,33 +808,35 @@ Fill in each cell from the ${outsideVoiceFor(ctx).label} and subagent outputs. C
- Litmus CONFIRMED failures → pre-loaded as known issues in the relevant pass
- Passes can skip discovery and go straight to fixing for pre-identified issues` :
isDesignConsultation ? `
**Synthesis:** ${outsideVoiceFor(ctx).nativeLabel} main references both ${outsideVoiceFor(ctx).label} and subagent proposals in the Phase 3 proposal. Present:
- Areas of agreement between all three voices (${outsideVoiceFor(ctx).nativeLabel} main + ${outsideVoiceFor(ctx).label} + subagent)
- Genuine divergences as creative alternatives for the user to choose from
- "${outsideVoiceFor(ctx).label} and I agree on X. ${outsideVoiceFor(ctx).label} suggested Y where I'm proposing Z — here's why..."` : `
**Synthesis:** In Phase 3, compare your primary direction with both completed proposals. Show agreements and creative alternatives; explain your recommendation and let the user choose.` : `
**Synthesis — Litmus scorecard:**
Use the same scorecard format as /plan-design-review (shown above). Fill in from both outputs.
Merge findings into the triage with \`[${outsideVoiceFor(ctx).id}]\` / \`[subagent]\` / \`[cross-model]\` tags.`;
return `## Design Outside Voices (parallel)
return `## Design Outside Voices (independent)
${optInSection}
**Check ${outsideVoiceFor(ctx).label} availability:**
${outsideVoicePreflight(ctx, { disabledBehavior: 'opt-in' })}
**If ${outsideVoiceFor(ctx).label} is available**, launch both voices simultaneously:
Declining opt-in skips both voices. Otherwise, non-ready (\`not_installed\`,
\`under_current_harness\`, etc.) means: skip the outside CLI, keep its repair
notice, use the native voice only, and record \`outside_status: unavailable\`
even if the native voice succeeds.
**When ready**, run both voices and await both before synthesis. Overlap calls
if supported; keep the native call blocking.
1. **${outsideVoiceFor(ctx).label} design voice** (via Bash):
Prompt (include the actual plan/product/frontend source context, not only file paths):
"${codexPrompt}"
${outsideVoiceInvocation(ctx, { timeoutMs: 300000, reasoningEffort })}
${outsideVoiceInvocation(ctx, { timeoutMs: 300000, reasoningEffort, ...(isDesignConsultation ? { purpose: 'design-direction' as const } : {}) })}
2. **${outsideVoiceFor(ctx).nativeLabel} design subagent** (via Agent tool, \`run_in_background: false\` — subagents default to background since ${CC_BACKGROUND_DEFAULT_SINCE}):
Dispatch a subagent with this prompt:
2. **${outsideVoiceFor(ctx).nativeLabel} design subagent** (Agent tool, \`run_in_background: false\`; await its result):
"${subagentPrompt}"
**Error handling (all non-blocking):**
@@ -844,15 +846,14 @@ Dispatch a subagent with this prompt:
- On any ${outsideVoiceFor(ctx).label} error: proceed with ${outsideVoiceFor(ctx).nativeLabel} subagent output only, tagged \`[single-model]\`.
- If ${outsideVoiceFor(ctx).nativeLabel} subagent also fails: "Outside voices unavailable — continuing with primary review."
Present ${outsideVoiceFor(ctx).label} output under a \`${outsideVoiceFor(ctx).label.toUpperCase()} SAYS (design ${isPlanDesignReview ? 'critique' : isDesignReview ? 'source audit' : 'direction'}):\` header.
Present subagent output under a \`${outsideVoiceFor(ctx).nativeLabel.toUpperCase()} SUBAGENT (design ${isPlanDesignReview ? 'completeness' : isDesignReview ? 'consistency' : 'direction'}):\` header.
Output headers: \`${outsideVoiceFor(ctx).label.toUpperCase()} SAYS (design ${isPlanDesignReview ? 'critique' : isDesignReview ? 'source audit' : 'direction'}):\` and \`${outsideVoiceFor(ctx).nativeLabel.toUpperCase()} SUBAGENT (design ${isPlanDesignReview ? 'completeness' : isDesignReview ? 'consistency' : 'direction'}):\`.
${synthesisSection}
**Log the result:**
\`\`\`bash
${ctx.paths.binDir}/gstack-review-log '{"skill":"design-outside-voices","timestamp":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'","status":"STATUS","source":"SOURCE","host":"${ctx.host}","outside_provider":"${outsideVoiceFor(ctx).id}","outside_status":"OUTSIDE_STATUS","phase":"design","commit":"'"$(git rev-parse --short HEAD)"'"}'
\`\`\`
Replace STATUS with "clean" only if a reviewer completed with no findings, "issues_found" for findings, or "unavailable" if neither reviewer completed, SOURCE with the completed provider or in-host.
${isDesignConsultation ? 'For proposals, STATUS="clean" requires a complete proposal with no blockers; "issues_found" means concrete concerns; "unavailable" means neither completed. Taste differences are alternatives, not defects.' : 'STATUS="clean" requires a completed review with no findings; use "issues_found" for findings, "unavailable" if neither completed.'} SOURCE is the completed provider or in-host.
${outsideVoiceProvenance(ctx, 'design')}`;
}
@@ -1280,7 +1281,7 @@ the approved variant.
5. Reload the board in the user's browser (same tab) — the URL is per-board
under daemon mode, so use \`<BOARD_URL>\` (from the \`BOARD_URL:\` stderr
line) as the base:
\`curl -s -X POST "\${BOARD_URL}api/reload" -H 'Content-Type: application/json' -d '{"html":"$_DESIGN_DIR/design-board.html"}'\`
\`jq -nc --arg html "$_DESIGN_DIR/design-board.html" '{html: $html}' | curl -sS -X POST "\${BOARD_URL}api/reload" -H 'Content-Type: application/json' --data-binary @-\`
Under \`--no-daemon\` the reload endpoint is \`/api/reload\` at the legacy
port; this path only matters if the caller explicitly opted out of the
daemon.
+7 -3
View File
@@ -103,6 +103,8 @@ export interface OutsideCommandOptions {
diffCommand?: string;
gate?: 'review' | 'structured' | 'spec';
reasoningEffort?: 'high' | 'medium';
/** Creative proposals retain the recommendation gate with task-specific wording. */
purpose?: 'design-direction';
}
/** One self-contained shell body. No shell functions/variables survive between blocks. */
@@ -158,17 +160,19 @@ export function outsideVoiceInvocation(ctx: TemplateContext, opts: OutsideComman
? 'Request exactly SCORE: N (integer 0-10) and AMBIGUITIES: ... (or NONE), as two distinct nonempty lines.'
: opts.gate === 'structured'
? 'Request severity-tagged findings or an explicit NO_FINDINGS conclusion.'
: 'Request a final Recommendation: <action> because <specific reason> line, including an explicit no-findings rationale.';
: opts.purpose === 'design-direction'
? 'Request a complete design proposal ending with Recommendation: <direction> because <product-specific reason>. No defect severity or no-findings conclusion is required.'
: 'Request a final Recommendation: <action> because <specific reason> line, including an explicit no-findings rationale.';
const preparation = nativeStructured
? 'Run Codex’s built-in structured review with the selected base. It supplies its own prompt and accepts no custom prompt file with --base. Require severity-tagged findings (including native P1:/P2: labels) or an explicit no-findings conclusion; arbitrary prose or a refusal is missing coverage.'
: `Write the **complete prompt and required context** to a private temporary file using the Write tool. Do not interpolate user text into shell source. Replace the literal \`<prepared-prompt-file>\` below with its shell-quoted pathname. Include the plan/spec/source content itself when needed: Claude Code review/challenge has no tools and cannot follow paths or execute git. ${completion} A refusal is never completion.`;
: `Use Write to save the **complete prompt and context** in a private file. Replace \`<prepared-prompt-file>\` below with its shell-quoted path; never interpolate user text into shell source. Include actual plan/spec/source content: Claude Code review/challenge has no tools, git, or path access. ${completion} A refusal is never completion.`;
return `${preparation}
\`\`\`bash
${outsideVoiceCommand(ctx, opts)}
\`\`\`
Present the full response inside a \`tool-output\` fence. Only successful execution **and** valid review markers establish completed outside coverage. Refusal, empty/malformed output, missing score/severity/completion markers, timeout, or CLI failure means \`outside_status: unavailable\`. Follow this caller's existing fallback/decision flow; never turn missing coverage into a clean/PASS result. ${nativeStructured ? 'The invocation removes its own scratch directory.' : 'After presentation or failure, delete the private prompt file you created (only that owned temporary file); the invocation already removes its own scratch directory.'}`;
Show the full response in a \`tool-output\` fence. Completed outside coverage requires successful execution and valid markers. Refusal, empty/malformed output, ${opts.purpose === 'design-direction' ? 'missing Recommendation marker' : 'missing score/severity/completion markers'}, timeout, or CLI failure means \`outside_status: unavailable\`. Follow this caller's fallback; missing coverage is never clean/PASS. ${nativeStructured ? 'The invocation removes its own scratch directory.' : 'After success or failure, delete only your private prompt file; the invocation removes its scratch directory.'}`;
}
export function outsideVoiceProvenance(ctx: TemplateContext, phase: string): string {
+13 -3
View File
@@ -706,7 +706,10 @@ Immediately before dispatching, check the preflight result again. On
\`CODEX_MODE: disabled\`, finish this section with \`outside_status: disabled\`;
do not dispatch. Otherwise, use this fallback for missing/broken CLI, failed
authentication/model selection, a failed preflight, or a failed outside invocation.
The disabled branch never reaches this fallback.
The disabled branch never reaches this fallback. In this section, \`under_codex\`
or \`under_current_harness\` also follows this native fallback: run no outside CLI,
report the setup repair and \`outside_status: unavailable\`, then use the native
subagent below. A native result never supplies outside coverage.
Dispatch via the Agent tool with \`run_in_background: false\` (subagents default to background since ${CC_BACKGROUND_DEFAULT_SINCE}; the findings must land before the workflow continues). The subagent has fresh context and no conversation bias — but it is the same harness; model identity stays unknown unless the runtime reports it; weigh its agreement accordingly.
Bound it the same way as ${outsideVoiceFor(ctx).label}: cap the dispatch at a 5-minute timeout so "never blocking"
@@ -743,7 +746,11 @@ For each substantive tension point, use AskUserQuestion:
> argues [Y]. [One sentence on what context you might be missing.]"
>
> RECOMMENDATION: Choose [A or B] because [one-line reason explaining which argument
> is more compelling and why]. Completeness: A=X/10, B=Y/10.
> is more compelling and why].
Score completeness only when the concrete remedies differ in coverage. Otherwise,
use the preamble's kind-not-coverage note; accepting, keeping, investigating, and
deferring do not themselves imply completeness scores.
Options:
- A) Accept the outside voice's recommendation (I'll apply this change)
@@ -839,7 +846,10 @@ Immediately before dispatching, check the preflight result again. On
\`CODEX_MODE: disabled\`, finish this section with \`outside_status: disabled\`;
do not dispatch. Otherwise, use this fallback for missing/broken CLI, failed
authentication/model selection, a failed preflight, or a failed outside invocation.
The disabled branch never reaches this fallback.
The disabled branch never reaches this fallback. In this section, \`under_codex\`
or \`under_current_harness\` also follows this native fallback: run no outside CLI,
report the setup repair and \`outside_status: unavailable\`, then use the native
subagent below. A native result never supplies outside coverage.
Dispatch via the Agent tool with the same prompt, passing \`run_in_background: false\` (subagents default to background since ${CC_BACKGROUND_DEFAULT_SINCE}). Bound it at a 5-minute timeout; if it never completes, treat the review as unavailable and continue.
Present findings under \`DOCUMENTATION REVIEW (${outsideVoiceFor(ctx).nativeLabel} subagent):\`. If it fails: "Doc review unavailable. Continuing to Step 9." Skip the apply gate, persist \`status: unavailable\`, \`outside_status: unavailable\`, and \`source: none\` below, then continue; unavailable is not a clean review.