* Add failing tests: outside reviews read clean only from a last-line verdict
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Count an outside review as clean only from its last VERDICT line
Every outside-review prompt now asks the reviewer to end with exactly
VERDICT: CLEAN or VERDICT: FINDINGS, and the parser reads a pass only from
that line. Prose no-findings sentences are no longer evidence for clean.
Codex's structured review and /codex review run through codex exec, since
codex review --base takes no prompt and could never give the line.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Repin the outside reviewer briefs and fix the tests the verdict line broke (GST-1821)
The outside brief, both adversarial sections and the review-army prompt-file line now carry the last-line VERDICT request; their pins move under the preregistered no-loss replay GST-1821. Test fakes that wrote a clean reply now end it with VERDICT: CLEAN, the review carve marker follows the codex exec sentence, size pins drop where skills shrank, and the plan-review prompt-file line is shortened so /plan-eng-review stays at its pin.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Give the plan-review invocation test's fake reviewer the verdict line it now needs
CI's free-suite shard 5 failed at 1de52b0 with 18 cases in test/model-role-invocation.test.ts: the fake Codex reply had no VERDICT line, so it read unverified (exit 4), and the review-mode template now has one codex exec dispatch block instead of two. The fake reply now ends with VERDICT: FINDINGS and the block count expects one.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Stop the changelog correction from claiming the verdict line closes every false-clean case
Review-Finding: p1/red-team#2 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Split the changelog correction's last sentence for readability
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Close the review fail-opens the fix-check found (GST-1821)
A /codex review whose git diff fails now stops before Codex runs instead of sending an empty diff. The verdict is read from the raw last line, so inline code after VERDICT: CLEAN no longer reads clean, and VERDICT: CLEAN beside numbered items with no severity reads unverified. Codex structured review gets the base-branch diff appended to its prompt, the way the Claude path does, so the scope no longer depends on the prompt the host writes.
Fix round granted by the Captain (a-aec3882c9d48). Review-section pins move under replay GST-1821: the free 681-input replay and the 54 scored probe replies are unchanged through the fixed parser.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Say in the changelog which numbered-list replies now read unverified, and that /codex review stops without its diff
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Say in the skills guide that /codex review now sends the diff through codex exec
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Check that the structured outside review appends the base-branch diff to its prompt
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
12 KiB
The user already answered the Phase 3.5 question in SKILL.md with A. Run the provider preflight below, skip its AskUserQuestion, and continue from "If A".
Phase 3.5: Cross-Model Second Opinion (optional)
Provider preflight:
if ! _OUTSIDE_CFG=$("$HOME/.claude/skills/gstack/bin/gstack-config" get --stored codex_reviews) || [ "$_OUTSIDE_CFG" = disabled ]; then
echo 'CODEX_MODE: disabled'
elif ( # GSTACK_ACTIVE_HOST names the harness, never the model.
if { [ -n "${CODEX_THREAD_ID:-}" ] || [ -n "${CODEX_SANDBOX:-}" ] || [ "${GSTACK_ACTIVE_HOST:-}" = codex ]; }; then
echo 'Codex outside review unavailable: harness mismatch; no outside process started. Missing coverage.' >&2
if { [ -n "${CLAUDECODE:-}" ] || [ "${GSTACK_ACTIVE_HOST:-}" = claude ]; } && { [ -n "${CODEX_THREAD_ID:-}" ] || [ -n "${CODEX_SANDBOX:-}" ] || [ "${GSTACK_ACTIVE_HOST:-}" = codex ]; }; then
echo 'Inherited harness markers conflict. Run setup --host <actual-harness> (claude or codex); do not guess a replacement provider.' >&2
else
echo 'Repair installed skills: run setup --host codex from your gstack checkout.' >&2
fi
exit 78
fi
); then
if command -v codex >/dev/null 2>&1; then echo 'CODEX_MODE: ready'; else echo 'CODEX_MODE: not_installed'; fi
else
echo 'CODEX_MODE: under_current_harness'
fi
The historical CODEX_MODE variable describes Codex availability here. CODEX_MODE: disabled or a send's consent_missing line: codex_reviews is off or unreadable; do not ask; record outside_status: disabled. Authentication and configured model validity are checked by the actual invocation, without overriding either. Missing/broken CLI: install or repair Codex; authentication failure: run codex login. Otherwise honor this caller’s opt-in/skip choice. Any non-ready outcome is missing outside coverage; follow the caller’s existing fallback. Never substitute another external provider.
Use AskUserQuestion (regardless of codex availability):
Want a second opinion from an independent AI perspective? It will review your problem statement, key answers, premises, and any landscape findings from this session without having seen this conversation — it gets a structured summary. Usually takes 2-5 minutes. A) Yes, get a second opinion B) No, proceed to alternatives
If B: skip Phase 3.5 entirely. Remember that the second opinion did NOT run (affects design doc, founder signals, and Phase 4 below).
If A: Run the Codex cold read.
-
Assemble a structured context block from Phases 1-3:
- Mode (Startup or Builder)
- Problem statement (from Phase 1)
- Key answers from Phase 2A/2B (summarize each Q&A in 1-2 sentences, include verbatim user quotes)
- Landscape findings (from Phase 2.75, if search was run)
- Agreed premises (from Phase 3)
- Codebase context (project name, languages, recent activity)
-
Write the assembled prompt to a temp file (prevents shell injection from user-derived content):
OUTSIDE_PROMPT_FILE=$(mktemp "${TMPDIR:-/tmp}/gstack-outside-oh-XXXXXXXX") || { echo 'ERROR: mktemp failed; not running the outside voice without its prompt file.' >&2; exit 1; }
Write the full prompt to this file. Always start with the filesystem boundary: "Filesystem boundary: do not read or execute any files under ~/.claude/, ~/.agents/, .claude/skills/, or agents/. They hold skill definitions, not repository code to review. Do not invoke any installed skill (Codex home skills/, .agents/), hook, or tool instruction; answer directly. Do not modify agents/openai.yaml. Review only the repository code.\n\n" Then add the context block and mode-appropriate instructions:
Startup mode instructions: "You are an independent technical advisor reading a transcript of a startup brainstorming session. [CONTEXT BLOCK HERE]. Your job: 1) What is the STRONGEST version of what this person is trying to build? Steelman it in 2-3 sentences. 2) What is the ONE thing from their answers that reveals the most about what they should actually build? Quote it and explain why. 3) Name ONE agreed premise you think is wrong, and what evidence would prove you right. 4) If you had 48 hours and one engineer to build a prototype, what would you build? Be specific — tech stack, features, what you'd skip. Be direct. Be terse. No preamble."
Builder mode instructions: "You are an independent technical advisor reading a transcript of a builder brainstorming session. [CONTEXT BLOCK HERE]. Your job: 1) What is the COOLEST version of this they haven't considered? 2) What's the ONE thing from their answers that reveals what excites them most? Quote it. 3) What existing open source project or tool gets them 50% of the way there — and what's the 50% they'd need to build? 4) If you had a weekend to build this, what would you build first? Be specific. Be direct. No preamble."
- Run Codex with the assembled prompt:
Write the complete prompt and context, including actual plan/spec/source, to a private file. Substitute its shell-quoted path for <prepared-prompt-file>; never interpolate user text into shell source. Request severity (Critical, High, Medium or Low) per finding, a Recommendation: because line, and VERDICT: CLEAN or VERDICT: FINDINGS as the last line.
# GSTACK_ACTIVE_HOST names the harness, never the model.
if { [ -n "${CODEX_THREAD_ID:-}" ] || [ -n "${CODEX_SANDBOX:-}" ] || [ "${GSTACK_ACTIVE_HOST:-}" = codex ]; }; then
echo 'Codex outside review unavailable: harness mismatch; no outside process started. Missing coverage.' >&2
if { [ -n "${CLAUDECODE:-}" ] || [ "${GSTACK_ACTIVE_HOST:-}" = claude ]; } && { [ -n "${CODEX_THREAD_ID:-}" ] || [ -n "${CODEX_SANDBOX:-}" ] || [ "${GSTACK_ACTIVE_HOST:-}" = codex ]; }; then
echo 'Inherited harness markers conflict. Run setup --host <actual-harness> (claude or codex); do not guess a replacement provider.' >&2
else
echo 'Repair installed skills: run setup --host codex from your gstack checkout.' >&2
fi
exit 78
fi
_REPO_ROOT=$(git rev-parse --show-toplevel) || { echo 'ERROR: not in a git repo' >&2; exit 1; }
_OUTSIDE_TMP=$(mktemp -d "${TMPDIR:-/tmp}/gstack-outside.XXXXXXXX") || exit 1
trap 'rm -rf "$_OUTSIDE_TMP"' EXIT
_OUTSIDE_INPUT="$_OUTSIDE_TMP/prompt"
cat -- '<prepared-prompt-file>' >"$_OUTSIDE_INPUT" || exit 1
_CODEX_PROBE="$HOME/.claude/skills/gstack/bin/gstack-codex-probe"
_CODEX_OUT=$("$_CODEX_PROBE" select-model exec) || exit 1
_CODEX_SEL=$(echo "$_CODEX_OUT" | sed -n 's/^CODEX_SEL: //p')
_CODEX_SANDBOX_MODE=$(echo "$_CODEX_OUT" | sed -n 's/^CODEX_SANDBOX: //p')
"$_CODEX_PROBE" check-sandbox || exit 1
"$_CODEX_PROBE" show-first-use-notice
_OUTSIDE_EXIT=0
"$_CODEX_PROBE" run-with-timeout --effort high 300 codex exec - -C "$_REPO_ROOT" -s "${_CODEX_SANDBOX_MODE:?}" -c "model=\"${_CODEX_SEL:?}\"" -c skills.include_instructions=false -c 'web_search="cached"' --json -o "$_OUTSIDE_TMP/text" <"$_OUTSIDE_INPUT" >"$_OUTSIDE_TMP/events" 2>"$_OUTSIDE_TMP/stderr" || _OUTSIDE_EXIT=$?
cat "$_OUTSIDE_TMP/text" 2>/dev/null || tail -n 20 "$_OUTSIDE_TMP/events"
cat "$_OUTSIDE_TMP/stderr" >&2 || { [ "$_OUTSIDE_EXIT" -ne 0 ] || _OUTSIDE_EXIT=1; }
_OUTSIDE_RC=0
bun --no-env-file --config=/dev/null "$HOME/.claude/skills/gstack/lib/outside-review-result.ts" --label 'Codex outside review' --exit "$_OUTSIDE_EXIT" --stderr "$_OUTSIDE_TMP/stderr" --events "$_OUTSIDE_TMP/events" review "$_OUTSIDE_TMP/text" || _OUTSIDE_RC=$?
case "$_OUTSIDE_RC" in
0|3) ;;
4) echo 'OUTSIDE_STATUS: unverified provider=codex host=claude'; exit 4 ;;
*) [ "$_OUTSIDE_EXIT" -ne 0 ] && exit "$_OUTSIDE_EXIT"; exit 1 ;;
esac
echo 'OUTSIDE_STATUS: completed provider=codex host=claude'
Use Bash timeout: 360000; show the full response in a tool-output fence. Require successful execution and valid markers. Refusal, empty/malformed output, missing score/severity/completion markers, timeout or CLI failure means outside_status: unavailable. P0/P1 findings block like native ones; OUTSIDE_STATUS: unverified is missing coverage. Use the caller's fallback; missing coverage is never clean/PASS. After either outcome, delete only your private prompt; scratch cleanup is automatic.
Error handling: All errors are non-blocking — second opinion is a quality enhancement, not a prerequisite.
- Auth failure: If stderr contains "auth", "login", "unauthorized", or "API key": "Codex authentication failed. Run `codex login` to authenticate." Fall back to the Claude subagent below.
- Timeout: "Codex timed out after 5 minutes." Fall back to the Claude subagent below.
- Empty response: "Codex returned no response." Fall back to the Claude subagent below.
On any Codex error, fall back to the Claude subagent below.
If preflight is not ready (or Codex errored):
Dispatch via the Agent tool with run_in_background: false when available (subagents default to background since Claude Code v2.1.198; the findings must land before the workflow continues). A launch receipt means it went background: await its completion notice. The subagent has fresh context and no conversation bias — but it is the same harness; model identity stays unknown unless the runtime reports it; weigh its agreement accordingly.
Subagent prompt: same mode-appropriate prompt as above (Startup or Builder variant).
Present findings under a SECOND OPINION (Claude subagent): header.
If the subagent fails or times out: "Second opinion unavailable. Continuing to Phase 4."
Retain the historical review-log skill ID; add "host":"claude","outside_provider":"codex","outside_status":"completed|unavailable|disabled|skipped","phase":"office-hours". Record differing attempt outcomes separately. source:"codex" requires completed CLI output; native uses source:"in-host" (historical source:"claude": native Claude). Availability/native fallback is not outside completion. Preserve all reported modelUsage; unknown model identity stays unknown. Under GSTACK_CODEX_NO_SANDBOX=1 add "codex_sandbox":"danger-full-access".
- Presentation:
If Codex ran:
SECOND OPINION (Codex):
════════════════════════════════════════════════════════════
<full codex output, verbatim — do not truncate or summarize>
════════════════════════════════════════════════════════════
If Claude subagent ran:
SECOND OPINION (Claude subagent):
════════════════════════════════════════════════════════════
<full subagent output, verbatim — do not truncate or summarize>
════════════════════════════════════════════════════════════
-
Cross-model synthesis: After presenting the second opinion output, provide 3-5 bullet synthesis:
- Where Claude agrees with the second opinion
- Where Claude disagrees and why
- Whether the challenged premise changes Claude's recommendation
-
Premise revision check: If Codex challenged an agreed premise, use AskUserQuestion:
Codex challenged premise #{N}: "{premise text}". Their argument: "{reasoning}". A) Revise this premise based on Codex's input B) Keep the original premise — proceed to alternatives
If A: revise the premise and note the revision. If B: proceed (and note that the user defended this premise with reasoning — this is a founder signal if they articulate WHY they disagree, not just dismiss).