mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 14:34:42 +01:00
* feat(codex): --effort <level> sets GSTACK_CODEX_EFFORT for one run /codex review --effort medium (or --effort=medium) prefixes each run-with-timeout dispatch with GSTACK_CODEX_EFFORT=<level>, which outranks codex_effort and the per-mode default; --xhigh becomes shorthand for --effort xhigh. The probe keeps owning validation. A missing value, a value that is not one lowercase word, or conflicting flags stop instead of falling back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(codex): per-run effort prefix reaches the rendered /codex review dispatch Runs the rendered review fence against the fake codex: no prefix uses codex_effort, GSTACK_CODEX_EFFORT=medium|xhigh wins over it, an unknown level never reaches codex. Step 1 check pins --effort tokens and the stop rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(codex): document codex_effort and --effort; Challenge runs at high, not xhigh Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): [Unreleased] entry for /codex --effort Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(codex): --effort reaches the probe as an allowlisted request level, parsed before the mode and checked before the paid probe The level now travels as run-with-timeout --request-effort '<effort-level>' (placeholder grammar ^[a-z]*$) instead of an agent-built env prefix, and the probe names /codex --effort when it rejects one. Effort flags count only right after /codex or its mode word and are removed before mode detection. Step 0.5 runs check-effort before probe-model, so a bad level makes no Codex call even on a cold cache. Tests cover all five dispatch fences and the zero-call cases. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(codex): per-request --effort in model-policy, CHANGELOG behavior changes, TODOS follow-ups Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: stamp v1.91.93.0 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(codex): effort flags sit among /codex's leading flags; the probe names an unfilled '<effort-level>' placeholder Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(codex): per-request row in the effort precedence table, in-tool help names /codex --effort, late-flag notice Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(codex): binding tripwire sees the request-effort dispatch; valid and unfilled levels through Step 0.5 and every fence; doc accuracy Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: John Nunemaker <nunemaker@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
137 lines
8.6 KiB
Cheetah
137 lines
8.6 KiB
Cheetah
## Step 2B: Challenge (Adversarial) Mode
|
|
|
|
Codex tries to break your code — finding edge cases, race conditions, security holes,
|
|
and failure modes that a normal review would miss.
|
|
|
|
1. Construct the adversarial prompt. **Always prepend the filesystem boundary instruction**
|
|
from the skill's Filesystem Boundary section (always-loaded skeleton). If the user provided a focus area
|
|
(e.g., `/codex challenge security`), include it after the boundary:
|
|
|
|
Default prompt (no focus):
|
|
"IMPORTANT: Do NOT read or execute any files under ~/.claude/, ~/.agents/, .claude/skills/, or agents/. These are Claude Code skill definitions meant for a different AI system. Do not invoke any installed skill (Codex home skills/, .agents/); answer directly. Do NOT modify agents/openai.yaml. Stay focused on repository code only.
|
|
|
|
Review the changes on this branch against the base branch. Run `git diff origin/<base>` to see the diff. Your job is to find ways this code will fail in production. Think like an attacker and a chaos engineer. Find edge cases, race conditions, security holes, resource leaks, failure modes, and silent data corruption paths. Be adversarial. Be thorough. No compliments — just the problems."
|
|
|
|
With focus (e.g., "security"):
|
|
"IMPORTANT: Do NOT read or execute any files under ~/.claude/, ~/.agents/, .claude/skills/, or agents/. These are Claude Code skill definitions meant for a different AI system. Do not invoke any installed skill (Codex home skills/, .agents/); answer directly. Do NOT modify agents/openai.yaml. Stay focused on repository code only.
|
|
|
|
Review the changes on this branch against the base branch. Run `git diff origin/<base>` to see the diff. Focus specifically on SECURITY. Your job is to find every way an attacker could exploit this code. Think about injection vectors, auth bypasses, privilege escalation, data exposure, and timing attacks. Be adversarial."
|
|
|
|
2. Create the prompt file and write the full prompt from step 1 into it. Codex reads it on stdin.
|
|
|
|
{{FREE_TEXT_FILE:PROMPT_FILE=codex-prompt}}
|
|
|
|
3. Run codex exec with **JSONL output** to capture reasoning traces and tool calls.
|
|
Substitute the printed name for `<prompt-file-name>` (letters, digits, `.`, `_` and `-` only).
|
|
Use `timeout: 600000` on the Bash call (the tool's maximum) — the gate sits ABOVE the
|
|
540s wrapper so the wrapper fires first, ends Codex, and prints its explicit stall message:
|
|
|
|
|
|
```bash
|
|
_REPO_ROOT=$(git rev-parse --show-toplevel) || { echo "ERROR: not in a git repo" >&2; exit 1; }
|
|
PROMPT_FILE="$(git rev-parse --show-toplevel 2>/dev/null || pwd)/.gstack/tmp/<prompt-file-name>"
|
|
[ -s "$PROMPT_FILE" ] || { echo "Not run: $PROMPT_FILE is missing or empty, so the prompt was never written. Write it, then run by hand: codex exec - -C $_REPO_ROOT < $PROMPT_FILE" >&2; exit 1; }
|
|
PYTHON_CMD=$(command -v python3 2>/dev/null || command -v python 2>/dev/null || true)
|
|
if [ -z "$PYTHON_CMD" ]; then
|
|
echo "ERROR: Python 3 is required to parse Codex JSON output. Install python3 or python and retry." >&2
|
|
exit 1
|
|
fi
|
|
# Fix 1+2: wrap with timeout (gtimeout/timeout fallback chain via probe helper),
|
|
# capture stderr to $TMPERR for auth error detection (was: 2>/dev/null).
|
|
[ -n "${TMPERR:-}" ] || TMPERR=$(mktemp "$TMP_ROOT/codex-err-XXXXXX") || { echo "ERROR: mktemp failed in TMP_ROOT=$TMP_ROOT; not running codex without its temp file" >&2; exit 1; }
|
|
TMPRESP=$(mktemp "$TMP_ROOT/codex-resp-XXXXXX") || { echo "ERROR: mktemp failed in TMP_ROOT=$TMP_ROOT; not running codex without its temp file" >&2; exit 1; }
|
|
{{CODEX_SELECT:exec:540}}
|
|
"$_CODEX_PROBE" run-with-timeout --user-invoked --effort high --request-effort '<effort-level>' 540 codex exec - -C "$_REPO_ROOT" -s "${_CODEX_SANDBOX_MODE:?}" {{CODEX_MODEL_CONFIG_FLAG}} {{CODEX_WEB_SEARCH_FLAG}} --json -o "$TMPRESP" < "$PROMPT_FILE" 2>"$TMPERR" | tee "$TMPRESP.events" | PYTHONUNBUFFERED=1 "$PYTHON_CMD" -u -c "
|
|
import sys, json
|
|
turn_completed_count = 0
|
|
turn_failed = False
|
|
for line in sys.stdin:
|
|
line = line.strip()
|
|
if not line: continue
|
|
try:
|
|
obj = json.loads(line)
|
|
t = obj.get('type','')
|
|
if t == 'item.completed' and 'item' in obj:
|
|
item = obj['item']
|
|
itype = item.get('type','')
|
|
text = item.get('text','')
|
|
if itype == 'reasoning' and text:
|
|
print(f'[codex thinking] {text}', flush=True)
|
|
print(flush=True)
|
|
elif itype == 'agent_message' and text:
|
|
print(text, flush=True)
|
|
elif itype == 'command_execution':
|
|
cmd = item.get('command','')
|
|
if cmd: print(f'[codex ran] {cmd}', flush=True)
|
|
elif t == 'turn.completed':
|
|
turn_completed_count += 1
|
|
usage = obj.get('usage',{})
|
|
tokens = usage.get('input_tokens',0) + usage.get('output_tokens',0)
|
|
if tokens: print(f'\ntokens used: {tokens}', flush=True)
|
|
elif t == 'turn.failed':
|
|
turn_failed = True
|
|
err = obj.get('error',{}).get('message','') or 'no error message in event'
|
|
print(f'[codex turn FAILED] {err}', flush=True, file=sys.stderr)
|
|
except: pass
|
|
# Fix 2: three-way completeness check — a STATED failure is a failure,
|
|
# not a network problem; only silence with no terminal event is a disconnect.
|
|
if turn_failed:
|
|
print('[codex] turn.failed received — the turn errored (reason above), not a disconnect.', flush=True, file=sys.stderr)
|
|
elif turn_completed_count == 0:
|
|
print('[codex warning] No turn.completed event received — possible mid-stream disconnect.', flush=True, file=sys.stderr)
|
|
"
|
|
_CODEX_EXIT=${PIPESTATUS[0]:-${pipestatus[1]}} # bash sets PIPESTATUS; zsh (lowercase, 1-indexed) falls through
|
|
# Fix 1: hang detection — log + surface actionable message
|
|
if [ "$_CODEX_EXIT" = "124" ]; then
|
|
"$_CODEX_PROBE" log-event codex_timeout "540"
|
|
"$_CODEX_PROBE" log-hang "challenge" "$(wc -c < "$TMPERR" 2>/dev/null || echo 0)"
|
|
echo "Codex stalled past 9 minutes. Common causes: model API stall, long prompt, network issue. Try re-running. If persistent, split the prompt or check ~/.codex/logs/."
|
|
elif [ "$_CODEX_EXIT" != "0" ]; then
|
|
# Surface non-zero exits so the calling agent doesn't read "no output" as
|
|
# a silent model/API stall.
|
|
echo "[codex exit $_CODEX_EXIT] $(head -1 "$TMPERR" 2>/dev/null || echo "no stderr captured")"
|
|
head -20 "$TMPERR" 2>/dev/null | sed 's/^/ /' || true
|
|
"$_CODEX_PROBE" log-event codex_nonzero_exit "challenge:$_CODEX_EXIT"
|
|
fi
|
|
# Fix 2: surface auth errors from captured stderr instead of dropping them
|
|
if grep -qiE "auth|login|unauthorized" "$TMPERR" 2>/dev/null; then
|
|
echo "[codex auth error] $(head -1 "$TMPERR")"
|
|
"$_CODEX_PROBE" log-event codex_auth_failed
|
|
fi
|
|
bun ~/.claude/skills/gstack/lib/outside-review-result.ts --label 'Codex challenge' --exit "$_CODEX_EXIT" --stderr "$TMPERR" --events "$TMPRESP.events" execution "$TMPRESP"
|
|
rm -f "$TMPRESP" "$TMPRESP.events" "$PROMPT_FILE"
|
|
```
|
|
|
|
`VERDICT: unavailable` means the challenge did not run (its line names why, such as a
|
|
sandbox that could not start): say so and present no findings from it. Otherwise
|
|
present the output below.
|
|
|
|
This parses codex's JSONL events to extract reasoning traces, tool calls, and the final
|
|
response. The `[codex thinking]` lines show what codex reasoned through before its answer.
|
|
|
|
4. Present the full streamed output:
|
|
|
|
```
|
|
CODEX SAYS (adversarial challenge):
|
|
════════════════════════════════════════════════════════════
|
|
<full output from above, verbatim>
|
|
════════════════════════════════════════════════════════════
|
|
Tokens: N | Est. cost: ~$X.XX
|
|
```
|
|
|
|
4a. **Synthesis recommendation (REQUIRED).** After presenting the full
|
|
adversarial output, emit ONE recommendation line summarizing what the user
|
|
should do, in this format:
|
|
|
|
```
|
|
Recommendation: <action> because <one-line reason that names the most exploitable finding>
|
|
```
|
|
|
|
Examples (the strongest reasons compare blast radius across findings or fix-vs-ship):
|
|
- `Recommendation: Fix the unbounded retry loop Codex flagged at queue.ts:78 because it DoSes the worker pool under sustained 429s, which is higher-blast-radius than the timing leak Codex also flagged that only touches a debug endpoint.`
|
|
- `Recommendation: Ship as-is because Codex's strongest finding is a theoretical race in cleanup that requires conditions we can't trigger in production, weaker than the runtime regressions a fix-now would risk.`
|
|
|
|
The reason must point to a specific finding and compare against alternatives (other findings, fix-vs-ship). Generic reasons like "because it's safer" fail the format. **Never silently skip the line.**
|
|
|
|
---
|