From ec069b9d07643aa2aea6f5e7b965dd63edc4c7df Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Thu, 26 Mar 2026 12:17:02 -0600 Subject: [PATCH] fix: unbuffer Python stdout in codex --json streaming MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Python fully buffers stdout when piped (not a TTY). The `codex exec --json | python3 -c "..."` pattern meant zero output visible until process exit — users saw nothing for 30+ minutes. Add PYTHONUNBUFFERED=1 env var, python3 -u flag, and flush=True to all print() calls in all three Python parser blocks (Challenge, Consult new session, Consult resumed session). Co-Authored-By: Claude Opus 4.6 (1M context) --- codex/SKILL.md | 30 +++++++++++++++--------------- codex/SKILL.md.tmpl | 30 +++++++++++++++--------------- 2 files changed, 30 insertions(+), 30 deletions(-) diff --git a/codex/SKILL.md b/codex/SKILL.md index ec9eea7c..5739604e 100644 --- a/codex/SKILL.md +++ b/codex/SKILL.md @@ -520,7 +520,7 @@ With focus (e.g., "security"): 2. Run codex exec with **JSONL output** to capture reasoning traces and tool calls (5-minute timeout): ```bash -codex exec "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>/dev/null | python3 -c " +codex exec "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>/dev/null | PYTHONUNBUFFERED=1 python3 -u -c " import sys, json for line in sys.stdin: line = line.strip() @@ -533,17 +533,17 @@ for line in sys.stdin: itype = item.get('type','') text = item.get('text','') if itype == 'reasoning' and text: - print(f'[codex thinking] {text}') - print() + print(f'[codex thinking] {text}', flush=True) + print(flush=True) elif itype == 'agent_message' and text: - print(text) + print(text, flush=True) elif itype == 'command_execution': cmd = item.get('command','') - if cmd: print(f'[codex ran] {cmd}') + if cmd: print(f'[codex ran] {cmd}', flush=True) elif t == 'turn.completed': usage = obj.get('usage',{}) tokens = usage.get('input_tokens',0) + usage.get('output_tokens',0) - if tokens: print(f'\ntokens used: {tokens}') + if tokens: print(f'\ntokens used: {tokens}', flush=True) except: pass " ``` @@ -605,7 +605,7 @@ THE PLAN: For a **new session:** ```bash -codex exec "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>"$TMPERR" | python3 -c " +codex exec "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>"$TMPERR" | PYTHONUNBUFFERED=1 python3 -u -c " import sys, json for line in sys.stdin: line = line.strip() @@ -615,31 +615,31 @@ for line in sys.stdin: t = obj.get('type','') if t == 'thread.started': tid = obj.get('thread_id','') - if tid: print(f'SESSION_ID:{tid}') + if tid: print(f'SESSION_ID:{tid}', flush=True) elif t == 'item.completed' and 'item' in obj: item = obj['item'] itype = item.get('type','') text = item.get('text','') if itype == 'reasoning' and text: - print(f'[codex thinking] {text}') - print() + print(f'[codex thinking] {text}', flush=True) + print(flush=True) elif itype == 'agent_message' and text: - print(text) + print(text, flush=True) elif itype == 'command_execution': cmd = item.get('command','') - if cmd: print(f'[codex ran] {cmd}') + if cmd: print(f'[codex ran] {cmd}', flush=True) elif t == 'turn.completed': usage = obj.get('usage',{}) tokens = usage.get('input_tokens',0) + usage.get('output_tokens',0) - if tokens: print(f'\ntokens used: {tokens}') + if tokens: print(f'\ntokens used: {tokens}', flush=True) except: pass " ``` For a **resumed session** (user chose "Continue"): ```bash -codex exec resume "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>"$TMPERR" | python3 -c " - +codex exec resume "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>"$TMPERR" | PYTHONUNBUFFERED=1 python3 -u -c " + " ``` diff --git a/codex/SKILL.md.tmpl b/codex/SKILL.md.tmpl index 77021c82..41e64fe7 100644 --- a/codex/SKILL.md.tmpl +++ b/codex/SKILL.md.tmpl @@ -159,7 +159,7 @@ With focus (e.g., "security"): 2. Run codex exec with **JSONL output** to capture reasoning traces and tool calls (5-minute timeout): ```bash -codex exec "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>/dev/null | python3 -c " +codex exec "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>/dev/null | PYTHONUNBUFFERED=1 python3 -u -c " import sys, json for line in sys.stdin: line = line.strip() @@ -172,17 +172,17 @@ for line in sys.stdin: itype = item.get('type','') text = item.get('text','') if itype == 'reasoning' and text: - print(f'[codex thinking] {text}') - print() + print(f'[codex thinking] {text}', flush=True) + print(flush=True) elif itype == 'agent_message' and text: - print(text) + print(text, flush=True) elif itype == 'command_execution': cmd = item.get('command','') - if cmd: print(f'[codex ran] {cmd}') + if cmd: print(f'[codex ran] {cmd}', flush=True) elif t == 'turn.completed': usage = obj.get('usage',{}) tokens = usage.get('input_tokens',0) + usage.get('output_tokens',0) - if tokens: print(f'\ntokens used: {tokens}') + if tokens: print(f'\ntokens used: {tokens}', flush=True) except: pass " ``` @@ -244,7 +244,7 @@ THE PLAN: For a **new session:** ```bash -codex exec "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>"$TMPERR" | python3 -c " +codex exec "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>"$TMPERR" | PYTHONUNBUFFERED=1 python3 -u -c " import sys, json for line in sys.stdin: line = line.strip() @@ -254,31 +254,31 @@ for line in sys.stdin: t = obj.get('type','') if t == 'thread.started': tid = obj.get('thread_id','') - if tid: print(f'SESSION_ID:{tid}') + if tid: print(f'SESSION_ID:{tid}', flush=True) elif t == 'item.completed' and 'item' in obj: item = obj['item'] itype = item.get('type','') text = item.get('text','') if itype == 'reasoning' and text: - print(f'[codex thinking] {text}') - print() + print(f'[codex thinking] {text}', flush=True) + print(flush=True) elif itype == 'agent_message' and text: - print(text) + print(text, flush=True) elif itype == 'command_execution': cmd = item.get('command','') - if cmd: print(f'[codex ran] {cmd}') + if cmd: print(f'[codex ran] {cmd}', flush=True) elif t == 'turn.completed': usage = obj.get('usage',{}) tokens = usage.get('input_tokens',0) + usage.get('output_tokens',0) - if tokens: print(f'\ntokens used: {tokens}') + if tokens: print(f'\ntokens used: {tokens}', flush=True) except: pass " ``` For a **resumed session** (user chose "Continue"): ```bash -codex exec resume "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>"$TMPERR" | python3 -c " - +codex exec resume "" -C "$(git rev-parse --show-toplevel)" -s read-only -c 'model_reasoning_effort="xhigh"' --enable web_search_cached --json 2>"$TMPERR" | PYTHONUNBUFFERED=1 python3 -u -c " + " ```