mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-13 16:38:56 +02:00
feat(codex): model round-trip probe — an unusable configured model fails fast with guidance (#2477)
The auth probe accepts 'auth exists' as readiness, but a ChatGPT account
with a stale model pin in ~/.codex/config.toml passes it and then EVERY
mode dies with an HTTP 400 ('The <model> model is not supported when using
Codex with a ChatGPT account') and no pointer to where the model came from
— one report burned ~40 minutes and four invocations plus a strings dump
of the binary before finding the one-line config fix.
bin/gstack-codex-probe gains _gstack_codex_model_probe: a short
codex exec 'reply OK' round trip with the configured model, gated behind
the cheap auth probe at all three preflight sites (codex Step 0.5, the
shared codexPreflight in scripts/resolvers/constants.ts — which grows a
model_unusable CODEX_MODE branch — and autoplan's availability chain).
Verdicts: MODEL_OK (cached 1h, keyed on config.toml + auth.json mtimes so
a pin edit or re-login re-probes immediately), MODEL_UNUSABLE (exit 1,
prints the rejection plus HINTs at the model= pin and the
[notice.model_migrations] table), MODEL_PROBE_INCONCLUSIVE (timeout or
transient: FAIL-OPEN so network luck never wedges codex mode).
The 'Model not supported (HTTP 400)' Error Handling entry already shipped
in v1.64.0.0; Step 0.5's prose now routes MODEL_UNUSABLE to it.
test/codex-model-probe.test.ts drives all four behaviors against a stubbed
codex binary (invocation-counted cache hit, hint content, fail-open
polarity, mtime invalidation).
Fixes #2477
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Fable 5
Claude Fable 5
parent
63e2b7ac2c
commit
73c96f9c12
+68
-1
@@ -4,6 +4,7 @@
|
||||
#
|
||||
# Functions (all prefixed with _gstack_codex_ for namespace hygiene):
|
||||
# _gstack_codex_auth_probe — multi-signal auth check (env + file)
|
||||
# _gstack_codex_model_probe — round-trip probe of the configured model (#2477)
|
||||
# _gstack_codex_version_check — warn on known-bad Codex CLI versions
|
||||
# _gstack_codex_timeout_wrapper — gtimeout -> timeout -> unwrapped fallback
|
||||
# _gstack_codex_log_event — telemetry emission to ~/.gstack/analytics/
|
||||
@@ -33,6 +34,72 @@ _gstack_codex_auth_probe() {
|
||||
return 1
|
||||
}
|
||||
|
||||
# --- Model round-trip probe (#2477) ------------------------------------------
|
||||
|
||||
_gstack_codex_model_probe() {
|
||||
# Auth-exists is a weaker signal than the auth probe implies: a ChatGPT
|
||||
# account with a stale `model = "..."` pin in ~/.codex/config.toml passes
|
||||
# the auth probe, then EVERY invocation dies with an HTTP 400 ("The
|
||||
# '<model>' model is not supported when using Codex with a ChatGPT
|
||||
# account") and no guidance. A short real round trip with the configured
|
||||
# model catches model rejection, entitlement changes, and stale pins in
|
||||
# one shot (#2477).
|
||||
#
|
||||
# Contract:
|
||||
# MODEL_OK (exit 0) — round trip succeeded; cached 1h.
|
||||
# MODEL_UNUSABLE (exit 1) — deterministic model 400; hints printed.
|
||||
# MODEL_PROBE_INCONCLUSIVE (exit 0) — timeout/transient; FAIL-OPEN so a
|
||||
# slow network never wedges codex mode (the per-invocation Error
|
||||
# Handling entry still covers a later 400).
|
||||
#
|
||||
# Only call this AFTER _gstack_codex_auth_probe passes — probing without
|
||||
# auth just measures the auth failure again.
|
||||
local _codex_home="${CODEX_HOME:-$HOME/.codex}"
|
||||
local _gstack_home="${GSTACK_HOME:-$HOME/.gstack}"
|
||||
local _cache="$_gstack_home/.codex-model-probe"
|
||||
# Cache signature: config.toml + auth.json mtimes. Editing the model pin
|
||||
# or re-logging-in invalidates the cached MODEL_OK immediately.
|
||||
local _cfg_m _auth_m _sig
|
||||
_cfg_m=$(stat -f %m "$_codex_home/config.toml" 2>/dev/null || stat -c %Y "$_codex_home/config.toml" 2>/dev/null || echo 0)
|
||||
_auth_m=$(stat -f %m "$_codex_home/auth.json" 2>/dev/null || stat -c %Y "$_codex_home/auth.json" 2>/dev/null || echo 0)
|
||||
_sig="${_cfg_m}-${_auth_m}"
|
||||
local _now
|
||||
_now=$(date +%s 2>/dev/null || echo 0)
|
||||
if [ -f "$_cache" ]; then
|
||||
local _c_line _c_status _c_ts _c_sig
|
||||
_c_line=$(head -1 "$_cache" 2>/dev/null)
|
||||
_c_status=$(printf '%s' "$_c_line" | cut -d' ' -f1)
|
||||
_c_ts=$(printf '%s' "$_c_line" | cut -d' ' -f2)
|
||||
_c_sig=$(printf '%s' "$_c_line" | cut -d' ' -f3)
|
||||
case "$_c_ts" in ''|*[!0-9]*) _c_ts=0 ;; esac
|
||||
if [ "$_c_status" = "MODEL_OK" ] && [ "$_c_sig" = "$_sig" ] && [ $((_now - _c_ts)) -lt 3600 ]; then
|
||||
echo "MODEL_OK (cached)"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
local _out _code
|
||||
_out=$(_gstack_codex_timeout_wrapper 30 codex exec --skip-git-repo-check -s read-only "reply OK" </dev/null 2>&1)
|
||||
_code=$?
|
||||
if [ "$_code" -eq 0 ]; then
|
||||
mkdir -p "$_gstack_home" 2>/dev/null || true
|
||||
printf 'MODEL_OK %s %s\n' "$_now" "$_sig" > "$_cache" 2>/dev/null || true
|
||||
echo "MODEL_OK"
|
||||
return 0
|
||||
fi
|
||||
if printf '%s' "$_out" | grep -qiE 'model.{0,40}is not supported|"status":[[:space:]]*400'; then
|
||||
echo "MODEL_UNUSABLE"
|
||||
printf '%s\n' "$_out" | grep -i "model" | head -3
|
||||
echo "HINT: the rejected model comes from the 'model = ' line in $_codex_home/config.toml."
|
||||
echo "HINT: check its [notice.model_migrations] table — Codex records the intended replacement there."
|
||||
_gstack_codex_log_event "codex_model_unusable" 2>/dev/null || true
|
||||
return 1
|
||||
fi
|
||||
# Timeout (124) or transient failure: fail-open with a warning. The probe
|
||||
# exists to catch the deterministic model 400, not to gate on network luck.
|
||||
echo "MODEL_PROBE_INCONCLUSIVE (exit $_code) — proceeding; if invocations fail with a model 400, see the codex skill's Error Handling entry."
|
||||
return 0
|
||||
}
|
||||
|
||||
# --- Version check ----------------------------------------------------------
|
||||
|
||||
_gstack_codex_version_check() {
|
||||
@@ -72,7 +139,7 @@ _gstack_codex_log_event() {
|
||||
# Emit a telemetry event to ~/.gstack/analytics/skill-usage.jsonl.
|
||||
# Gated on $_TEL != "off" (caller sets this from gstack-config).
|
||||
# Event types: codex_timeout, codex_auth_failed, codex_cli_missing,
|
||||
# codex_version_warning.
|
||||
# codex_version_warning, codex_model_unusable.
|
||||
# Payload schema: {skill, event, duration_s, ts}. NEVER includes prompt
|
||||
# content, env var values, or auth tokens.
|
||||
local _event="$1"
|
||||
|
||||
Reference in New Issue
Block a user