From 957fab79c5da5d5654470164411788cfcc602376 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Thu, 19 Nov 2026 18:27:37 +0800 Subject: [PATCH] Second-model review keeps the reviewer's result and reports budget stops (GST-3181) (#4515) When Codex reviews a change, gstack now keeps the review file the reviewer wrote instead of replacing it with Codex's one-line closing message, and a provider that refuses with HTTP 402 is reported as a used-up budget instead of "exit 1" or a login failure. Why: gstack-evals lost 22 review attempts to the overwrite and 5 of 45 sessions to a 402 from a metering proxy whose cause gstack threw away. Nothing to do: a 401 or 403 is still a login problem. Open notes, one of them a known gap, are listed below. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> Review-Finding: p1/specialist-batched#1 reader=testing Review-Finding: p1/red-team#1 reader=red-team Review-Finding: p1/red-team#2 reader=red-team --- .../report-review-budget-stop-not-login.md | 4 + codex/SKILL.md | 8 +- codex/SKILL.md.tmpl | 8 +- docs/troubleshooting.md | 18 +++++ lib/claude-code.ts | 9 ++- lib/gate-outcomes.ts | 5 ++ lib/outside-review-result.ts | 29 ++++++- lib/outside-voice-runner.ts | 10 ++- test/claude-code-runner.test.ts | 25 ++++++ test/model-catalog-isolation.test.ts | 1 + test/outside-review-result.test.ts | 78 +++++++++++++++++++ test/outside-runner-contract.test.ts | 64 ++++++++++++++- test/outside-voice-failure-policy.test.ts | 16 ++++ test/outside-voice-invocation.test.ts | 46 +++++++++++ 14 files changed, 311 insertions(+), 10 deletions(-) create mode 100644 changelog.d/report-review-budget-stop-not-login.md diff --git a/changelog.d/report-review-budget-stop-not-login.md b/changelog.d/report-review-budget-stop-not-login.md new file mode 100644 index 000000000..afcc70a30 --- /dev/null +++ b/changelog.d/report-review-budget-stop-not-login.md @@ -0,0 +1,4 @@ +**The second-model review now keeps the reviewer's own result, and reports a used-up budget as a budget stop.** Two fixes to the Codex review that /autoplan, /review, /ship and the other skills run: + +- When the reviewer wrote its review to the result file it was asked for, Codex then replaced that file with its short closing message ("Done, I wrote the review"), so the review was thrown away and reported as missing its markers. Codex's closing message now goes to a separate `.last-message` file, and gstack keeps whichever of the two carries the review. +- When the provider refuses the call with HTTP 402 Payment Required (usually a metering proxy that hit its allowance, or an account out of credits or quota), gstack used to record only "exit 1". It now reads the cause from Codex's JSON error events and reports "the provider refused the review with HTTP 402 (payment required)", and says to check the proxy's allowance, or the account's quota and credits. It has its own reason code, `budget_exhausted`, in the run record and in docs/troubleshooting.md, and the skills' "authentication failed, run `codex login`" rule no longer fires on it. A 401 or 403 is still a login problem, and its record now keeps Codex's status line instead of "exit 1". diff --git a/codex/SKILL.md b/codex/SKILL.md index 35ca68472..06980f672 100644 --- a/codex/SKILL.md +++ b/codex/SKILL.md @@ -560,7 +560,13 @@ If token count is not available, display: `Tokens: unknown` ## Error Handling - **Binary not found:** Detected in Step 0. Stop with install instructions. -- **Auth error:** Codex prints an auth error to stderr. Surface the error: +- **Budget or quota stop (HTTP 402, payment required, quota):** the validator prints + `unavailable: the provider refused the review with HTTP 402` or names the quota. This is + not an auth error, even when the provider's message mentions the API key. Relay Codex's line + and tell the user: "Codex refused the call because its budget or quota is used up. Check the + allowance on the proxy or gateway in front of the provider, or the account's quota and + credits; logging in again will not fix it." +- **Auth error:** Codex prints an auth error to stderr (401 or 403). Surface the error: "Codex authentication failed. Run `codex login` in your terminal to authenticate via ChatGPT." - **Timeout (Bash outer gate):** Every Bash gate sits ABOVE its inner wrapper (360s gate over the 330s review wrapper; 600s gate, the tool maximum, over the 540s challenge/consult diff --git a/codex/SKILL.md.tmpl b/codex/SKILL.md.tmpl index e9a6b1e38..733384e7e 100644 --- a/codex/SKILL.md.tmpl +++ b/codex/SKILL.md.tmpl @@ -281,7 +281,13 @@ If token count is not available, display: `Tokens: unknown` ## Error Handling - **Binary not found:** Detected in Step 0. Stop with install instructions. -- **Auth error:** Codex prints an auth error to stderr. Surface the error: +- **Budget or quota stop (HTTP 402, payment required, quota):** the validator prints + `unavailable: the provider refused the review with HTTP 402` or names the quota. This is + not an auth error, even when the provider's message mentions the API key. Relay Codex's line + and tell the user: "Codex refused the call because its budget or quota is used up. Check the + allowance on the proxy or gateway in front of the provider, or the account's quota and + credits; logging in again will not fix it." +- **Auth error:** Codex prints an auth error to stderr (401 or 403). Surface the error: "Codex authentication failed. Run `codex login` in your terminal to authenticate via ChatGPT." - **Timeout (Bash outer gate):** Every Bash gate sits ABOVE its inner wrapper (360s gate over the 330s review wrapper; 600s gate, the tool maximum, over the 540s challenge/consult diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index bf328f6c1..8e53a8f3d 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -545,6 +545,24 @@ for the account or API key on the provider's dashboard. **Expected result.** `CODEX_MODE: ready`, and the review completes. + +### `unavailable: the provider refused the review with HTTP 402 (payment required)` + +**Meaning.** The second-model review (Codex, or Claude Code from a Codex +session) reached its provider, and the provider refused the call with HTTP 402 +Payment Required. The money behind the key is used up: a metering proxy or +gateway in front of the provider hit its allowance, or the account ran out of +credits or quota. The login worked, so this is not an authentication failure, +even when the provider's message mentions the API key. gstack reports outside +coverage as unavailable, never as a pass. + +**Fix.** Check the allowance on the proxy or gateway that sits in front of the +provider (its dashboard or its admin), or the account's quota and credits on +the provider's billing page. Raise it or wait for it to reset, then re-run the +review. Signing in again (`codex login`, `claude auth login`) does not help. + +**Expected result.** The review completes with `OUTSIDE_STATUS: completed`. + ### `CODEX_MODE: unverified` / `MODEL_PROBE_INCONCLUSIVE` diff --git a/lib/claude-code.ts b/lib/claude-code.ts index 28f39ff6f..432193328 100644 --- a/lib/claude-code.ts +++ b/lib/claude-code.ts @@ -4,6 +4,7 @@ import { resolveClaudeCommand, type ClaudeCommand } from './claude-bin'; import { initializeWindowsReviewJob, WindowsReviewSupervisionError } from './claude-code-windows-job'; import { ModelPolicyError, describeSelection, resolvePlanReviewModel, selectionRepair, type ModelSelection } from './model-policy'; import { emitModelPolicyNotice } from './model-policy-notice'; +import { BUDGET_FAILURE, QUOTA_FAILURE } from './outside-review-result'; export const CLAUDE_CODE_OUTPUT_LIMIT = 32 * 1024 * 1024; const DRAIN_TIMEOUT_MS = 500; @@ -113,8 +114,13 @@ export function parseClaudeCodeResult(stdout: string, stderr: string, exitCode: const obj = isObject(raw) ? raw : undefined; const response = typeof obj?.result === 'string' ? obj.result : typeof obj?.response === 'string' ? obj.response : ''; const actualFailure = exitCode !== 0 || !obj || Boolean(obj.is_error) || !response.trim(); + const failureText = `${diagnostic}\n${response}\n${stdout.slice(0, 4096)}`; + const budget = actualFailure ? `${diagnostic}\n${obj ? response : stdout.slice(0, 4096)}`.split(/\r?\n/).map(line => line.trim()).find(line => BUDGET_FAILURE.test(line) || QUOTA_FAILURE.test(line)) : undefined; let result: ClaudeCodeResult; - if (actualFailure && /\b(?:authentication(?:[_ ](?:failed|error))?|unauthorized|not authenticated|invalid (?:x-)?api[- ]key|login required|please (?:run .*login|log in)|not logged in)\b/i.test(`${diagnostic}\n${response}\n${stdout.slice(0, 4096)}`)) { + if (budget) { + const cause = BUDGET_FAILURE.test(budget) ? 'its budget is used up (HTTP 402, payment required)' : 'its usage limit or quota is used up'; + result = failure('budget', `Claude Code's provider refused the call because ${cause}: ${budget.slice(0, 240)}. Check the allowance on the proxy or gateway in front of the provider, or the account's quota and credits; logging in again will not fix it.`, true); + } else if (actualFailure && /\b(?:authentication(?:[_ ](?:failed|error))?|unauthorized|not authenticated|invalid (?:x-)?api[- ]key|login required|please (?:run .*login|log in)|not logged in)\b/i.test(failureText)) { result = failure('authentication', 'Claude Code authentication failed. Run claude interactively in this execution context to authenticate.', true); } else if (exitCode !== 0) { result = failure('exit', `Claude Code exited with ${exitCode === null ? 'a signal' : `code ${exitCode}`}.${response ? ` ${response.slice(0, 4096)}` : ''}`); @@ -139,6 +145,7 @@ export function parseClaudeCodeResult(stdout: string, stderr: string, exitCode: } result.exit_code = exitCode; if (diagnostic) result.stderr = diagnostic; + if (result.error?.code === 'budget') result.stderr = [diagnostic, `ERROR: ${result.error.message}`].filter(Boolean).join('\n'); return result; } diff --git a/lib/gate-outcomes.ts b/lib/gate-outcomes.ts index 5ba6cbc1a..9ff9fd534 100644 --- a/lib/gate-outcomes.ts +++ b/lib/gate-outcomes.ts @@ -93,6 +93,11 @@ export const GATE_OUTCOMES = { summary: "Codex refused the call for this account's usage limit", fix: "wait for the reset time in Codex's message above, or add credits or a higher plan for that account; the model choice is fine. To re-check before gstack's 15-minute cache expires, set GSTACK_CODEX_PROBE_RETRY=1 (https://github.com/garrytan/gstack/blob/main/docs/troubleshooting.md#codex-quota-exhausted)", }, + budget_exhausted: { + state: 'unavailable', anchor: 'outside-review-budget-exhausted', + summary: 'the provider refused the review with HTTP 402 (payment required): the budget, allowance or quota behind this key is used up', + fix: "check the allowance on the proxy or gateway in front of the provider, or the account's quota and credits, then re-run the review; signing in again will not help (https://github.com/garrytan/gstack/blob/main/docs/troubleshooting.md#outside-review-budget-exhausted)", + }, rate_limited: { state: 'unavailable', anchor: 'codex-rate-limited', summary: 'Codex rate-limited the review (HTTP 429), so it did not complete', diff --git a/lib/outside-review-result.ts b/lib/outside-review-result.ts index 6824a9acb..dbd6db715 100644 --- a/lib/outside-review-result.ts +++ b/lib/outside-review-result.ts @@ -66,6 +66,8 @@ function codexErrorLines(stderr: string): string[] { } /** The account's usage limit: Codex's line carries the reset time, so it is relayed verbatim. Same signature as the probe. */ export const QUOTA_FAILURE = /usage limit|insufficient_quota|exceeded your current quota|quota exceeded/i; +/** HTTP 402 Payment Required: a metering proxy's allowance or the account's budget, never an auth failure. */ +export const BUDGET_FAILURE = /(?:^|[^0-9])402(?:[^0-9]|$)|payment.?required/i; /** OpenAI's cyber-security refusal (R95): Capy relays it as `OpenAI key declined the request: cyber_policy: ...`; read from error lines only. */ export const CYBER_POLICY_REFUSAL = /\bcyber_policy\b/; /** A plain HTTP 429 or rate limit: transient, so a separate reason from the quota. */ @@ -88,6 +90,19 @@ function commandEvidence(events: string): { attempted: number; succeeded: number return { attempted, succeeded, failedOutput }; } +/** Codex's own failure events under `--json` (`error`, `turn.failed`): its stderr carries no error line for an HTTP refusal. */ +function failureEventMessages(events: string): string[] { + const messages: string[] = []; + for (const line of events.split(/\r?\n/)) { + if (!line.trim().startsWith('{')) continue; + let event: any; + try { event = JSON.parse(line); } catch { continue; } + const message = event?.type === 'error' ? event.message : event?.type === 'turn.failed' ? event.error?.message : undefined; + if (typeof message === 'string' && message.trim()) messages.push(message.trim()); + } + return messages; +} + function stderrHead(stderr: string): string | undefined { return stderr.split(/\r?\n/).map(line => line.trim()).find(Boolean)?.slice(0, 240); } @@ -96,19 +111,27 @@ function execution(input: OutsideReviewInput): OutsideReviewClassification['exec const stderr = input.stderr ?? ''; const exit = input.exit ?? 0; const sandbox = (output: string) => output.match(SANDBOX_FAILURE)?.[0].trim().slice(0, 240); - const cyber = (input.providerError ?? '').split(/\r?\n/).map(line => line.trim()).find(line => CYBER_POLICY_REFUSAL.test(line)); + const hostErrors = (input.providerError ?? '').split(/\r?\n/).map(line => line.trim()); + const cyber = hostErrors.find(line => CYBER_POLICY_REFUSAL.test(line)); if (cyber) return { state: 'unavailable', reason: 'cyber_policy_refused', detail: cyber.slice(0, 240) }; + const hostBudget = hostErrors.find(line => BUDGET_FAILURE.test(line)); + if (hostBudget) return { state: 'unavailable', reason: 'budget_exhausted', detail: hostBudget.slice(0, 240) }; + const hostQuota = hostErrors.find(line => QUOTA_FAILURE.test(line)); + if (hostQuota) return { state: 'unavailable', reason: 'quota_exhausted', detail: hostQuota.slice(0, 240) }; if (exit !== 0) { const detail = sandbox(stderr); if (detail) return { state: 'unavailable', reason: 'sandbox_unavailable', detail }; - const errors = exit !== 124 ? codexErrorLines(stderr) : []; + const failures = exit !== 124 ? failureEventMessages(input.events ?? '') : []; + const errors = exit !== 124 ? [...codexErrorLines(stderr), ...failures] : []; const refusal = errors.find(line => CYBER_POLICY_REFUSAL.test(line)); if (refusal) return { state: 'unavailable', reason: 'cyber_policy_refused', detail: refusal.slice(0, 240) }; + const budget = errors.find(line => BUDGET_FAILURE.test(line)); + if (budget) return { state: 'unavailable', reason: 'budget_exhausted', detail: budget.slice(0, 240) }; const quota = errors.find(line => QUOTA_FAILURE.test(line)); if (quota) return { state: 'unavailable', reason: 'quota_exhausted', detail: quota.slice(0, 240) }; const rateLimit = errors.find(line => RATE_LIMIT_FAILURE.test(line)); if (rateLimit) return { state: 'unavailable', reason: 'rate_limited', detail: rateLimit.slice(0, 240) }; - const head = stderrHead(stderr); + const head = failures[failures.length - 1]?.slice(0, 240) ?? stderrHead(stderr); return { state: 'unavailable', reason: exit === 124 ? 'timeout' : 'execution_failed', detail: head ? `exit ${exit}: ${head}` : `exit ${exit}` }; } const commands = commandEvidence(input.events ?? ''); diff --git a/lib/outside-voice-runner.ts b/lib/outside-voice-runner.ts index c85c657f7..8caafee92 100644 --- a/lib/outside-voice-runner.ts +++ b/lib/outside-voice-runner.ts @@ -142,16 +142,22 @@ class CodexCliRunner implements OutsideRunner { } const eventsFile = `${req.outFile}.events.jsonl`; const stderrFile = `${req.outFile}.stderr`; + const lastMessageFile = `${req.outFile}.last-message`; fs.mkdirSync(path.dirname(req.outFile), { recursive: true }); fs.rmSync(req.outFile, { force: true }); + fs.rmSync(lastMessageFile, { force: true }); const argv = ['codex', 'exec', '-', '-C', req.cwd, '-s', sandbox, '-c', `model="${selected}"`, '-c', 'skills.include_instructions=false', - '-c', 'model_reasoning_effort="high"', '-c', 'web_search="cached"', '--json', '-o', req.outFile]; + '-c', 'model_reasoning_effort="high"', '-c', 'web_search="cached"', '--json', '-o', lastMessageFile]; this.child = Bun.spawn(argv, { cwd: req.cwd, env, stdin: new Blob([prompt]), stdout: Bun.file(eventsFile), stderr: Bun.file(stderrFile) }); const timer = setTimeout(() => { this.child?.kill('SIGTERM'); setTimeout(() => this.child?.kill('SIGKILL'), 10_000).unref(); }, req.timeoutMs); const exit = await this.child.exited; clearTimeout(timer); const timedOut = Date.now() - started >= req.timeoutMs && exit !== 0; - const text = fs.existsSync(req.outFile) ? fs.readFileSync(req.outFile, 'utf8') : ''; + const own = fs.existsSync(req.outFile) ? fs.readFileSync(req.outFile, 'utf8') : ''; + const last = fs.existsSync(lastMessageFile) ? fs.readFileSync(lastMessageFile, 'utf8') : ''; + const marked = (candidate: string) => candidate.trim() !== '' && !['missing_markers', 'empty_response'].includes(classifyOutsideReview({ text: candidate, gate: req.gate }).reason ?? ''); + const text = marked(own) ? own : marked(last) || !own.trim() ? last : own; + if (text !== own) fs.writeFileSync(req.outFile, text); const events = fs.existsSync(eventsFile) ? fs.readFileSync(eventsFile, 'utf8') : ''; const stderr = fs.existsSync(stderrFile) ? fs.readFileSync(stderrFile, 'utf8') : ''; const usage = usageFromEvents(events); diff --git a/test/claude-code-runner.test.ts b/test/claude-code-runner.test.ts index acb3433c5..8fe4c1eb6 100644 --- a/test/claude-code-runner.test.ts +++ b/test/claude-code-runner.test.ts @@ -82,6 +82,12 @@ if (mode === 'overflow') { for(let i=0;i<18;i++) { process.stdout.write(block); process.stderr.write(block); } await new Promise(() => {}); } +if (mode === 'payment' || mode === 'http401' || mode === 'http401-402-fields' || mode === 'usage-limit') { + const reply = mode === 'payment' ? 'API Error: 402 Payment Required {"error":{"message":"API key allowance exhausted"}}' : mode === 'usage-limit' ? 'Claude AI usage limit reached|1763600000' : 'API Error: 401 {"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}'; + const fields = mode === 'http401-402-fields' ? { session_id: '5c1e-402a-9b7f', duration_ms: 402, total_cost_usd: 0.402 } : { session_id: 'session-123' }; + process.stdout.write(JSON.stringify({ type: 'result', is_error: true, result: reply, ...fields })); + process.exit(1); +} if (mode === 'malformed') { process.stdout.write('broken JSON'); process.exit(0); } if (mode === 'array') { process.stdout.write('[]'); process.exit(0); } if (mode === 'null') { process.stdout.write('null'); process.exit(0); } @@ -276,6 +282,25 @@ describe('Claude Code restricted execution', () => { }); } + test('a 402 from the provider is a budget stop, not a login; a 401 stays an authentication failure', async () => { + const budget = await run('payment'); + expect([budget.status, budget.error?.code]).toEqual(['unavailable', 'budget']); + expect(budget.error?.message).toContain('API Error: 402 Payment Required'); + expect(budget.error?.message).toContain('allowance'); + expect(budget.error?.message).not.toMatch(/log ?in|authenticat/i); + const auth = await run('http401'); + expect([auth.status, auth.error?.code]).toEqual(['unavailable', 'authentication']); + }); + + test('a 401 whose result JSON carries 402 in other fields stays authentication; a usage limit is not called a 402', async () => { + const auth = await run('http401-402-fields'); + expect([auth.status, auth.error?.code]).toEqual(['unavailable', 'authentication']); + const quota = await run('usage-limit'); + expect([quota.status, quota.error?.code]).toEqual(['unavailable', 'budget']); + expect(quota.error?.message).toContain('Claude AI usage limit reached'); + expect(quota.error?.message).not.toMatch(/402|payment required/i); + }); + test('the combined stdout/stderr limit terminates a noisy CLI', async () => { const result = await run('overflow'); expect(result.status).toBe('error'); diff --git a/test/model-catalog-isolation.test.ts b/test/model-catalog-isolation.test.ts index 83c6ed112..0b2944cf0 100644 --- a/test/model-catalog-isolation.test.ts +++ b/test/model-catalog-isolation.test.ts @@ -11,6 +11,7 @@ test('replacing the shipped catalog changes role choices, not eval rulers, overl const files = [ 'lib/model-catalog.ts', 'lib/model-policy.ts', 'lib/model-policy-notice.ts', 'lib/state-root.ts', 'lib/eval-model.ts', 'lib/claude-code.ts', 'lib/claude-bin.ts', 'lib/claude-code-windows-job.ts', + 'lib/outside-review-result.ts', 'lib/autoplan-reconcile.ts', 'lib/gate-outcomes.ts', 'lib/design-catalog.ts', 'scripts/resolve-codex-generation-model.ts', 'scripts/models.ts', 'scripts/resolvers/constants.ts', 'scripts/resolvers/model-overlay.ts', ...fs.readdirSync(path.join(source, 'hosts')).filter(file => file.endsWith('.ts')).map(file => `hosts/${file}`), diff --git a/test/outside-review-result.test.ts b/test/outside-review-result.test.ts index 11b806595..53c10b73d 100644 --- a/test/outside-review-result.test.ts +++ b/test/outside-review-result.test.ts @@ -599,3 +599,81 @@ describe('a mid-run Codex usage limit is unavailable (quota_exhausted); a mid-ru } finally { fs.rmSync(home, { recursive: true, force: true }); } }); }); + +describe('an HTTP 402 or payment-required refusal is unavailable (budget_exhausted), never an auth failure', () => { + const PAYMENT_402 = 'ERROR: unexpected status 402 Payment Required: {"error":{"message":"API key allowance exhausted for this billing period","type":"payment_required"}}'; + const STREAM_402 = 'stream error: exceeded retry limit, last status: 402 Payment Required'; + const BODY_ONLY = 'ERROR: {"type":"error","error":{"type":"payment_required","message":"Proxy allowance used up for this API key"}}'; + const AUTH_401 = 'ERROR: unexpected status 401 Unauthorized: {"error":{"message":"Incorrect API key provided","code":"invalid_api_key"}}'; + const AUTH_403 = 'ERROR: unexpected status 403 Forbidden: {"error":{"message":"This API key is not allowed to use this project","type":"permission_error"}}'; + const ECHOED = 'user\nHandle the 402 Payment Required answer from the billing proxy in this diff\n'; + + test('a failed call with a 402 line -> unavailable, reason budget_exhausted, detail is the line', () => { + for (const line of [PAYMENT_402, STREAM_402, BODY_ONLY]) { + const result = classifyOutsideReview({ text: '', stderr: `Reading prompt from stdin...\n${line}\n`, exit: 1, gate: 'review' }); + expect({ line, verdict: result.verdict, reason: result.reason, detail: result.detail }).toEqual({ line, verdict: 'unavailable', reason: 'budget_exhausted', detail: line }); + } + }); + + test('the printed line names the budget, says what to check and never says to log in', () => { + const line = gateOutcomeLine('Codex outside review', 'budget_exhausted', PAYMENT_402); + expect(line).toContain('HTTP 402'); + expect(line).toContain('allowance'); + expect(line).toContain('quota'); + expect(line).toContain('https://github.com/garrytan/gstack/blob/main/docs/troubleshooting.md#outside-review-budget-exhausted'); + expect(line).not.toMatch(/codex login|claude auth login|authentication failed/i); + }); + + test('401 and 403 controls are not a budget stop: they stay execution_failed with the status in the detail', () => { + for (const [line, status] of [[AUTH_401, '401'], [AUTH_403, '403']] as const) { + const result = classifyOutsideReview({ text: '', stderr: `${line}\n`, exit: 1, gate: 'review' }); + expect({ status, reason: result.reason }).toEqual({ status, reason: 'execution_failed' }); + expect(result.detail).toContain(status); + } + }); + + test('an echoed prompt that names a 402 stays execution_failed, and a timeout keeps its reason', () => { + expect(classifyOutsideReview({ text: '', stderr: `${ECHOED}ERROR: unexpected status 500 Internal Server Error\n`, exit: 1, gate: 'review' }).reason).toBe('execution_failed'); + expect(classifyOutsideReview({ text: '', stderr: `${PAYMENT_402}\n`, exit: 124, gate: 'review' }).reason).toBe('timeout'); + }); + + test('a host error for a dispatched reviewer (a Capy subagent) that names a 402 or a quota is the budget or quota stop', () => { + const budget = classifyOutsideReview({ text: '', gate: 'review', exit: 1, providerError: 'OpenAI key declined the request: 402 Payment Required: allowance exhausted' }); + expect([budget.verdict, budget.reason]).toEqual(['unavailable', 'budget_exhausted']); + const quota = classifyOutsideReview({ text: '', gate: 'review', exit: 1, providerError: 'OpenAI key declined the request: insufficient_quota: You exceeded your current quota' }); + expect(quota.reason).toBe('quota_exhausted'); + const auth = classifyOutsideReview({ text: '', gate: 'review', exit: 1, providerError: 'OpenAI key declined the request: 401 Unauthorized: invalid_api_key' }); + expect(auth.reason).toBe('execution_failed'); + }); + + test('the readiness probe does not stop on a 402: the review still runs once and its own result names the cause', () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-budget-probe-')); + try { + const stubDir = path.join(home, 'bin'); + const codexHome = path.join(home, '.codex'); + fs.mkdirSync(stubDir); + fs.mkdirSync(codexHome); + fs.writeFileSync(path.join(codexHome, 'auth.json'), '{}'); + fs.writeFileSync(path.join(stubDir, 'codex'), '#!/usr/bin/env bash\nprintf \'%s\\n\' "$STUB_STDERR" >&2\nexit 1\n', { mode: 0o755 }); + const probe = spawnSync('bash', ['-c', `source "${path.join(ROOT, 'bin', 'gstack-codex-probe')}" && _gstack_codex_model_probe`], { + encoding: 'utf8', timeout: 20_000, + env: { PATH: `${stubDir}:${process.env.PATH ?? ''}`, HOME: home, CODEX_HOME: codexHome, GSTACK_HOME: path.join(home, '.gstack'), STUB_STDERR: PAYMENT_402, _TEL: 'off' }, + }); + expect(probe.status).toBe(0); + expect(probe.stdout).toContain('MODEL_PROBE_INCONCLUSIVE'); + } finally { fs.rmSync(home, { recursive: true, force: true }); } + }); +}); + +describe('the execution_failed detail names the terminal Codex failure', () => { + test('a transient reconnect error before turn.failed does not become the detail', () => { + const fatal = 'unexpected status 500 Internal Server Error, url: http://127.0.0.1:41000/v1/responses'; + const events = [ + JSON.stringify({ type: 'error', message: 'Reconnecting... 1/5 (stream disconnected before completion)' }), + JSON.stringify({ type: 'turn.failed', error: { message: fatal } }), + ].join('\n'); + const result = classifyOutsideReview({ text: '', gate: 'review', exit: 1, stderr: 'Reading additional input from stdin...', events }); + expect([result.verdict, result.reason]).toEqual(['unavailable', 'execution_failed']); + expect(result.detail).toBe(`exit 1: ${fatal}`); + }); +}); diff --git a/test/outside-runner-contract.test.ts b/test/outside-runner-contract.test.ts index 1510f3292..e2bcceca3 100644 --- a/test/outside-runner-contract.test.ts +++ b/test/outside-runner-contract.test.ts @@ -12,6 +12,7 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { createRunner, familyConflict, modelFamily, runnerTable, selectRunner, usageFromEvents } from '../lib/outside-voice-runner'; +import { RESULT_FORMAT } from '../lib/autoplan-prompts'; const ROOT = path.resolve(import.meta.dir, '..'); const BIN = path.join(ROOT, 'bin', 'gstack-outside-voice'); @@ -19,6 +20,8 @@ const TMP = fs.mkdtempSync(path.join(os.tmpdir(), 'outside-runner-')); afterAll(() => fs.rmSync(TMP, { recursive: true, force: true })); const REVIEW = `INPUT: ceo abc\n\n[P2] The install check lies by omission.\n\nRecommendation: revise because the premise is unmeasured.\nVERDICT: FINDINGS\n`; +const SHA = 'a'.repeat(64); +const AUTOPLAN_REVIEW = `INPUT: ceo ${SHA}\n\nThe premise is unmeasured, so the plan may solve the wrong problem.\n\n\`\`\`gstack-findings\n{"id":"F1","severity":"High","title":"premise unmeasured","fix":"measure the install check first"}\n\`\`\`\n`; const promptFile = path.join(TMP, 'prompt.md'); fs.writeFileSync(promptFile, 'You are the outside CEO reviewer.\n'); @@ -143,7 +146,12 @@ out=""; while [ $# -gt 0 ]; do case "$1" in -o) out="$2"; shift;; esac; shift; d printf '%s\\n' "$FAKE_ARGV_SINK" >/dev/null cat >"$out.prompt" if [ -n "\${FAKE_CODEX_SLEEP:-}" ]; then sleep "$FAKE_CODEX_SLEEP"; fi -cat "$FAKE_CODEX_REVIEW" >"$out" +if [ -n "\${FAKE_CODEX_EVENTS:-}" ]; then cat "$FAKE_CODEX_EVENTS"; printf '%s\\n' "\${FAKE_CODEX_STDERR:-}" >&2; : >"$out"; exit 1; fi +if [ -n "\${FAKE_CODEX_OWN:-}" ]; then + own=$(sed -n 's/^3\\. Write the whole result to this file: //p' "$out.prompt") + cat "$FAKE_CODEX_REVIEW" >"$own" + printf '%s\\n' "\${FAKE_CODEX_LAST:-Wrote the review to the result file.}" >"$out" +else cat "$FAKE_CODEX_REVIEW" >"$out"; fi echo '{"type":"item.completed","item":{"type":"command_execution","status":"completed","exit_code":0}}' echo '{"type":"turn.completed","usage":{"input_tokens":1200,"output_tokens":340}}' `, { mode: 0o755 }); @@ -158,11 +166,63 @@ echo '{"type":"turn.completed","usage":{"input_tokens":1200,"output_tokens":340} const result = await runner.run({ promptFile, outFile: out, model: 'gpt-5.5', timeoutMs: 30_000, cwd: ROOT, gate: 'review', env, binDir: gstackBin }); expect(result).toMatchObject({ runner: 'codex-cli', status: 'completed', model: 'gpt-5.5', family: 'openai', exit: 0, usage: { input_tokens: 1200, output_tokens: 340, usd: 'unknown' } }); expect(result.verdict!.verdict).toBe('clean'); - expect(fs.readFileSync(`${out}.prompt`, 'utf8')).toBe(fs.readFileSync(promptFile, 'utf8')); + expect(fs.readFileSync(`${out}.last-message.prompt`, 'utf8')).toBe(fs.readFileSync(promptFile, 'utf8')); expect(fs.readFileSync(out, 'utf8')).toBe(REVIEW); expect(usageFromEvents('{"usage":{"input_tokens":5}}\nnot json\n')).toEqual({ input_tokens: 5, usd: 'unknown' }); }); + test('the reviewer\'s own result file survives Codex\'s closing message (-o goes to a separate file)', async () => { + const review = path.join(TMP, 'own-review.md'); + fs.writeFileSync(review, AUTOPLAN_REVIEW); + const out = path.join(TMP, 'own', 'ceo-outside.md'); + const prompt = path.join(TMP, 'own-prompt.md'); + fs.writeFileSync(prompt, `You are the outside CEO reviewer.\n${RESULT_FORMAT('ceo', SHA, out)}`); + const result = await createRunner('codex-cli').run({ promptFile: prompt, outFile: out, timeoutMs: 30_000, cwd: ROOT, gate: 'autoplan', env: { ...cleanEnv(fakeBin), FAKE_CODEX_REVIEW: review, FAKE_CODEX_OWN: '1' }, binDir: gstackBin }); + expect(fs.readFileSync(out, 'utf8')).toBe(AUTOPLAN_REVIEW); + expect(fs.readFileSync(`${out}.last-message`, 'utf8')).toBe('Wrote the review to the result file.\n'); + expect(result.reason).toBeUndefined(); + expect(result.status).toBe('completed'); + }); + + test('when the reviewer\'s own file lacks the gate\'s markers and the closing message has them, the closing message is the result', async () => { + const review = path.join(TMP, 'own-junk.md'); + fs.writeFileSync(review, 'draft notes, not the result\n'); + const out = path.join(TMP, 'own-junk', 'ceo-outside.md'); + const prompt = path.join(TMP, 'own-junk-prompt.md'); + fs.writeFileSync(prompt, `You are the outside CEO reviewer.\n${RESULT_FORMAT('ceo', SHA, out)}`); + const result = await createRunner('codex-cli').run({ promptFile: prompt, outFile: out, timeoutMs: 30_000, cwd: ROOT, gate: 'autoplan', env: { ...cleanEnv(fakeBin), FAKE_CODEX_REVIEW: review, FAKE_CODEX_OWN: '1', FAKE_CODEX_LAST: AUTOPLAN_REVIEW.trimEnd() }, binDir: gstackBin }); + expect(fs.readFileSync(out, 'utf8')).toBe(AUTOPLAN_REVIEW); + expect(result.status).toBe('completed'); + }); + + // Captured from codex-cli 0.170.0 against a local stub answering 402 (application/problem+json, a metering proxy's allowance) and 401. + const REFUSAL_EVENTS = (status: string) => [ + '{"type":"thread.started","thread_id":"01a1f2ae-a04c-7583-8012-638cc20a388d"}', + '{"type":"turn.started"}', + JSON.stringify({ type: 'error', message: status }), + JSON.stringify({ type: 'turn.failed', error: { message: status } }), + ].join('\n') + '\n'; + const STATUS_402 = 'unexpected status 402 Payment Required: {"type":"https://meter.example/problems/allowance","title":"E_ALLOWANCE","status":402,"code":"E_ALLOWANCE","detail":"E_ALLOWANCE: metering proxy: the slot\'s allowance of $15 cannot hold this call\'s worst case"}, url: http://127.0.0.1:33037/v1/responses'; + const STATUS_401 = 'unexpected status 401 Unauthorized: Incorrect API key provided, url: http://127.0.0.1:45349/v1/responses'; + const STDERR = 'Reading additional input from stdin...'; + + test('a 402 in Codex\'s JSON failure events is recorded as budget_exhausted with the cause, not "exit 1"', async () => { + const events = path.join(TMP, 'refusal-402.jsonl'); + fs.writeFileSync(events, REFUSAL_EVENTS(STATUS_402)); + const result = await createRunner('codex-cli').run({ promptFile, outFile: path.join(TMP, 'r402', 'o.md'), timeoutMs: 30_000, cwd: ROOT, gate: 'review', env: { ...cleanEnv(fakeBin), FAKE_CODEX_EVENTS: events, FAKE_CODEX_STDERR: STDERR }, binDir: gstackBin }); + expect([result.status, result.exit, result.reason]).toEqual(['unavailable', 1, 'budget_exhausted']); + expect(result.detail).toStartWith('unexpected status 402 Payment Required'); + expect(result.detail).toContain('E_ALLOWANCE'); + }); + + test('a 401 in the same events stays execution_failed and keeps the 401 cause in its detail', async () => { + const events = path.join(TMP, 'refusal-401.jsonl'); + fs.writeFileSync(events, REFUSAL_EVENTS(STATUS_401)); + const result = await createRunner('codex-cli').run({ promptFile, outFile: path.join(TMP, 'r401', 'o.md'), timeoutMs: 30_000, cwd: ROOT, gate: 'review', env: { ...cleanEnv(fakeBin), FAKE_CODEX_EVENTS: events, FAKE_CODEX_STDERR: STDERR }, binDir: gstackBin }); + expect([result.status, result.reason]).toEqual(['unavailable', 'execution_failed']); + expect(result.detail).toBe(`exit 1: ${STATUS_401}`); + }); + test('cancel() terminates the child and reports cancelled', async () => { const review = path.join(TMP, 'codex-review-2.md'); fs.writeFileSync(review, REVIEW); diff --git a/test/outside-voice-failure-policy.test.ts b/test/outside-voice-failure-policy.test.ts index 5b1c510ad..6117d5794 100644 --- a/test/outside-voice-failure-policy.test.ts +++ b/test/outside-voice-failure-policy.test.ts @@ -13,6 +13,7 @@ import * as path from 'node:path'; import { ALL_HOST_CONFIGS } from '../hosts/index'; import { outsideVoiceFailurePolicy, outsideVoiceFor, type OutsideVoiceFailurePolicyOptions } from '../scripts/resolvers/outside-voice'; import { generateAdversarialStep } from '../scripts/resolvers/outside-voice-steps'; +import { gateOutcomeLine } from '../lib/gate-outcomes'; import { HOST_PATHS, type TemplateContext } from '../scripts/resolvers/types'; const ROOT = path.resolve(import.meta.dir, '..'); @@ -195,3 +196,18 @@ describe('ratchet (d): outside-voice failure prose has one owner', () => { expect(problems).toEqual([`${ALLOWLIST} entry for codex/SKILL.md.tmpl has no reason: ${allowlist[0].text}`]); }); }); + +describe('a 402 or quota stop is never reported as an auth failure', () => { + test('the validator\'s budget line does not itself trip the auth rule\'s keywords', () => { + const line = gateOutcomeLine('Codex outside review', 'budget_exhausted'); + expect(line).not.toMatch(/auth|login|unauthorized|API key/i); + expect(line).toContain('HTTP 402'); + }); + + test('the /codex skill\'s own error handling names the budget stop before its auth error', () => { + const text = fs.readFileSync(path.join(ROOT, 'codex/SKILL.md'), 'utf8'); + const budget = text.indexOf('- **Budget or quota stop (HTTP 402, payment required, quota):**'); + expect(budget).toBeGreaterThan(0); + expect(budget).toBeLessThan(text.indexOf('- **Auth error:**')); + }); +}); diff --git a/test/outside-voice-invocation.test.ts b/test/outside-voice-invocation.test.ts index 4b18f5725..f94465ee8 100644 --- a/test/outside-voice-invocation.test.ts +++ b/test/outside-voice-invocation.test.ts @@ -478,3 +478,49 @@ echo 'Recommendation: approve because the late answer arrived.' } }); }); + +describe('a 402 budget stop reads as a budget stop on both outside voices; 401 and 403 do not', () => { + const stderrFile = (name: string, text: string) => { const p = path.join(TMP, name); fs.writeFileSync(p, `${text}\n`); return p; }; + const BUDGET = 'outside review unavailable: the provider refused the review with HTTP 402'; + + test('Codex voice: a metering proxy\'s 402 is unavailable with the budget line and no login advice', () => { + const result = invoke('claude', {}, { FAKE_MODE: 'nonzero', FAKE_STDERR_FILE: stderrFile('codex-402.stderr', 'ERROR: unexpected status 402 Payment Required: {"error":{"message":"API key allowance exhausted"}}') }); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain(`Codex ${BUDGET}`); + expect(result.stderr).toContain('#outside-review-budget-exhausted'); + expect(result.stderr).not.toContain('codex login'); + expect(result.stdout).not.toContain('OUTSIDE_STATUS: completed'); + }); + + test('Codex voice: a 402 that appears only in Codex\'s JSON failure events (its stderr names no cause) is the budget line', () => { + const message = 'unexpected status 402 Payment Required: {"title":"E_ALLOWANCE","status":402,"code":"E_ALLOWANCE","detail":"E_ALLOWANCE: metering proxy: the slot\'s allowance of $15 cannot hold this call\'s worst case"}'; + const events = stderrFile('codex-402.events.jsonl', [JSON.stringify({ type: 'turn.started' }), JSON.stringify({ type: 'error', message }), JSON.stringify({ type: 'turn.failed', error: { message } })].join('\n')); + const result = invoke('claude', {}, { FAKE_MODE: 'nonzero', FAKE_EVENTS_FILE: events, FAKE_STDERR_FILE: stderrFile('codex-402-plain.stderr', 'Reading additional input from stdin...') }); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain(`Codex ${BUDGET}`); + expect(result.stderr).toContain('E_ALLOWANCE'); + }); + + for (const [status, line] of [ + ['401', 'ERROR: unexpected status 401 Unauthorized: {"error":{"message":"Incorrect API key provided","code":"invalid_api_key"}}'], + ['403', 'ERROR: unexpected status 403 Forbidden: {"error":{"message":"This API key is not allowed to use this project"}}'], + ] as const) { + test(`Codex voice: a ${status} is missing coverage with its status shown, never a budget stop`, () => { + const result = invoke('claude', {}, { FAKE_MODE: 'nonzero', FAKE_STDERR_FILE: stderrFile(`codex-${status}.stderr`, line) }); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain(`Codex outside review unavailable: the reviewer process failed (exit 4: ${line.slice(0, 60)}`); + expect(result.stderr).not.toContain('HTTP 402'); + }); + } + + test('Claude Code voice: a 402 is unavailable with the budget line; a 401 stays an authentication failure', () => { + const budget = invoke('codex', {}, { FAKE_MODE: 'nonzero', FAKE_RESPONSE: 'API Error: 402 Payment Required {"error":{"message":"API key allowance exhausted"}}' }); + expect(budget.status).not.toBe(0); + expect(budget.stderr).toContain(`Claude Code ${BUDGET}`); + expect(budget.stdout).toContain('"code":"budget"'); + const auth = invoke('codex', {}, { FAKE_MODE: 'nonzero', FAKE_RESPONSE: 'API Error: 401 {"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}' }); + expect(auth.status).not.toBe(0); + expect(auth.stdout).toContain('"code":"authentication"'); + expect(auth.stderr).not.toContain('HTTP 402'); + }); +});