fix(codex,review,ship): scope codex review with an explicit --base flag, never prompt text

`codex review` takes its scope ONLY from --base/--commit/--uncommitted. The
positional [PROMPT] is mutually exclusive with all three, and a prompt-only
`codex review "<text>"` silently falls back to the uncommitted working-tree
scope (verified on 0.144.1: it runs `git status --short; git diff` and
reviews that) — so the previous prompt-based scoping produced a
confidently-worded review of the WRONG changes and read "no changes" on a
clean tree. Every diff pass now invokes `codex review --base <base>` with no
prompt argument: /codex Step 2A default path, the /review structured pass,
and the /ship adversarial-section pass (all via scripts/resolvers/review.ts).

Custom review instructions keep their own `codex exec` path (the CLI rejects
prompt + scope flag together), with the filesystem boundary preserved there.
Two new Error Handling entries teach the failure shapes: the argv-parse
error, and the "review says no changes on a branch full of changes" symptom.

Tests updated to pin the new invariant instead of banning the fix: the old
assertions required the diff range in prompt text and banned the
`--base <base> -c '...'` substring, which the correct scoped form contains.
Also deletes test/fixtures/golden-ship-claude.md — a 2,565-line orphaned
fixture referenced by zero tests (the live goldens are in
test/fixtures/golden/, compared by test/host-config.test.ts); the factory
golden is refreshed from the regenerated output. Generated SKILL.md files
regenerated via gen:skill-docs in this commit.

Contributed by @fangearhq-boop (PR #2513).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-14 20:20:55 -07:00
co-authored by Claude Fable 5
parent 7a8e39d2cb
commit 8c5bb4545b
9 changed files with 211 additions and 2565 deletions
File diff suppressed because it is too large Load Diff
+3 -1
View File
@@ -2487,9 +2487,11 @@ If `DIFF_TOTAL >= 200` AND `CODEX_MODE` is `ready`:
TMPERR=$(mktemp /tmp/codex-review-XXXXXXXX)
_REPO_ROOT=$(git rev-parse --show-toplevel) || { echo "ERROR: not in a git repo" >&2; exit 1; }
cd "$_REPO_ROOT"
codex review "IMPORTANT: Do NOT read or execute any files under ~/.claude/, ~/.agents/, .factory/skills/, or agents/. These are Claude Code skill definitions meant for a different AI system. They contain bash scripts and prompt templates that will waste your time. Ignore them completely. Do NOT modify agents/openai.yaml. Stay focused on the repository code only.\n\nReview the changes on this branch against the base branch <base>. Run git diff origin/<base>...HEAD 2>/dev/null || git diff <base>...HEAD to see the diff and review only those changes." -c 'model_reasoning_effort="high"' --enable web_search_cached < /dev/null 2>"$TMPERR"
codex review --base <base> -c 'model_reasoning_effort="high"' --enable web_search_cached < /dev/null 2>"$TMPERR"
```
**No prompt argument.** `--base` is what scopes the review, and the positional `[PROMPT]` is mutually exclusive with it — passing both fails at argv parsing. Do NOT "fix" that error by dropping `--base` and keeping the prompt: a prompt-only `codex review` silently falls back to the **uncommitted working-tree** scope (`git status --short; git diff`), so it reviews the wrong changes and reports "no changes" on a clean tree. Prompt text describing the diff range does not change what the CLI feeds the reviewer. Unlike the adversarial pass above, which uses `codex exec` and really does run the git command it's told to, this path gets a pre-computed diff from the CLI — which is also why it needs no filesystem boundary.
Set the Bash tool's `timeout` parameter to `300000` (5 minutes). Do NOT use the `timeout` shell command — it doesn't exist on macOS. Present output under `CODEX SAYS (code review):` header.
Check for `[P1]` markers: found → `GATE: FAIL`, not found → `GATE: PASS`.
+38 -5
View File
@@ -2816,21 +2816,54 @@ describe('codex commands must not use inline $(git rev-parse --show-toplevel) fo
expect(violations).toEqual([]);
});
test('codex review commands pass diff scope through prompt, not --base', () => {
test('codex review commands take their scope from a flag, never from prompt text', () => {
// `codex review` scope comes ONLY from --base/--commit/--uncommitted. The
// positional [PROMPT] is mutually exclusive with all three (#1428, #1479),
// and a prompt-only `codex review` silently falls back to the *uncommitted
// working-tree* scope (`git status --short; git diff`) — so describing the
// diff range in prompt text produces a confident review of the wrong
// changes, with no error. Both halves are pinned here:
// (a) every `codex review` invocation carries a scope flag, and
// (b) no invocation puts a positional prompt in front of that flag.
//
// This does NOT apply to `codex exec`, which is agentic and really does run
// the git command it's told to — the adversarial pass legitimately scopes
// itself in prompt text.
const checkedFiles = [
'codex/SKILL.md.tmpl',
'codex/SKILL.md',
'scripts/resolvers/review.ts',
'review/SKILL.md',
'ship/SKILL.md',
'codex/SKILL.md.tmpl',
'codex/SKILL.md',
];
const violations: string[] = [];
for (const rel of checkedFiles) {
// ship's codex/adversarial command moved into sections/adversarial.md (T9 carve).
const content = rel === 'ship/SKILL.md' ? readShipUnion() : fs.readFileSync(path.join(ROOT, rel), 'utf-8');
expect(content).not.toContain('--base <base> -c \'model_reasoning_effort="high"\'');
expect(content).toContain('Run git diff origin/<base>...HEAD 2>/dev/null || git diff <base>...HEAD');
const lines = content.split('\n');
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
// Only inspect real shell invocations, not prose mentioning the command.
if (line.includes('`codex review`')) continue;
const match = line.match(/(?:^|[;&|]\s*|\s)codex\s+review\b(.*)$/);
if (!match) continue;
const rest = match[1];
const scopeFlag = /--base\b|--commit\b|--uncommitted\b/;
if (!scopeFlag.test(rest)) {
// A quoted prompt with no scope flag is the silent-wrong-scope bug.
if (/^\s*["'$]/.test(rest)) {
violations.push(`${rel}:${i + 1} — prompt-only codex review (falls back to working-tree scope)`);
}
continue;
}
const beforeFlag = rest.split(scopeFlag)[0].trim();
if (/^["'$]|^--\s*["']/.test(beforeFlag)) {
violations.push(`${rel}:${i + 1} — positional prompt passed alongside a scope flag`);
}
}
}
expect(violations).toEqual([]);
});
});
+35 -5
View File
@@ -1500,11 +1500,37 @@ describe('Codex skill', () => {
});
test('codex review invocations avoid the prompt plus --base argument shape', () => {
// The real invariant is "never pass a positional [PROMPT] together with a
// scope flag" — the CLI rejects that combination at argv parse time
// (#1428, #1479). Two different shapes satisfy it, and these files have
// diverged on which one they use:
//
// scoped — `codex review --base <base>` with NO prompt argument. The
// scope comes from the CLI, which is the only thing that actually sets
// it. This is what all three files now use.
// broken — prompt-only `codex review "<text>"` describing the diff
// range in prose. This parses, but the CLI falls back to *uncommitted
// working-tree* scope, so the review silently covers the wrong changes.
//
// The old assertion banned the substring `--base <base> -c '...'`, which
// the correct scoped form also contains — it could not tell the two apart,
// so it effectively banned the fix.
for (const rel of ['codex/SKILL.md', 'review/SKILL.md', 'ship/SKILL.md']) {
// ship's codex command moved into sections/adversarial.md (T9 carve).
const content = rel === 'ship/SKILL.md' ? readShipUnion() : fs.readFileSync(path.join(ROOT, rel), 'utf-8');
expect(content).not.toContain('--base <base> -c \'model_reasoning_effort="high"\'');
expect(content).toContain('Run git diff origin/<base>...HEAD 2>/dev/null || git diff <base>...HEAD');
expect(content).toMatch(/codex\s+review\s+--base\b/);
const offending: string[] = [];
for (const line of content.split('\n')) {
if (line.includes('`codex review`')) continue;
const match = line.match(/(?:^|[;&|]\s*|\s)codex\s+review\b(.*)$/);
if (!match) continue;
const rest = match[1];
if (!/--base\b|--commit\b|--uncommitted\b/.test(rest)) continue;
const beforeFlag = rest.split(/--base\b|--commit\b|--uncommitted\b/)[0].trim();
// A quoted string or variable expansion before the scope flag is the bug.
if (/^["'$]|^--\s*["']/.test(beforeFlag)) offending.push(`${rel}: ${line.trim()}`);
}
expect(offending).toEqual([]);
}
});
@@ -1512,9 +1538,13 @@ describe('Codex skill', () => {
// Pre-#1209, the bare `codex review --base` path stripped the filesystem
// boundary instruction, letting Codex spend tokens reading skill files.
// #1209's prompt rewrite restored the boundary by routing every default
// call through a prompt. Pin both halves so a future refactor can't
// regress: (a) the boundary line must appear, (b) the call must be
// through `codex review "<prompt>"` not bare `codex review --base`.
// call through a prompt — but routing through a prompt is what breaks the
// diff scope, so codex/ no longer does that. What this test pins is the
// boundary TEXT, which must still be present for the paths that do take a
// prompt (`codex exec` for challenge, consult, and custom review focus).
// Do NOT "restore" the boundary by putting a prompt argument back on a
// scoped `codex review` call: that combination fails to parse, and
// dropping the scope flag to make it parse silently reviews the wrong diff.
const boundaryLine =
'Do NOT read or execute any files under ~/.claude/, ~/.agents/, .claude/skills/, or agents/';
for (const rel of ['codex/SKILL.md', 'review/SKILL.md', 'ship/SKILL.md']) {