Second-model plan reviews in autoplan stop being reported unavailable; --help stops printing a fake refusal (GST-3051) (#4471)

A second-model plan review in `/autoplan` is no longer reported as
unavailable when it actually ran, and `gstack-outside-voice --help` no
longer prints a line that reads like a real consent refusal.

Why: sessions believed the false label and paid to re-run the review, or
closed the plan phase without its second-model review.

Nothing to do: the command autoplan already runs now picks the right
check by itself.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-18 02:12:43 +00:00
1 parent 29e950227b
commit c9b6fe0ab3
9 files changed
+159 -18

No files matched your search

+13 -8
View File
@@ -4,7 +4,7 @@
*
* gstack-outside-voice run --runner codex-cli|api|host-subagent --prompt <file> --out <file>
* [--model <id>] [--result <file>] [--native-model <id>] [--fallback <runner>]
* [--gate review|structured|spec|proposal] [--timeout-ms N] [--max-tokens N] [--cwd <dir>] [--json]
* [--gate review|structured|spec|proposal|autoplan] [--timeout-ms N] [--max-tokens N] [--cwd <dir>] [--json]
* gstack-outside-voice runners [--model <id>] [--json] the runner table: capabilities and availability here
* gstack-outside-voice family <model-id> the model's family (same-family rule)
*
@@ -17,6 +17,7 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import { EXIT, exitTableHelp } from '../lib/headless-artifacts';
import { RESULT_FORMAT_HEADER } from '../lib/autoplan-reconcile';
import { findingsLabel, type OutsideGate } from '../lib/outside-review-result';
import { RUNNER_IDS, createRunner, familyConflict, modelFamily, outsideStatusLine, runnerTable, selectRunner, type RunnerId } from '../lib/outside-voice-runner';
import { outsideConsent } from '../lib/outside-consent';
@@ -26,7 +27,7 @@ import { resultLine } from '../lib/result-codes';
const HELP = `gstack-outside-voice — one outside voice, three runners (docs/unattended.md)
run --runner <r> --prompt <file> --out <file> [--model <id>] [--result <file>] [--native-model <id>]
[--fallback <r>] [--gate review|structured|spec|proposal] [--timeout-ms N] [--max-tokens N] [--cwd <dir>] [--json]
[--fallback <r>] [--gate review|structured|spec|proposal|autoplan] [--timeout-ms N] [--max-tokens N] [--cwd <dir>] [--json]
runners [--model <id>] [--json] the runner table: id, capabilities, available here
family <model-id> print the model family (anthropic, openai, google, ...)
@@ -35,10 +36,14 @@ OPENAI_API_KEY/ANTHROPIC_API_KEY; needs --model), host-subagent (binds --result
cross-family subagent the host ran; needs --model). --max-tokens caps the api runner's output
(Anthropic default 16384; its default time limit grows with it); a response cut off at the limit
is unavailable (truncated). --native-model enforces the same-family
rule. GSTACK_OUTSIDE_RUNNER names the fallback when --fallback is absent.
Prints OUTSIDE_STATUS: <status> provider=<runner> model=<id> family=<f> and writes <out>.meta.json.
codex_reviews=disabled (or an unreadable config) refuses every runner before it starts:
OUTSIDE_STATUS: not_run reason=consent_missing, exit 3. Enable: gstack-config set codex_reviews enabled.
rule. GSTACK_OUTSIDE_RUNNER names the fallback when --fallback is absent. --gate defaults to
autoplan (its INPUT receipt and gstack-findings fence) when the prompt carries gstack-autoplan's
RESULT FORMAT, else review. Writes <out>.meta.json and prints a status line, for example
OUTSIDE_STATUS: <status> provider=<runner> model=<id> family=<f>
codex_reviews=disabled (or an unreadable config) refuses every runner before it starts, exit 3,
with the status line
OUTSIDE_STATUS: not_run reason=consent_missing
Enable: gstack-config set codex_reviews enabled.
${exitTableHelp()}`;
function flag(args: string[], name: string): string | undefined {
@@ -76,8 +81,8 @@ async function main(argv: string[]): Promise<number> {
if (!isRunner(requested)) return usage(`--runner must be one of ${RUNNER_IDS.join(', ')}`);
if (!promptFile || !outFile) return usage('run needs --prompt <file> --out <file>');
if (!fs.existsSync(promptFile)) return usage(`prompt file not found: ${promptFile}`);
const gate = (flag(rest, '--gate') ?? 'review') as OutsideGate;
if (!['review', 'structured', 'spec', 'proposal'].includes(gate)) return usage('--gate must be review, structured, spec or proposal');
const gate = (flag(rest, '--gate') ?? (fs.readFileSync(promptFile, 'utf8').includes(RESULT_FORMAT_HEADER) ? 'autoplan' : 'review')) as OutsideGate;
if (!['review', 'structured', 'spec', 'proposal', 'autoplan'].includes(gate)) return usage('--gate must be review, structured, spec, proposal or autoplan');
const maxTokensText = flag(rest, '--max-tokens');
const maxTokens = maxTokensText === undefined ? undefined : Number(maxTokensText);
if (maxTokens !== undefined && (!Number.isInteger(maxTokens) || maxTokens <= 0)) return usage('--max-tokens needs a positive integer');
@@ -0,0 +1 @@
- **A second-model plan review in `/autoplan` is no longer reported as unavailable when it actually ran (GST-3051).** `gstack-outside-voice run` judged autoplan's reviews by `/review`'s rule, which wants a `Recommendation: ... because ...` line autoplan never asks for, so a finished review printed `VERDICT: unavailable` (`missing_markers`) and sessions paid to re-run it or dropped it. A prompt that carries autoplan's result format is now checked against that format (its `INPUT:` line and `gstack-findings` fence; `--gate autoplan` asks for it explicitly), and a reply without them, or an empty one, is still unavailable. `gstack-outside-voice --help` also no longer prints a line that starts with `OUTSIDE_STATUS:`, which logs and models read as a real consent refusal.
+6 -1
View File
@@ -386,7 +386,12 @@ it; `gstack-outside-voice run` sends that default to Anthropic only.
**Meaning.** The answer did not contain what the gate checks: a
`Recommendation: ... because ...` line (or a `## Recommendation` heading whose
next non-blank line is a plain sentence with a `because` reason), a
`SCORE:`/`AMBIGUITIES:` pair, or severity tags.
`SCORE:`/`AMBIGUITIES:` pair, autoplan's `INPUT: <phase> <sha256>` line and
`gstack-findings` fence, or severity tags. `gstack-outside-voice run` judges a
prompt that carries autoplan's RESULT FORMAT by that format (`--gate autoplan`),
so an autoplan second-model review that reports `missing review completion
recommendation` came from a gstack older than that fix: read its result file
before re-running it.
**Fix.** Re-run the review.
+2 -1
View File
@@ -13,6 +13,7 @@ import * as fs from 'node:fs';
import * as path from 'node:path';
import { bunIsolationArgs } from './bun-isolation';
import { detectDocsOnly, readScopeMarker, scopeMarker, scopeTokens, type ScopeFlags, type UiEvidence } from './autoplan-scope';
import { RESULT_FORMAT_HEADER } from './autoplan-reconcile';
import { COORDINATOR_CONTRACT } from './coordinator-contract';
import { capyBootstrapCommand } from './doctor-components';
import { DX_ERROR_COMPACT } from './dx-error-contract';
@@ -117,7 +118,7 @@ export function extractHeading(text: string, heading: string): string | undefine
export const RESULT_FORMAT = (phase: string, sha256: string, resultPath: string) => `
---
RESULT FORMAT (required by gstack-autoplan; the parent binds your file by this contract):
${RESULT_FORMAT_HEADER}
1. The first line of your result is exactly: INPUT: ${phase} ${sha256}
2. Write the complete review as prose, then END with one fenced block of canonical findings, one JSON object per line:
\`\`\`gstack-findings
+7 -4
View File
@@ -15,6 +15,8 @@
import type { FindingRow, Severity, Voice } from './headless-artifacts';
export const SEVERITIES: readonly Severity[] = ['Critical', 'High', 'Medium', 'Low', 'Informational'];
/** The RESULT FORMAT block's opening words (lib/autoplan-prompts.ts); a prompt that carries them asks for this contract. */
export const RESULT_FORMAT_HEADER = 'RESULT FORMAT (required by gstack-autoplan; the parent binds your file by this contract):';
const SEVERITY_ALIASES: Record<string, Severity> = {
critical: 'Critical', p0: 'Critical', blocker: 'Critical',
high: 'High', p1: 'High', major: 'High',
@@ -49,8 +51,8 @@ export function severityOf(value: unknown): Severity | undefined {
return SEVERITY_ALIASES[value.trim().toLowerCase()];
}
/** Parse a reviewer's result: the INPUT receipt for this phase and snapshot, then the canonical findings fence(s). */
export function parseReviewResult(text: string, expected: { phase: string; sha256: string }): ParsedReviewResult {
/** Parse a reviewer's result: the INPUT receipt for this phase and snapshot (any, without expected), then the canonical findings fence(s). */
export function parseReviewResult(text: string, expected?: { phase: string; sha256: string }): ParsedReviewResult {
const errors: string[] = [];
const receipt = parseReceipt(text, expected);
if (!receipt.ok) errors.push(receipt.error!);
@@ -59,12 +61,13 @@ export function parseReviewResult(text: string, expected: { phase: string; sha25
return { receipt, findings, fenceFound, errors };
}
/** The full-read receipt: `INPUT: <phase> <sha256>` in the first five non-empty lines, matching this phase and snapshot. */
export function parseReceipt(text: string, expected: { phase: string; sha256: string }): ParsedReviewResult['receipt'] & { present: boolean; error?: string } {
/** The full-read receipt: `INPUT: <phase> <sha256>` in the first five non-empty lines, matching this phase and snapshot when expected is given. */
export function parseReceipt(text: string, expected?: { phase: string; sha256: string }): ParsedReviewResult['receipt'] & { present: boolean; error?: string } {
const head = text.split('\n').map(l => l.trim()).filter(Boolean).slice(0, 5);
const receiptLine = head.find(l => /^INPUT:\s*\S+\s+[0-9a-f]{64}\b/i.test(l));
const m = receiptLine ? /^INPUT:\s*(\S+)\s+([0-9a-f]{64})/i.exec(receiptLine) : null;
if (!m) return { ok: false, present: false, error: 'no INPUT: <phase> <sha256> line in the first five lines' };
if (!expected) return { ok: true, present: true, phase: m[1]!.toLowerCase(), sha256: m[2]!.toLowerCase(), line: receiptLine };
const ok = m[1]!.toLowerCase() === expected.phase && m[2]!.toLowerCase() === expected.sha256.toLowerCase();
return { ok, present: true, phase: m[1]!.toLowerCase(), sha256: m[2]!.toLowerCase(), line: receiptLine,
...(ok ? {} : { error: `receipt names ${m[1]} ${m[2]!.slice(0, 12)}…, expected ${expected.phase} ${expected.sha256.slice(0, 12)}…` }) };
+1 -1
View File
@@ -11,7 +11,7 @@ const BANNER = '<!-- AUTO-GENERATED from';
export const CLAUDE_CODE_RUNTIME_FILES = [
'bin/gstack-claude-code', 'bin/gstack-models', 'bin/gstack-config', 'bin/gstack-lock.sh', 'bin/gstack-state-root.sh',
'lib/claude-code.ts', 'lib/claude-code-windows-job.ts', 'lib/claude-bin.ts',
'lib/outside-review-result.ts', 'lib/gate-outcomes.ts', 'lib/outside-consent.ts', 'lib/state-root.ts',
'lib/outside-review-result.ts', 'lib/autoplan-reconcile.ts', 'lib/gate-outcomes.ts', 'lib/outside-consent.ts', 'lib/state-root.ts',
'lib/model-catalog.ts', 'lib/model-policy.ts', 'lib/model-policy-cli.ts', 'lib/model-policy-notice.ts', 'lib/agent-model.ts',
];
+11 -2
View File
@@ -7,9 +7,11 @@
* the highest finding severity, and the verdict. validateOutsideReview() is the
* older text-only shape, kept for direct importers and stale rendered skills.
*/
import { parseReviewResult } from './autoplan-reconcile';
import { GATE_OUTCOMES, gateOutcomeLine, type GateReason } from './gate-outcomes';
export type OutsideGate = 'review' | 'structured' | 'spec' | 'execution' | 'proposal';
/** autoplan: the INPUT receipt and gstack-findings fence that lib/autoplan-prompts.ts RESULT_FORMAT asks every plan reviewer for. */
export type OutsideGate = 'review' | 'structured' | 'spec' | 'execution' | 'proposal' | 'autoplan';
export type OutsideVerdict = 'clean' | 'findings' | 'unverified' | 'unavailable';
export type Severity = 'P0' | 'P1' | 'P2' | 'P3';
@@ -132,6 +134,7 @@ function execution(input: OutsideReviewInput): OutsideReviewClassification['exec
* Critical and high block like P0/P1; medium and low are advisory like P2/P3.
*/
const SEVERITY_WORDS: Record<string, Severity> = { critical: 'P0', high: 'P1', medium: 'P2', low: 'P3' };
const AUTOPLAN_LEVEL: Record<string, Severity | undefined> = { Critical: 'P0', High: 'P1', Medium: 'P2', Low: 'P3' };
const WORD = '(critical|high|medium|low)';
const SEVERITY_LABELS = [
new RegExp(`\\b(?:severity|priority)\\b[\\t ]*[:=][\\t ]*(?:\\*\\*|__|\\[|\`)*${WORD}\\b`, 'gim'),
@@ -264,6 +267,12 @@ export function classifyOutsideReview(input: OutsideReviewInput): OutsideReviewC
const blocking = findings.highest === 'P0' || findings.highest === 'P1';
const result = (verdict: OutsideVerdict, reason?: GateReason, detail?: string): OutsideReviewClassification =>
({ execution: ran, findings, verdict, ...(reason ? { reason } : {}), ...(detail ? { detail } : {}) });
if (input.gate === 'autoplan') {
const parsed = parseReviewResult(input.text);
if (parsed.errors.length) return result('unavailable', 'missing_markers', `autoplan result: ${parsed.errors.join('; ')}`.slice(0, 240));
const highest = parsed.findings.map(f => AUTOPLAN_LEVEL[f.severity]).filter((s): s is Severity => !!s).sort()[0] ?? null;
return { execution: ran, findings: { highest }, verdict: highest === 'P0' || highest === 'P1' ? 'findings' : 'clean' };
}
if (input.gate === 'spec') {
const scores = [...input.text.matchAll(/^SCORE:[\t ]*(10|[0-9])[\t ]*\r?$/gm)];
const ambiguities = [...input.text.matchAll(/^AMBIGUITIES:[\t ]*(\S[^\r\n]*)\r?$/gm)];
@@ -350,7 +359,7 @@ export function validateOutsideReview(text: string, gate: OutsideGate): { comple
}
const VERDICT_EXIT: Record<OutsideVerdict, number> = { clean: 0, findings: 3, unverified: 4, unavailable: 1 };
const GATES = ['review', 'structured', 'spec', 'execution', 'proposal'];
const GATES = ['review', 'structured', 'spec', 'execution', 'proposal', 'autoplan'];
const USAGE = `Usage: outside-review-result.ts <gate> <response-file>
outside-review-result.ts --verdict [--stderr <file>] [--exit <code>] [--events <file>] [--provider-error <file>] [--label <name>] <gate> <response-file>
Gates: ${GATES.join('|')}. Two-argument form exits 0 completed, 1 unavailable, 2 usage.
+1 -1
View File
@@ -150,7 +150,7 @@ describe('gstack-claude-code', () => {
describe('API sends: gstack-outside-voice and the unattended runner', () => {
let server: ReturnType<typeof Bun.serve>;
let requests = 0;
const REVIEW = '[P2] A finding.\n\nRecommendation: revise because the premise is unmeasured.\n';
const REVIEW = `INPUT: ceo ${'a'.repeat(64)}\n\n[P2] A finding.\n\nRecommendation: revise because the premise is unmeasured.\n\n\`\`\`gstack-findings\n\`\`\`\n`;
beforeAll(() => {
server = Bun.serve({ port: 0, fetch() { requests++; return Response.json({ model: 'gpt-6-astra', choices: [{ message: { content: REVIEW } }], usage: {} }); } });
});
+117
View File
@@ -0,0 +1,117 @@
/**
* GST-3051: an autoplan outside review run the DRIVER.md way (no --gate) is
* judged by autoplan's own result contract, the INPUT receipt and the
* gstack-findings fence (lib/autoplan-prompts.ts RESULT_FORMAT), never by the
* /review gate's Recommendation line, so a review that ran is not labeled
* unavailable (missing_markers). A reply without that contract, or an empty
* one, still is. And --help never prints a line that starts like a real
* OUTSIDE_STATUS or VERDICT result.
*/
import { afterAll, describe, expect, test } from 'bun:test';
import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { writePrompts } from '../lib/autoplan-prompts';
const ROOT = path.resolve(import.meta.dir, '..');
const BIN = path.join(ROOT, 'bin', 'gstack-outside-voice');
const TMP = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-autoplan-'));
afterAll(() => fs.rmSync(TMP, { recursive: true, force: true }));
const SHA = 'a'.repeat(64);
const snapshotPath = path.join(TMP, 'snapshot.md');
fs.writeFileSync(snapshotPath, '# Plan\n\n## Implementation plan\n\n- Add a flag.\n');
const prompts = writePrompts({ phase: 'ceo', snapshot: { nativePrompt: 'native\n', snapshotPath, sha256: SHA }, outDir: TMP, priorConsensus: [] });
const env: Record<string, string> = {};
for (const [k, v] of Object.entries(process.env)) if (v !== undefined && !/^(OPENAI|ANTHROPIC)_/.test(k) && k !== 'GSTACK_OUTSIDE_RUNNER') env[k] = v;
env.GSTACK_STATE_ROOT = path.join(TMP, 'state');
function run(args: string[]) {
return spawnSync('bun', [BIN, ...args], { cwd: ROOT, env, encoding: 'utf8', timeout: 60_000 });
}
/** The DRIVER.md:29 form: host-subagent binding the reviewer's file, no --gate. */
function review(name: string, text: string, extra: string[] = [], prompt = prompts.outside) {
const result = path.join(TMP, name + '.result.md');
fs.writeFileSync(result, text);
const out = path.join(TMP, name + '.out.md');
const r = run(['run', '--runner', 'host-subagent', '--prompt', prompt, '--out', out, '--result', result, '--model', 'openai/gpt-6-astra', '--native-model', 'claude-opus-5-5', ...extra]);
const meta = JSON.parse(fs.readFileSync(out + '.meta.json', 'utf8'));
return { ...r, meta };
}
const fence = (rows: string[]) => '\x60\x60\x60gstack-findings\n' + rows.join('\n') + (rows.length ? '\n' : '') + '\x60\x60\x60\n';
const HIGH = '{"id":"F1","severity":"High","title":"The premise is unmeasured","fix":"Measure it first"}';
const MEDIUM = '{"id":"F2","severity":"Medium","title":"Name the flag plainly"}';
const receipt = 'INPUT: ceo ' + SHA + '\n\n';
describe('autoplan outside review without --gate', () => {
test('a review that follows the RESULT FORMAT is completed with its findings, not missing_markers', () => {
const r = review('high', receipt + 'The plan assumes demand it never measured.\n\n' + fence([HIGH, MEDIUM]));
expect(r.status, r.stderr).toBe(0);
expect(r.stdout).toContain('VERDICT: findings\nFINDINGS: P1\n');
expect(r.stdout).toMatch(/^OUTSIDE_STATUS: completed provider=host-subagent model=openai\/gpt-6-astra family=openai/m);
expect(r.meta.status).toBe('completed');
expect(r.meta.verdict).toMatchObject({ execution: { state: 'ran' }, verdict: 'findings', findings: { highest: 'P1' } });
expect(r.meta.reason).toBeUndefined();
});
test('an empty fence is a clean review; advisory-only findings are clean with their severity', () => {
const none = review('none', receipt + 'Nothing to add.\n\n' + fence([]));
expect(none.status, none.stderr).toBe(0);
expect(none.stdout).toContain('VERDICT: clean\nFINDINGS: none\n');
const medium = review('medium', receipt + 'One naming nit.\n\n' + fence([MEDIUM]));
expect(medium.status, medium.stderr).toBe(0);
expect(medium.stdout).toContain('VERDICT: clean\nFINDINGS: P2\n');
});
test('a reply without the fence or the INPUT receipt is still unavailable (missing_markers)', () => {
for (const [name, text] of [['no-fence', receipt + 'High: the premise is unmeasured.\n'], ['no-receipt', 'Review follows.\n\n' + fence([HIGH])]] as const) {
const r = review(name, text);
expect(r.status).toBe(1);
expect(r.stdout).toContain('VERDICT: unavailable\n');
expect(r.meta).toMatchObject({ status: 'unavailable', reason: 'missing_markers' });
}
});
test('an empty reply is still unavailable', () => {
const r = review('empty', '');
expect(r.status).toBe(1);
expect(r.meta).toMatchObject({ status: 'unavailable', reason: 'empty_response' });
});
test('a malformed findings row is unavailable and names the row', () => {
const r = review('bad-row', receipt + fence(['{"severity":"Huge","title":"x"}']));
expect(r.status).toBe(1);
expect(r.meta).toMatchObject({ status: 'unavailable', reason: 'missing_markers' });
expect(r.meta.detail).toContain('findings row 1');
});
});
describe('gate selection', () => {
test('--gate autoplan applies the same contract to any prompt', () => {
const plain = path.join(TMP, 'plain-prompt.md');
fs.writeFileSync(plain, 'Review this plan.\n');
const r = review('explicit', receipt + fence([HIGH]), ['--gate', 'autoplan'], plain);
expect(r.status, r.stderr).toBe(0);
expect(r.stdout).toContain('VERDICT: findings\nFINDINGS: P1\n');
});
test("a prompt without autoplan's RESULT FORMAT keeps the review gate", () => {
const plain = path.join(TMP, 'review-prompt.md');
fs.writeFileSync(plain, 'Review this diff.\n');
const r = review('review-default', receipt + fence([HIGH]), [], plain);
expect(r.status).toBe(1);
expect(r.meta).toMatchObject({ status: 'unavailable', reason: 'missing_markers', detail: 'missing review completion recommendation' });
});
});
describe('--help', () => {
test('no help line starts like a real result line', () => {
for (const r of [run(['--help']), run([]), run(['run'])]) {
const text = r.stdout + r.stderr;
expect(text).toContain('consent_missing');
expect(text.split('\n').filter(line => /^(?:OUTSIDE_STATUS|VERDICT):/.test(line))).toEqual([]);
}
});
});