mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 06:24:41 +01:00
INV-1: gate outcome contract for outside reviews
classifyOutsideReview() returns separate execution, findings and verdict fields from text, stderr and exit status; reason codes, anchors and fix actions live in lib/gate-outcomes.ts. validateOutsideReview() stays as the text-only wrapper and the two-argument CLI keeps exit 0/1/2 for importers and installed skills that predate a re-render. The --verdict form (implied by --stderr/--exit) prints VERDICT/FINDINGS/REASON and exits 0/3/4/1; the generated outside-voice caller moves to it and branches on each status (unverified prints OUTSIDE_STATUS: unverified, never completed). Adds docs/troubleshooting.md with one anchored section per reason code, the anchor test, and the gate-tail lint (query scripts may not swallow failures, gate calls may not end in || true, other best-effort tails say why).
This commit is contained in:
1 parent
ff5fbd09ee
commit
9aba7d9bd7
11 files changed
+686
-41
No files matched your search
@@ -260,7 +260,7 @@ _gstack_codex_timeout_wrapper() {
|
||||
"$@" &
|
||||
local _cmd_pid=$!
|
||||
(
|
||||
sleep "$_duration" && trap '' TERM && kill -TERM "$_cmd_pid" 2>/dev/null || exit 0
|
||||
sleep "$_duration" && trap '' TERM && kill -TERM "$_cmd_pid" 2>/dev/null || exit 0 # best-effort: the command already exited, nothing to stop
|
||||
_GSTACK_CODEX_WAITED=0
|
||||
while [ "$_GSTACK_CODEX_WAITED" -lt "$_grace" ] && kill -0 "$_cmd_pid" 2>/dev/null; do
|
||||
sleep 1
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
# Troubleshooting gstack messages
|
||||
|
||||
Find the message you saw, then follow its fix. Every check gstack runs ends in
|
||||
one of three states:
|
||||
|
||||
- **ran**: the check executed. It may still have found problems.
|
||||
- **not run**: gstack chose not to run it (you turned it off, or nothing applied).
|
||||
- **unavailable**: gstack tried and could not run it. This is missing coverage,
|
||||
never a pass.
|
||||
|
||||
A message that is not `ran` has this shape:
|
||||
|
||||
```
|
||||
<check>: not run (<reason>). Fix: <command>
|
||||
<check> unavailable: <reason> (<detail>). No review ran; this is missing coverage, not a pass. Fix: <command>
|
||||
<check>: ran, verdict unverified (<reason>). Fix: read the output above
|
||||
```
|
||||
|
||||
Search this page for the words after `unavailable:` or `not run (`. Each
|
||||
section has a stable link anchor; the reason codes and anchors come from
|
||||
`lib/gate-outcomes.ts`, and a free test fails if a code has no section here.
|
||||
|
||||
---
|
||||
|
||||
## Outside reviews (Codex and Claude Code)
|
||||
|
||||
Outside reviews send your diff, plan or question to a second AI provider. Their
|
||||
verdict comes from `lib/outside-review-result.ts`, which prints
|
||||
`VERDICT: clean|findings|unverified|unavailable`. A `findings` verdict with a
|
||||
P0 or P1 finding blocks exactly like a native P0/P1. `unverified` and
|
||||
`unavailable` are missing coverage: /ship and /review continue, show the gap in
|
||||
the readiness dashboard and the PR body, and never count it as a pass.
|
||||
|
||||
<a id="codex-sandbox-unavailable"></a>
|
||||
### `Codex outside review unavailable: Codex's sandbox could not start here (...)`
|
||||
|
||||
**Meaning.** Codex runs every command inside a Linux sandbox (bubblewrap). In
|
||||
many containers and devcontainers the kernel does not allow unprivileged user
|
||||
namespaces, so the sandbox cannot start and every command Codex tries fails.
|
||||
gstack reports this instead of trusting a review that read nothing.
|
||||
`CODEX_MODE: sandbox_unavailable` is the same condition found by the free
|
||||
preflight before any paid call.
|
||||
|
||||
**What is kept.** Nothing was sent for review, or the review was discarded. Your
|
||||
code and files are unchanged.
|
||||
|
||||
**Fix.** Enable unprivileged user namespaces for the container (for Docker, a
|
||||
seccomp profile that allows them), or, inside a container you trust, run
|
||||
Codex without its sandbox for this shell:
|
||||
|
||||
```bash
|
||||
export GSTACK_CODEX_NO_SANDBOX=1
|
||||
```
|
||||
|
||||
**Expected result.** With namespaces enabled, the next review prints
|
||||
`OUTSIDE_STATUS: completed`. With `GSTACK_CODEX_NO_SANDBOX=1`, every review
|
||||
prints `WARNING: GSTACK_CODEX_NO_SANDBOX=1: ...` because Codex can then read
|
||||
and write anything your user can. Only the exact value `1` works, and only from
|
||||
your shell environment.
|
||||
|
||||
<a id="outside-review-commands-failed"></a>
|
||||
### `... outside review unavailable: the reviewer could not run commands or read the diff (...)`
|
||||
|
||||
**Meaning.** The reviewer tried to run commands but none succeeded, or its
|
||||
answer says it could not run commands or read the diff. A "no issues found"
|
||||
written after that is not a review.
|
||||
|
||||
**What is kept.** Nothing; the answer is shown above but not counted.
|
||||
|
||||
**Fix.** Read the reviewer's stderr above, repair what it names, and re-run.
|
||||
|
||||
**Expected result.** `OUTSIDE_STATUS: completed` with a `VERDICT:` line.
|
||||
|
||||
<a id="outside-review-execution-failed"></a>
|
||||
### `... outside review unavailable: the reviewer process failed (exit N: ...)`
|
||||
|
||||
**Meaning.** The provider CLI exited non-zero. The first stderr line is in the
|
||||
parentheses.
|
||||
|
||||
**What is kept.** Partial output is shown above; it is not counted.
|
||||
|
||||
**Fix.** Repair the cause in the provider diagnosis (log in again, fix the
|
||||
model, check the network), then re-run.
|
||||
|
||||
**Expected result.** `OUTSIDE_STATUS: completed`.
|
||||
|
||||
<a id="outside-review-timeout"></a>
|
||||
### `... outside review unavailable: the reviewer hit its time limit and was stopped (exit 124)`
|
||||
|
||||
**Meaning.** The provider did not finish within its deadline and was stopped.
|
||||
|
||||
**What is kept.** Partial output is shown above; it is not counted.
|
||||
|
||||
**Fix.** Re-run with a smaller scope, or check the provider's status and your
|
||||
network.
|
||||
|
||||
**Expected result.** `OUTSIDE_STATUS: completed`.
|
||||
|
||||
<a id="outside-review-empty-response"></a>
|
||||
### `... outside review unavailable: the reviewer returned no response`
|
||||
|
||||
**Meaning.** The provider exited successfully but returned nothing.
|
||||
|
||||
**Fix.** Read the stderr above, then re-run.
|
||||
|
||||
**Expected result.** `OUTSIDE_STATUS: completed`.
|
||||
|
||||
<a id="outside-review-refused"></a>
|
||||
### `... outside review unavailable: the reviewer declined to review`
|
||||
|
||||
**Meaning.** The provider answered with a refusal instead of a review.
|
||||
|
||||
**Fix.** Re-run; if it declines again, rely on the native review.
|
||||
|
||||
**Expected result.** `OUTSIDE_STATUS: completed`.
|
||||
|
||||
<a id="outside-review-missing-markers"></a>
|
||||
### `... outside review unavailable: the response lacks the markers this gate requires (...)`
|
||||
|
||||
**Meaning.** The answer did not contain what the gate checks: a
|
||||
`Recommendation: ... because ...` line, a `SCORE:`/`AMBIGUITIES:` pair, or
|
||||
severity tags.
|
||||
|
||||
**Fix.** Re-run the review.
|
||||
|
||||
**Expected result.** `OUTSIDE_STATUS: completed`.
|
||||
|
||||
<a id="outside-review-unverified"></a>
|
||||
### `... outside review: ran, verdict unverified (...)` / `OUTSIDE_STATUS: unverified` / `GATE: UNVERIFIED`
|
||||
|
||||
**Meaning.** The review completed but tagged nothing and gave no explicit
|
||||
no-findings conclusion, so no pass or fail can be read from it.
|
||||
|
||||
**What is kept.** The full answer is shown above.
|
||||
|
||||
**Fix.** Read the output above and decide. It is not a pass, and /ship and
|
||||
/review list it as missing coverage.
|
||||
|
||||
<a id="codex-model-unusable"></a>
|
||||
### `CODEX_MODE: model_unusable` / `MODEL_UNUSABLE`
|
||||
|
||||
**Meaning.** Codex rejected the selected model: the account cannot use it
|
||||
(HTTP 400), the model is retired, or a custom provider's `base_url` is wrong
|
||||
(HTTP 404). The `CODEX_MODEL:` line names the model and where it came from.
|
||||
|
||||
**Fix.** Choose a model your account can use, or correct the provider:
|
||||
|
||||
```bash
|
||||
export GSTACK_CODEX_MODEL=<supported-model>
|
||||
# or edit model / base_url in ${CODEX_HOME:-~/.codex}/config.toml
|
||||
```
|
||||
|
||||
**Expected result.** `CODEX_MODE: ready`.
|
||||
|
||||
<a id="codex-mode-unverified"></a>
|
||||
### `CODEX_MODE: unverified` / `MODEL_PROBE_INCONCLUSIVE`
|
||||
|
||||
**Meaning.** The short model check timed out or hit a network error, so gstack
|
||||
could not confirm the model works. The review still runs and its own result is
|
||||
checked.
|
||||
|
||||
**Fix.** Nothing now. If the review then fails, its message says why.
|
||||
|
||||
<a id="codex-auth-failed"></a>
|
||||
### `CODEX_MODE: not_authed` / `AUTH_FAILED`
|
||||
|
||||
**Meaning.** No Codex credentials were found: no `CODEX_API_KEY`, no
|
||||
`OPENAI_API_KEY`, no `auth.json`, and no set environment variable named by a
|
||||
custom provider's `env_key` in `config.toml`.
|
||||
|
||||
**Fix.**
|
||||
|
||||
```bash
|
||||
codex login
|
||||
```
|
||||
|
||||
or export the variable your provider's `env_key` names.
|
||||
|
||||
**Expected result.** `CODEX_MODE: ready`.
|
||||
|
||||
<a id="outside-review-disabled"></a>
|
||||
### `Codex review skipped (codex_reviews disabled)` / `CODEX_MODE: disabled`
|
||||
|
||||
**Meaning.** You turned outside reviews off. This is `not run`, never a pass and
|
||||
never an outage.
|
||||
|
||||
**Fix.** To turn them back on:
|
||||
|
||||
```bash
|
||||
gstack-config set codex_reviews enabled
|
||||
```
|
||||
@@ -0,0 +1,92 @@
|
||||
/**
|
||||
* INV-1 gate outcome contract: every gate or outside review ends as
|
||||
* `ran` (with findings or a verdict), `not_run` (gstack chose not to run it) or
|
||||
* `unavailable` (it tried and could not). Each reason code maps to one state,
|
||||
* one stable anchor in docs/troubleshooting.md and one fix action; the
|
||||
* validator prints these lines and test/troubleshooting-anchors.test.ts checks
|
||||
* every anchor exists. Add a row here before printing a new reason anywhere.
|
||||
*/
|
||||
export type GateState = 'ran' | 'not_run' | 'unavailable';
|
||||
|
||||
export interface GateOutcome {
|
||||
state: GateState;
|
||||
/** Stable `<a id>` in docs/troubleshooting.md. */
|
||||
anchor: string;
|
||||
/** What happened, in plain words; `<detail>` adds the real value (exit code, stderr line). */
|
||||
summary: string;
|
||||
fix: string;
|
||||
}
|
||||
|
||||
export const GATE_OUTCOMES = {
|
||||
sandbox_unavailable: {
|
||||
state: 'unavailable', anchor: 'codex-sandbox-unavailable',
|
||||
summary: "Codex's sandbox could not start here",
|
||||
fix: 'enable unprivileged user namespaces for this container, or set GSTACK_CODEX_NO_SANDBOX=1',
|
||||
},
|
||||
commands_failed: {
|
||||
state: 'unavailable', anchor: 'outside-review-commands-failed',
|
||||
summary: 'the reviewer could not run commands or read the diff',
|
||||
fix: 'read the reviewer stderr above, repair the environment it names, then re-run the review',
|
||||
},
|
||||
execution_failed: {
|
||||
state: 'unavailable', anchor: 'outside-review-execution-failed',
|
||||
summary: 'the reviewer process failed',
|
||||
fix: 'read the provider diagnosis above (auth, model, network), repair it, then re-run the review',
|
||||
},
|
||||
timeout: {
|
||||
state: 'unavailable', anchor: 'outside-review-timeout',
|
||||
summary: 'the reviewer hit its time limit and was stopped',
|
||||
fix: 're-run the review with a smaller scope, or check the provider status and network',
|
||||
},
|
||||
empty_response: {
|
||||
state: 'unavailable', anchor: 'outside-review-empty-response',
|
||||
summary: 'the reviewer returned no response',
|
||||
fix: 'read the reviewer stderr above, then re-run the review',
|
||||
},
|
||||
review_refused: {
|
||||
state: 'unavailable', anchor: 'outside-review-refused',
|
||||
summary: 'the reviewer declined to review',
|
||||
fix: 're-run the review; if it declines again, review the change natively',
|
||||
},
|
||||
missing_markers: {
|
||||
state: 'unavailable', anchor: 'outside-review-missing-markers',
|
||||
summary: 'the response lacks the markers this gate requires',
|
||||
fix: 're-run the review; a response without its required markers never counts as a pass',
|
||||
},
|
||||
untagged_review: {
|
||||
state: 'ran', anchor: 'outside-review-unverified',
|
||||
summary: 'the review completed without severity tags or an explicit no-findings line',
|
||||
fix: 'read the output above and decide; it is not a pass',
|
||||
},
|
||||
model_unusable: {
|
||||
state: 'unavailable', anchor: 'codex-model-unusable',
|
||||
summary: 'Codex could not use the selected model',
|
||||
fix: 'set GSTACK_CODEX_MODEL=<supported-model> or fix model (and base_url for a custom provider) in your Codex config.toml',
|
||||
},
|
||||
probe_inconclusive: {
|
||||
state: 'ran', anchor: 'codex-mode-unverified',
|
||||
summary: 'the Codex model check timed out or hit a network error, so readiness is unverified',
|
||||
fix: 'nothing to do now; the review itself is still checked, and a failure there is reported',
|
||||
},
|
||||
auth_failed: {
|
||||
state: 'unavailable', anchor: 'codex-auth-failed',
|
||||
summary: 'no Codex credentials were found',
|
||||
fix: 'run codex login, or set the API key your Codex config.toml provider names in env_key',
|
||||
},
|
||||
disabled: {
|
||||
state: 'not_run', anchor: 'outside-review-disabled',
|
||||
summary: 'outside reviews are turned off (codex_reviews=disabled)',
|
||||
fix: 'gstack-config set codex_reviews enabled',
|
||||
},
|
||||
} as const satisfies Record<string, GateOutcome>;
|
||||
|
||||
export type GateReason = keyof typeof GATE_OUTCOMES;
|
||||
|
||||
/** One user-facing line for an outcome that is not a clean `ran`. */
|
||||
export function gateOutcomeLine(gate: string, reason: GateReason, detail?: string): string {
|
||||
const outcome: GateOutcome = GATE_OUTCOMES[reason];
|
||||
const what = detail ? `${outcome.summary} (${detail})` : outcome.summary;
|
||||
if (outcome.state === 'not_run') return `${gate}: not run (${what}). Fix: ${outcome.fix}.`;
|
||||
if (outcome.state === 'ran') return `${gate}: ran, verdict unverified (${what}). Fix: ${outcome.fix}.`;
|
||||
return `${gate} unavailable: ${what}. No review ran; this is missing coverage, not a pass. Fix: ${outcome.fix}.`;
|
||||
}
|
||||
+139
-31
@@ -1,38 +1,146 @@
|
||||
/** Review-specific completion evidence, separate from provider transport success. */
|
||||
export type OutsideGate = 'review' | 'structured' | 'spec';
|
||||
export function validateOutsideReview(text: string, gate: OutsideGate): { completed: boolean; reason?: string; score?: number; gate?: 'pass' | 'fail' } {
|
||||
if (!text.trim()) return { completed: false, reason: 'empty response' };
|
||||
// "I cannot find any issues" is a legitimate clean conclusion. Match a
|
||||
// refused/unavailable review, not every use of a negative auxiliary verb.
|
||||
if (/\b(?:(?:I (?:cannot|can't|won't|will not|am unable to)|I'm unable to)\s+(?:review|analy[sz]e|evaluate|assess|inspect|access|complete|perform|provide|assist|help|proceed)|unable to (?:review|analy[sz]e)|I must (?:decline|refuse))\b/i.test(text)) return { completed: false, reason: 'review refused' };
|
||||
if (gate === 'spec') {
|
||||
const scores = [...text.matchAll(/^SCORE:[\t ]*(10|[0-9])[\t ]*\r?$/gm)];
|
||||
const ambiguities = [...text.matchAll(/^AMBIGUITIES:[\t ]*(\S[^\r\n]*)\r?$/gm)];
|
||||
if (scores.length !== 1 || ambiguities.length !== 1) return { completed: false, reason: 'missing or invalid SCORE/AMBIGUITIES markers' };
|
||||
const score = Number(scores[0][1]);
|
||||
return { completed: true, score, gate: score >= 7 ? 'pass' : 'fail' };
|
||||
}
|
||||
if (gate === 'structured') {
|
||||
const plain = plainReview(text);
|
||||
const severity = /\[P[0-3]\]|^P[0-3]:/m.test(plain);
|
||||
const clear = /\bNO_FINDINGS\b|\bno (?:actionable |significant |new |concrete )?(?:bugs|issues|findings|problems)\b|\b(?:did not|didn't) (?:find|identify) any (?:actionable |new |concrete )?(?:bugs|issues|findings|problems)\b/i.test(text);
|
||||
if (!severity && !clear) return { completed: false, reason: 'missing severity or explicit no-findings conclusion' };
|
||||
return { completed: true, gate: /\[P1\]|^P1:/m.test(plain) ? 'fail' : 'pass' };
|
||||
}
|
||||
// Formatting the requested marker in bold, inline code, or a list does not
|
||||
// invalidate a completed review. Preserve the explicit action + reason gate.
|
||||
const plain = plainReview(text);
|
||||
if (!/^Recommendation:[\t ]*[^\r\n]+\bbecause\b[\t ]*\S[^\r\n]+$/im.test(plain)) return { completed: false, reason: 'missing review completion recommendation' };
|
||||
return { completed: true };
|
||||
/**
|
||||
* Review-specific completion evidence, separate from provider transport success.
|
||||
*
|
||||
* classifyOutsideReview() is the INV-1 contract: it takes the review text, the
|
||||
* provider's stderr and exit status and returns three separate answers: did
|
||||
* the review execute, what is the highest finding severity, and the verdict. validateOutsideReview() is the
|
||||
* older text-only shape, kept for direct importers and stale rendered skills.
|
||||
*/
|
||||
import { GATE_OUTCOMES, gateOutcomeLine, type GateReason } from './gate-outcomes';
|
||||
|
||||
export type OutsideGate = 'review' | 'structured' | 'spec' | 'execution';
|
||||
export type OutsideVerdict = 'clean' | 'findings' | 'unverified' | 'unavailable';
|
||||
export type Severity = 'P0' | 'P1' | 'P2' | 'P3';
|
||||
|
||||
export interface OutsideReviewInput {
|
||||
text: string;
|
||||
gate: OutsideGate;
|
||||
stderr?: string;
|
||||
exit?: number;
|
||||
}
|
||||
|
||||
export interface OutsideReviewClassification {
|
||||
execution: { state: 'ran' } | { state: 'unavailable'; reason: GateReason; detail?: string };
|
||||
findings: { highest: Severity | null };
|
||||
verdict: OutsideVerdict;
|
||||
/** Set for every verdict except clean/findings. */
|
||||
reason?: GateReason;
|
||||
detail?: string;
|
||||
score?: number;
|
||||
}
|
||||
|
||||
const REFUSAL = /\b(?:(?:I (?:cannot|can't|won't|will not|am unable to)|I'm unable to)\s+(?:review|analy[sz]e|evaluate|assess|inspect|access|complete|perform|provide|assist|help|proceed)|unable to (?:review|analy[sz]e)|I must (?:decline|refuse))\b/i;
|
||||
|
||||
function stderrHead(stderr: string): string | undefined {
|
||||
return stderr.split(/\r?\n/).map(line => line.trim()).find(Boolean)?.slice(0, 240);
|
||||
}
|
||||
|
||||
function execution(input: OutsideReviewInput): OutsideReviewClassification['execution'] {
|
||||
const exit = input.exit ?? 0;
|
||||
if (exit !== 0) {
|
||||
const head = stderrHead(input.stderr ?? '');
|
||||
const detail = head ? `exit ${exit}: ${head}` : `exit ${exit}`;
|
||||
return { state: 'unavailable', reason: exit === 124 ? 'timeout' : 'execution_failed', detail };
|
||||
}
|
||||
if (!input.text.trim()) return { state: 'unavailable', reason: 'empty_response' };
|
||||
if (REFUSAL.test(input.text)) return { state: 'unavailable', reason: 'review_refused' };
|
||||
return { state: 'ran' };
|
||||
}
|
||||
|
||||
/** Formatting the requested marker in bold, inline code, or a list does not invalidate a completed review. */
|
||||
function plainReview(text: string): string {
|
||||
return text.split(/\r?\n/).map(line => line.replace(/^[\t ]*(?:#{1,6}[\t ]+|[-+*][\t ]+)?/, '').replace(/[*_`]/g, '')).join('\n');
|
||||
}
|
||||
|
||||
export function classifyOutsideReview(input: OutsideReviewInput): OutsideReviewClassification {
|
||||
const plain = plainReview(input.text);
|
||||
const levels = [...plain.matchAll(/\[(P[0-3])\]|^(P[0-3]):/gm)].map(m => (m[1] ?? m[2]) as Severity);
|
||||
const findings = { highest: levels.length ? levels.sort()[0]! : null };
|
||||
const ran = execution(input);
|
||||
if (ran.state === 'unavailable') return { execution: ran, findings, verdict: 'unavailable', reason: ran.reason, detail: ran.detail };
|
||||
const blocking = findings.highest === 'P0' || findings.highest === 'P1';
|
||||
const result = (verdict: OutsideVerdict, reason?: GateReason, detail?: string): OutsideReviewClassification =>
|
||||
({ execution: ran, findings, verdict, ...(reason ? { reason } : {}), ...(detail ? { detail } : {}) });
|
||||
if (input.gate === 'spec') {
|
||||
const scores = [...input.text.matchAll(/^SCORE:[\t ]*(10|[0-9])[\t ]*\r?$/gm)];
|
||||
const ambiguities = [...input.text.matchAll(/^AMBIGUITIES:[\t ]*(\S[^\r\n]*)\r?$/gm)];
|
||||
if (scores.length !== 1 || ambiguities.length !== 1) return result('unavailable', 'missing_markers', 'missing or invalid SCORE/AMBIGUITIES markers');
|
||||
const score = Number(scores[0]![1]);
|
||||
return { ...result(score >= 7 ? 'clean' : 'findings'), score };
|
||||
}
|
||||
if (input.gate === 'structured') {
|
||||
const clear = /\bNO_FINDINGS\b|\bno (?:actionable |significant |new |concrete )?(?:bugs|issues|findings|problems)\b|\b(?:did not|didn't) (?:find|identify) any (?:actionable |new |concrete )?(?:bugs|issues|findings|problems)\b/i.test(input.text);
|
||||
if (!findings.highest && !clear) return result('unverified', 'untagged_review', 'missing severity or explicit no-findings conclusion');
|
||||
return result(blocking ? 'findings' : 'clean');
|
||||
}
|
||||
if (input.gate === 'review' && !/^Recommendation:[\t ]*[^\r\n]+\bbecause\b[\t ]*\S[^\r\n]+$/im.test(plain)) {
|
||||
return result('unavailable', 'missing_markers', 'missing review completion recommendation');
|
||||
}
|
||||
return result(blocking ? 'findings' : 'clean');
|
||||
}
|
||||
|
||||
const LEGACY_REASONS: Partial<Record<GateReason, string>> = {
|
||||
empty_response: 'empty response',
|
||||
review_refused: 'review refused',
|
||||
};
|
||||
|
||||
/** Text-only compatibility shape for direct importers (claude-code/SKILL.md.tmpl) and the two-argument CLI. */
|
||||
export function validateOutsideReview(text: string, gate: OutsideGate): { completed: boolean; reason?: string; score?: number; gate?: 'pass' | 'fail' } {
|
||||
const checked = classifyOutsideReview({ text, gate });
|
||||
if (checked.verdict === 'unavailable' || checked.verdict === 'unverified') {
|
||||
const reason = LEGACY_REASONS[checked.reason!] ?? checked.detail ?? GATE_OUTCOMES[checked.reason!].summary;
|
||||
return { completed: false, reason };
|
||||
}
|
||||
if (gate === 'spec') return { completed: true, score: checked.score, gate: checked.verdict === 'clean' ? 'pass' : 'fail' };
|
||||
if (gate === 'structured') return { completed: true, gate: /\[P1\]|^P1:/m.test(plainReview(text)) ? 'fail' : 'pass' };
|
||||
return { completed: true };
|
||||
}
|
||||
|
||||
const VERDICT_EXIT: Record<OutsideVerdict, number> = { clean: 0, findings: 3, unverified: 4, unavailable: 1 };
|
||||
const GATES = ['review', 'structured', 'spec', 'execution'];
|
||||
const USAGE = `Usage: outside-review-result.ts <gate> <response-file>
|
||||
outside-review-result.ts --verdict [--stderr <file>] [--exit <code>] [--label <name>] <gate> <response-file>
|
||||
Gates: ${GATES.join('|')}. Two-argument form exits 0 completed, 1 unavailable, 2 usage.
|
||||
Verdict form prints VERDICT: clean|findings|unverified|unavailable and exits 0, 3, 4 or 1.`;
|
||||
|
||||
async function readOptional(file: string | undefined): Promise<string> {
|
||||
if (!file) return '';
|
||||
try { return await Bun.file(file).text(); } catch { return ''; }
|
||||
}
|
||||
|
||||
if (import.meta.main) {
|
||||
const [gate, path] = process.argv.slice(2);
|
||||
if (!['review', 'structured', 'spec'].includes(gate) || !path) { console.error('Usage: outside-review-result.ts review|structured|spec <response-file>'); process.exit(2); }
|
||||
try {
|
||||
const result = validateOutsideReview(await Bun.file(path).text(), gate as OutsideGate);
|
||||
const args = process.argv.slice(2);
|
||||
const flags: Record<string, string> = {};
|
||||
const positional: string[] = [];
|
||||
let verdictMode = false;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i]!;
|
||||
if (arg === '--verdict') verdictMode = true;
|
||||
else if (['--stderr', '--exit', '--label'].includes(arg) && i + 1 < args.length) { flags[arg] = args[++i]!; verdictMode ||= arg !== '--label'; }
|
||||
else if (arg.startsWith('--')) { console.error(USAGE); process.exit(2); }
|
||||
else positional.push(arg);
|
||||
}
|
||||
const [gate, path] = positional;
|
||||
const exit = flags['--exit'] === undefined ? 0 : Number(flags['--exit']);
|
||||
if (!gate || !GATES.includes(gate) || !path || positional.length !== 2 || !Number.isInteger(exit)
|
||||
|| (!verdictMode && gate === 'execution')) { console.error(USAGE); process.exit(2); }
|
||||
let text: string;
|
||||
try { text = await Bun.file(path).text(); } catch (error) {
|
||||
if (!verdictMode) { console.error(`Outside review unavailable: ${error}`); process.exit(1); }
|
||||
text = '';
|
||||
}
|
||||
if (!verdictMode) {
|
||||
const result = validateOutsideReview(text, gate as OutsideGate);
|
||||
if (!result.completed) { console.error(`Outside review unavailable: ${result.reason}; missing coverage.`); process.exit(1); }
|
||||
} catch (error) { console.error(`Outside review unavailable: ${error}`); process.exit(1); }
|
||||
process.exit(0);
|
||||
}
|
||||
const checked = classifyOutsideReview({ text, gate: gate as OutsideGate, exit,
|
||||
stderr: await readOptional(flags['--stderr']) });
|
||||
console.log(`VERDICT: ${checked.verdict}`);
|
||||
console.log(`FINDINGS: ${checked.findings.highest ?? 'none'}`);
|
||||
if (checked.reason) {
|
||||
console.log(`REASON: ${checked.reason}`);
|
||||
console.error(gateOutcomeLine(flags['--label'] ?? 'Outside review', checked.reason, checked.detail));
|
||||
}
|
||||
process.exit(VERDICT_EXIT[checked.verdict]);
|
||||
}
|
||||
@@ -158,12 +158,13 @@ ${v.id === 'codex' && ctx.skillName === 'autoplan' ? `if [ "$_OUTSIDE_EXIT" -eq
|
||||
_gstack_codex_log_hang "autoplan" "0" || true
|
||||
fi` : ''}
|
||||
cat "$_OUTSIDE_TMP/stderr" >&2 || { [ "$_OUTSIDE_EXIT" -ne 0 ] || _OUTSIDE_EXIT=1; }
|
||||
if [ "$_OUTSIDE_EXIT" -ne 0 ]; then
|
||||
echo '${v.label} outside review unavailable: execution failed; missing coverage. Check the provider diagnosis above.' >&2
|
||||
exit "$_OUTSIDE_EXIT"
|
||||
fi
|
||||
bun "${root}/lib/outside-review-result.ts" ${opts.gate ?? 'review'} "$_OUTSIDE_TMP/text" || exit 1
|
||||
${v.id === 'claude-code' ? 'cat "$_OUTSIDE_TMP/text" || exit 1' : ''}
|
||||
_OUTSIDE_RC=0
|
||||
bun "${root}/lib/outside-review-result.ts" --label '${v.label} outside review' --exit "$_OUTSIDE_EXIT" --stderr "$_OUTSIDE_TMP/stderr" ${opts.gate ?? 'review'} "$_OUTSIDE_TMP/text" || _OUTSIDE_RC=$?
|
||||
${v.id === 'claude-code' ? '[ "$_OUTSIDE_RC" -eq 1 ] || cat "$_OUTSIDE_TMP/text" || exit 1\n' : ''}case "$_OUTSIDE_RC" in
|
||||
0|3) ;;
|
||||
4) echo 'OUTSIDE_STATUS: unverified provider=${v.id} host=${ctx.host}'; exit 4 ;;
|
||||
*) [ "$_OUTSIDE_EXIT" -ne 0 ] && exit "$_OUTSIDE_EXIT"; exit 1 ;;
|
||||
esac
|
||||
echo 'OUTSIDE_STATUS: completed provider=${v.id} host=${ctx.host}'`;
|
||||
}
|
||||
|
||||
@@ -197,7 +198,7 @@ export function outsideVoiceInvocation(ctx: TemplateContext, requested: OutsideC
|
||||
${outsideVoiceCommand(ctx, opts)}
|
||||
\`\`\`
|
||||
|
||||
Use Bash \`timeout: ${gateMs}\`; show the full response in a \`tool-output\` fence. Require successful execution and valid markers. Refusal, empty/malformed output, missing ${planRecommendation ? 'Recommendation: <action> because <reason>' : opts.purpose === 'design-direction' ? 'Recommendation' : 'score/severity/completion'} markers, timeout or CLI failure means \`outside_status: unavailable\`. ${opts.purpose === 'design-direction' ? 'Continue completed proposals; native completion does not count as outside coverage.' : opts.nativeAlreadyRequired ? 'Retain the required native pass without duplicating it; it cannot complete outside coverage.' : "Use the caller's fallback; missing coverage is never clean/PASS."} ${nativeStructured ? 'Scratch cleanup is automatic.' : 'After either outcome, delete only your private prompt; scratch cleanup is automatic.'}`;
|
||||
Use Bash \`timeout: ${gateMs}\`; show the full response in a \`tool-output\` fence. Require successful execution and valid markers. Refusal, empty/malformed output, missing ${planRecommendation ? 'Recommendation: <action> because <reason>' : opts.purpose === 'design-direction' ? 'Recommendation' : 'score/severity/completion'} markers, timeout or CLI failure means \`outside_status: unavailable\`. P0/P1 findings block like native ones; \`OUTSIDE_STATUS: unverified\` is missing coverage. ${opts.purpose === 'design-direction' ? 'Continue completed proposals; native completion does not count as outside coverage.' : opts.nativeAlreadyRequired ? 'Retain the required native pass without duplicating it; it cannot complete outside coverage.' : "Use the caller's fallback; missing coverage is never clean/PASS."} ${nativeStructured ? 'Scratch cleanup is automatic.' : 'After either outcome, delete only your private prompt; scratch cleanup is automatic.'}`;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -169,6 +169,8 @@ test.each([
|
||||
'Outside review unavailable: empty response; missing coverage.',
|
||||
'Outside review unavailable: review refused; missing coverage.',
|
||||
'Outside review unavailable: missing review completion recommendation; missing coverage.',
|
||||
'Codex outside review unavailable: the reviewer process failed (exit 1: error: 401 Unauthorized). No review ran; this is missing coverage, not a pass. Fix: read the provider diagnosis above (auth, model, network), repair it, then re-run the review.',
|
||||
'Codex outside review unavailable: the response lacks the markers this gate requires (missing review completion recommendation). No review ran; this is missing coverage, not a pass. Fix: re-run the review; a response without its required markers never counts as a pass.',
|
||||
])('actual owned post-execution error proves attempted outside voice, not completion: %s',diagnostic=>{
|
||||
const f=fixture();f.events[7]=ack('outside','Exit code 1\n'+diagnostic,true);
|
||||
expect(f.read()).toMatchObject({claudeVoiceFired:true,codexAttempted:true,codexVoiceFired:false,codexUnavailable:true,failedOutsideToolUseId:'outside'});
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* INV-1 lint: a check that could not run must never look like "nothing found".
|
||||
* 1. The learnings/timeline query scripts may not end in `2>/dev/null || exit 0`
|
||||
* (a missing bun read as "no learnings recorded").
|
||||
* 2. No template or resolver may wrap a gate binary in `|| true`, which turns
|
||||
* "could not run" into success.
|
||||
* 3. Any other `2>/dev/null || exit 0` tail in bin/ is a deliberate best-effort
|
||||
* exit and says why on the same line (`# best-effort: <reason>`).
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const QUERY_SCRIPTS = ['bin/gstack-learnings-search', 'bin/gstack-timeline-read'];
|
||||
const GATE_CALLS = ['outside-review-result.ts', 'gstack-learnings-search', 'gstack-timeline-read',
|
||||
'_gstack_codex_auth_probe', '_gstack_codex_model_probe', '_gstack_codex_sandbox_preflight'];
|
||||
const SILENT_EXIT = /2>\/dev\/null \|\| exit 0\b/;
|
||||
|
||||
function walk(dir: string, keep: (file: string) => boolean, out: string[] = []): string[] {
|
||||
for (const entry of fs.readdirSync(path.join(ROOT, dir), { withFileTypes: true })) {
|
||||
if (entry.name === 'node_modules' || entry.name.startsWith('.')) continue;
|
||||
const rel = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) walk(rel, keep, out);
|
||||
else if (keep(rel)) out.push(rel);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function sources(): string[] {
|
||||
const templates = fs.readdirSync(ROOT, { withFileTypes: true })
|
||||
.filter(d => d.isDirectory() && !d.name.startsWith('.') && !['node_modules', 'test'].includes(d.name))
|
||||
.flatMap(d => walk(d.name, f => f.endsWith('.tmpl')));
|
||||
return [...templates, ...walk('scripts/resolvers', f => f.endsWith('.ts')), ...walk('bin', () => true)];
|
||||
}
|
||||
|
||||
describe('gate tails never hide a check that did not run', () => {
|
||||
test('query scripts do not swallow a failed run as an empty result', () => {
|
||||
const offenders = QUERY_SCRIPTS.filter(f => SILENT_EXIT.test(fs.readFileSync(path.join(ROOT, f), 'utf8')));
|
||||
expect(offenders).toEqual([]);
|
||||
});
|
||||
|
||||
test('no gate binary call is wrapped in || true', () => {
|
||||
const offenders: string[] = [];
|
||||
for (const rel of sources()) {
|
||||
fs.readFileSync(path.join(ROOT, rel), 'utf8').split('\n').forEach((line, i) => {
|
||||
if (/^\s*(?:#|\/\/|\*)/.test(line)) return;
|
||||
if (GATE_CALLS.some(g => line.includes(g)) && /\|\|\s*true\b/.test(line)) offenders.push(`${rel}:${i + 1}: ${line.trim()}`);
|
||||
});
|
||||
}
|
||||
expect(offenders).toEqual([]);
|
||||
});
|
||||
|
||||
test('other best-effort exits in bin/ carry a same-line reason', () => {
|
||||
const offenders: string[] = [];
|
||||
for (const rel of walk('bin', () => true)) {
|
||||
if (QUERY_SCRIPTS.includes(rel)) continue;
|
||||
fs.readFileSync(path.join(ROOT, rel), 'utf8').split('\n').forEach((line, i) => {
|
||||
if (SILENT_EXIT.test(line) && !/#\s*best-effort:\s*\S/.test(line)) offenders.push(`${rel}:${i + 1}: ${line.trim()}`);
|
||||
});
|
||||
}
|
||||
expect(offenders).toEqual([]);
|
||||
});
|
||||
|
||||
test('negative controls: each rule flags a planted offender', () => {
|
||||
expect(SILENT_EXIT.test('" 2>/dev/null || exit 0')).toBe(true);
|
||||
expect(/#\s*best-effort:\s*\S/.test('mkdir -p "$X" 2>/dev/null || exit 0 # best-effort: spool is optional')).toBe(true);
|
||||
const line = '$GSTACK_BIN/gstack-learnings-search --limit 10 2>/dev/null || true';
|
||||
expect(GATE_CALLS.some(g => line.includes(g)) && /\|\|\s*true\b/.test(line)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -247,9 +247,10 @@ export function autoplanDualVoiceEvidence(transcript: unknown[], options: Autopl
|
||||
if (execution.succeeded && /^OUTSIDE_STATUS: completed provider=codex host=claude$/m.test(execution.output)) {
|
||||
result.codexAttempted = result.codexVoiceFired = true; result.outsideToolUseId = outside.id;
|
||||
} else if (!execution.background && !execution.succeeded && outside.result.error &&
|
||||
/^(?:Codex outside review unavailable: execution failed; missing coverage\. Check the provider diagnosis above\.|Outside review unavailable: (?:empty response|review refused|missing review completion recommendation); missing coverage\.)$/m.test(execution.output)) {
|
||||
/^(?:Codex outside review unavailable: execution failed; missing coverage\. Check the provider diagnosis above\.|Outside review unavailable: (?:empty response|review refused|missing review completion recommendation); missing coverage\.|Codex outside review unavailable: [^\n]+\. No review ran; this is missing coverage, not a pass\. Fix: [^\n]+)$/m.test(execution.output)) {
|
||||
// These finite diagnostics occur only after the exact generated
|
||||
// command reaches Codex or its review validator. They prove an
|
||||
// command reaches Codex or its review validator (lib/gate-outcomes.ts
|
||||
// lines from the verdict form; the first two are pre-verdict renders). They prove an
|
||||
// attempted voice, never a completed review or comparison.
|
||||
result.codexAttempted = result.codexUnavailable = true;
|
||||
result.failedOutsideToolUseId = outside.id;
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* INV-1 outside-review contract: classifyOutsideReview() separates execution,
|
||||
* findings and verdict; validateOutsideReview() and the two-argument CLI keep
|
||||
* their old shape for importers and installed skills that predate a re-render.
|
||||
*/
|
||||
import { afterAll, describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { classifyOutsideReview, validateOutsideReview, type OutsideGate } from '../lib/outside-review-result';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const LIB = path.join(ROOT, 'lib', 'outside-review-result.ts');
|
||||
const TMP = fs.mkdtempSync(path.join(os.tmpdir(), 'outside-review-result-'));
|
||||
afterAll(() => fs.rmSync(TMP, { recursive: true, force: true }));
|
||||
|
||||
let n = 0;
|
||||
function file(content: string): string {
|
||||
const p = path.join(TMP, `f${n++}.txt`);
|
||||
fs.writeFileSync(p, content);
|
||||
return p;
|
||||
}
|
||||
function cli(args: string[]) {
|
||||
const r = spawnSync(process.execPath, [LIB, ...args], { encoding: 'utf8', timeout: 10000 });
|
||||
return { status: r.status, stdout: r.stdout, stderr: r.stderr };
|
||||
}
|
||||
|
||||
const RECOMMEND = 'Recommendation: fix the guard because changed.ts loses data.';
|
||||
|
||||
describe('classifyOutsideReview: separate execution, findings and verdict', () => {
|
||||
const cases: Array<[string, { text: string; gate: OutsideGate; stderr?: string; exit?: number }, string, string | null, string | undefined]> = [
|
||||
['clean review', { text: RECOMMEND, gate: 'review' }, 'clean', null, undefined],
|
||||
['review with P1', { text: `[P1] data loss\n${RECOMMEND}`, gate: 'review' }, 'findings', 'P1', undefined],
|
||||
['review with only P2', { text: `[P2] naming\n${RECOMMEND}`, gate: 'review' }, 'clean', 'P2', undefined],
|
||||
['review missing recommendation', { text: 'A few observations.', gate: 'review' }, 'unavailable', null, 'missing_markers'],
|
||||
['structured untagged prose', { text: 'The change looks reasonable overall.', gate: 'structured' }, 'unverified', null, 'untagged_review'],
|
||||
['structured explicit clear', { text: 'NO_FINDINGS', gate: 'structured' }, 'clean', null, undefined],
|
||||
['structured native P1 label', { text: 'P1: race in cleanup', gate: 'structured' }, 'findings', 'P1', undefined],
|
||||
['spec passing score', { text: 'SCORE: 8\nAMBIGUITIES: NONE', gate: 'spec' }, 'clean', null, undefined],
|
||||
['spec failing score', { text: 'SCORE: 4\nAMBIGUITIES: scope', gate: 'spec' }, 'findings', null, undefined],
|
||||
['nonzero exit', { text: RECOMMEND, gate: 'review', exit: 1, stderr: '\nerror: 401 Unauthorized\n' }, 'unavailable', null, 'execution_failed'],
|
||||
['timeout', { text: 'Partial', gate: 'review', exit: 124 }, 'unavailable', null, 'timeout'],
|
||||
['empty', { text: ' \n', gate: 'structured' }, 'unavailable', null, 'empty_response'],
|
||||
['refusal', { text: `I cannot review this request. ${RECOMMEND}`, gate: 'review' }, 'unavailable', null, 'review_refused'],
|
||||
['consult answer without review markers', { text: 'Use a queue here; the writer pool blocks.', gate: 'execution', stderr: '' }, 'clean', null, undefined],
|
||||
];
|
||||
for (const [label, input, verdict, highest, reason] of cases) {
|
||||
test(label, () => {
|
||||
const result = classifyOutsideReview(input);
|
||||
expect(result.verdict).toBe(verdict as any);
|
||||
expect(result.findings.highest).toBe(highest as any);
|
||||
expect(result.reason).toBe(reason as any);
|
||||
expect(result.execution.state).toBe(['execution_failed', 'timeout', 'empty_response', 'review_refused'].includes(reason ?? '') ? 'unavailable' : 'ran');
|
||||
});
|
||||
}
|
||||
|
||||
test('a non-zero exit names the exit code and the first stderr line', () => {
|
||||
const result = classifyOutsideReview({ text: '', gate: 'review', exit: 1, stderr: '\nerror: 401 Unauthorized\nmore' });
|
||||
expect(result.detail).toBe('exit 1: error: 401 Unauthorized');
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateOutsideReview keeps the text-only shape for direct importers', () => {
|
||||
test('existing shapes are unchanged', () => {
|
||||
expect(validateOutsideReview('', 'review')).toEqual({ completed: false, reason: 'empty response' });
|
||||
expect(validateOutsideReview('I must refuse.', 'review')).toEqual({ completed: false, reason: 'review refused' });
|
||||
expect(validateOutsideReview('notes', 'review')).toEqual({ completed: false, reason: 'missing review completion recommendation' });
|
||||
expect(validateOutsideReview('notes', 'structured')).toEqual({ completed: false, reason: 'missing severity or explicit no-findings conclusion' });
|
||||
expect(validateOutsideReview('SCORE: 8', 'spec')).toEqual({ completed: false, reason: 'missing or invalid SCORE/AMBIGUITIES markers' });
|
||||
expect(validateOutsideReview('SCORE: 6\nAMBIGUITIES: NONE', 'spec')).toEqual({ completed: true, score: 6, gate: 'fail' });
|
||||
expect(validateOutsideReview(RECOMMEND, 'review')).toEqual({ completed: true });
|
||||
expect(validateOutsideReview('[P2] nit', 'structured')).toEqual({ completed: true, gate: 'pass' });
|
||||
expect(validateOutsideReview('[P1] bug', 'structured')).toEqual({ completed: true, gate: 'fail' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('outside-review-result CLI', () => {
|
||||
test('two-argument form keeps exit 0 completed / 1 unavailable / 2 usage, findings included', () => {
|
||||
expect(cli(['structured', file('[P1] data loss')]).status).toBe(0);
|
||||
expect(cli(['review', file(`[P0] data loss\n${RECOMMEND}`)]).status).toBe(0);
|
||||
const untagged = cli(['structured', file('looks fine')]);
|
||||
expect(untagged.status).toBe(1);
|
||||
expect(untagged.stderr).toBe('Outside review unavailable: missing severity or explicit no-findings conclusion; missing coverage.\n');
|
||||
expect(cli(['structured', path.join(TMP, 'missing-file')]).status).toBe(1);
|
||||
expect(cli(['bogus', file('x')]).status).toBe(2);
|
||||
expect(cli(['execution', file('x')]).status).toBe(2);
|
||||
expect(cli(['review']).status).toBe(2);
|
||||
// No VERDICT line: stale callers only read the status.
|
||||
expect(cli(['structured', file('[P1] data loss')]).stdout).toBe('');
|
||||
});
|
||||
|
||||
test('verdict form prints VERDICT and exits 0 clean / 3 findings / 4 unverified / 1 unavailable', () => {
|
||||
const clean = cli(['--verdict', 'review', file(RECOMMEND)]);
|
||||
expect([clean.status, clean.stdout]).toEqual([0, 'VERDICT: clean\nFINDINGS: none\n']);
|
||||
const findings = cli(['--verdict', 'structured', file('[P0] corrupts data')]);
|
||||
expect([findings.status, findings.stdout]).toEqual([3, 'VERDICT: findings\nFINDINGS: P0\n']);
|
||||
const unverified = cli(['--verdict', 'structured', file('looks fine')]);
|
||||
expect(unverified.status).toBe(4);
|
||||
expect(unverified.stdout).toContain('VERDICT: unverified\n');
|
||||
expect(unverified.stderr).toContain('Outside review: ran, verdict unverified');
|
||||
const unavailable = cli(['--verdict', '--label', 'Codex outside review', 'review', file('')]);
|
||||
expect(unavailable.status).toBe(1);
|
||||
expect(unavailable.stdout).toContain('VERDICT: unavailable\nFINDINGS: none\nREASON: empty_response\n');
|
||||
expect(unavailable.stderr).toMatch(/^Codex outside review unavailable: the reviewer returned no response\. .*Fix: /);
|
||||
});
|
||||
|
||||
test('--stderr and --exit imply the verdict form; a missing stderr file reads as empty', () => {
|
||||
const failed = cli(['--exit', '7', '--stderr', file('fatal: provider exploded\n'), 'review', file(RECOMMEND)]);
|
||||
expect(failed.status).toBe(1);
|
||||
expect(failed.stdout).toContain('REASON: execution_failed');
|
||||
expect(failed.stderr).toContain('(exit 7: fatal: provider exploded)');
|
||||
const missing = cli(['--stderr', path.join(TMP, 'never-written'), 'review', file(RECOMMEND)]);
|
||||
expect([missing.status, missing.stdout]).toEqual([0, 'VERDICT: clean\nFINDINGS: none\n']);
|
||||
expect(cli(['--exit', 'x', 'review', file(RECOMMEND)]).status).toBe(2);
|
||||
});
|
||||
|
||||
test('an old rendered caller (pre-verdict skill text) still records findings as completed', () => {
|
||||
// The tail every generated outside-review fence carried before the verdict
|
||||
// form; bin/ and lib/ run live from the checkout, rendered text may lag.
|
||||
const response = file('[P1] Seeded data-loss bug\nREVIEW_COMPLETE');
|
||||
const old = `bun "$LIB" structured "$RESPONSE" || exit 1\necho 'OUTSIDE_STATUS: completed provider=codex host=claude'`;
|
||||
const r = spawnSync('bash', ['-c', old], { encoding: 'utf8', timeout: 10000,
|
||||
env: { ...process.env, LIB, RESPONSE: response } });
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stdout).toBe('OUTSIDE_STATUS: completed provider=codex host=claude\n');
|
||||
});
|
||||
});
|
||||
@@ -196,6 +196,21 @@ describe('generated outside-review dispatch', () => {
|
||||
}
|
||||
});
|
||||
|
||||
test('INV-1: the generated caller branches on VERDICT: findings complete, untagged is unverified', () => {
|
||||
for (const host of ['claude', 'codex'] as const) {
|
||||
const findings = invoke(host, { structuredBase: 'main', gate: 'structured' }, { FAKE_RESPONSE: '[P0] Seeded corruption' });
|
||||
expect(findings.status).toBe(0);
|
||||
expect(findings.stdout).toContain('VERDICT: findings\nFINDINGS: P0\n');
|
||||
expect(findings.stdout).toContain('OUTSIDE_STATUS: completed');
|
||||
const untagged = invoke(host, { structuredBase: 'main', gate: 'structured' }, { FAKE_RESPONSE: 'The change reads fine to me.' });
|
||||
expect(untagged.status).toBe(4);
|
||||
expect(untagged.stdout).toContain('VERDICT: unverified');
|
||||
expect(untagged.stdout).toContain('OUTSIDE_STATUS: unverified');
|
||||
expect(untagged.stdout).not.toContain('OUTSIDE_STATUS: completed');
|
||||
expect(untagged.stderr).toContain('ran, verdict unverified');
|
||||
}
|
||||
});
|
||||
|
||||
test('malformed Claude JSON cannot reach completion evaluation', () => {
|
||||
const result = invoke('codex',{}, {FAKE_MODE:'malformed'});
|
||||
expect(result.status).toBe(1);
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
/** INV-1: every gate reason code users can see has a section in docs/troubleshooting.md. */
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { GATE_OUTCOMES, gateOutcomeLine, type GateReason } from '../lib/gate-outcomes';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const doc = fs.readFileSync(path.join(ROOT, 'docs/troubleshooting.md'), 'utf8');
|
||||
const anchors = new Set([...doc.matchAll(/<a id="([a-z0-9-]+)"><\/a>/g)].map(m => m[1]));
|
||||
|
||||
describe('docs/troubleshooting.md anchors', () => {
|
||||
for (const [code, outcome] of Object.entries(GATE_OUTCOMES)) {
|
||||
test(`${code} → #${outcome.anchor}`, () => {
|
||||
expect(anchors.has(outcome.anchor)).toBe(true);
|
||||
// The anchor introduces a section: the next line is its heading.
|
||||
const after = doc.slice(doc.indexOf(`<a id="${outcome.anchor}"></a>`)).split('\n')[1] ?? '';
|
||||
expect(after).toMatch(/^#{2,4} /);
|
||||
});
|
||||
}
|
||||
|
||||
test('anchors are unique and every outcome has a fix action', () => {
|
||||
const all = [...doc.matchAll(/<a id="([a-z0-9-]+)"><\/a>/g)].map(m => m[1]);
|
||||
expect(new Set(all).size).toBe(all.length);
|
||||
const codes = Object.values(GATE_OUTCOMES).map(o => o.anchor);
|
||||
expect(new Set(codes).size).toBe(codes.length);
|
||||
for (const outcome of Object.values(GATE_OUTCOMES)) expect(outcome.fix.trim().length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('outcome lines follow the not run / unavailable / unverified shapes', () => {
|
||||
const line = (code: GateReason, detail?: string) => gateOutcomeLine('Codex outside review', code, detail);
|
||||
expect(line('disabled')).toBe('Codex outside review: not run (outside reviews are turned off (codex_reviews=disabled)). Fix: gstack-config set codex_reviews enabled.');
|
||||
expect(line('sandbox_unavailable', 'bwrap: No permissions to create new namespace'))
|
||||
.toBe("Codex outside review unavailable: Codex's sandbox could not start here (bwrap: No permissions to create new namespace). No review ran; this is missing coverage, not a pass. Fix: enable unprivileged user namespaces for this container, or set GSTACK_CODEX_NO_SANDBOX=1.");
|
||||
expect(line('untagged_review')).toMatch(/^Codex outside review: ran, verdict unverified \(.+\)\. Fix: read the output above/);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user