v1.91.45.0 feat: configurable model tiers and advisory freshness (#3073)

* feat: add configurable model tiers and advisory freshness

* fix: disclose role selections before Claude review execution

* fix: reject malformed model policies and keep model pins portable

* fix: separate readiness diagnostics and bound version checks

* fix: preserve actionable model failures and validate review inputs before probes

* fix: retry released probe locks and retain late-rejection repair context

* feat: finalize configurable model tiers and verified release repairs

* fix: preserve strict model policy reads on Windows

* fix: keep fixture probes read-only and clarify pending CEO decisions

* fix: verify native terminal closure before bounded descriptor settlement
This commit is contained in:
Garry Tan authored and GitHub committed 2026-10-08 08:00:08 +00:00
1 parent 52eec18c63
commit 92cfd07a79
96 files changed
+6539 -411

No files matched your search

+241
View File
@@ -0,0 +1,241 @@
# Model policy
Use a frontier model to challenge a plan, then a smart model to build it.
These are curated recommendations, not a claim that one model wins every task.
User choices take precedence, and a recommendation never switches the running
session or edits the host's native configuration.
| Tier | Default job | Anthropic | OpenAI |
| --- | --- | --- | --- |
| `frontier` | Independent plan review | `claude-fable-5-1` | `gpt-6-astra` |
| `smart` | Implementation recommendation | `claude-opus-5-5` | `gpt-6.1-sol` |
## Inspect and change your policy
This quickstart assumes gstack is installed. It uses no network, credentials or
paid model calls. Run it from the project's repository root in Bash, Git Bash or MSYS:
automated outside reviews use that directory. Inspection from a subdirectory
reflects that subdirectory's native settings and may differ. If your install is vendored
or elsewhere, set `GSTACK_ROOT` to that installed gstack runtime instead.
```bash
if [ -x "$HOME/.claude/skills/gstack/bin/gstack-models" ]; then
GSTACK_ROOT="$HOME/.claude/skills/gstack"
elif [ -x "${CODEX_HOME:-$HOME/.codex}/skills/gstack/bin/gstack-models" ]; then
GSTACK_ROOT="${CODEX_HOME:-$HOME/.codex}/skills/gstack"
else
printf '%s\n' 'Upgrade the gstack install for your host, then rerun this quickstart.' >&2
exit 1
fi
"$GSTACK_ROOT/bin/gstack-models" list
"$GSTACK_ROOT/bin/gstack-models" resolve --role plan-review --provider openai
```
The output explains the selected model and which setting won. It also shows
catalog verification metadata. Offline selection does **not** prove that a CLI
is installed, that your account can use the model, or that a vendor's current
catalog has been checked.
Choose the less expensive review tier, then inspect the effective result:
```bash
"$GSTACK_ROOT/bin/gstack-config" set plan_review_tier smart
"$GSTACK_ROOT/bin/gstack-models" resolve --role plan-review --provider openai
```
Pin a provider's model within a tier, or reset a pin to the shipped default:
```bash
"$GSTACK_ROOT/bin/gstack-config" set model_frontier_openai gpt-6-astra
"$GSTACK_ROOT/bin/gstack-config" set model_frontier_claude claude-fable-5-1
"$GSTACK_ROOT/bin/gstack-config" unset model_frontier_openai
"$GSTACK_ROOT/bin/gstack-config" unset plan_review_tier
```
To keep using native CLI model settings for plan reviews:
```bash
"$GSTACK_ROOT/bin/gstack-config" set plan_review_tier host
```
Settings apply to the next invocation without rerendering skills. An invocation
already being checked keeps its selected model through execution. Configuration
uses the existing [gstack state root](state-root.md), not a new per-repository
settings file. Invalid writes are rejected and leave the previous value intact;
use `unset`, not an empty value, to restore a default.
A malformed known setting (for example `plan_review_tier = smart`) stops
selection rather than acting as an absent key. Correct or remove the named line
in the reported config file; `unset` only removes correctly spelled `key:` records.
Codex plan-review readiness and dispatch share one deadline. Readiness is
budgeted half the remaining time, with the existing termination grace. A slow
preflight leaves less time for the review instead of extending the caller's
limit. An expired deadline starts no new command and means missing review
coverage, not a completed review.
## Settings and precedence
| Key | Default | Accepted value |
| --- | --- | --- |
| `plan_review_tier` | `frontier` | `frontier`, `smart`, `host` |
| `implementation_tier` | `smart` | `frontier`, `smart` |
| `model_frontier_claude` | Catalog's Anthropic frontier model | Explicit Anthropic model identifier |
| `model_frontier_openai` | Catalog's OpenAI frontier model | Explicit OpenAI model identifier |
| `model_smart_claude` | Catalog's Anthropic smart model | Explicit Anthropic model identifier |
| `model_smart_openai` | Catalog's OpenAI smart model | Explicit OpenAI model identifier |
For plan reviews, a model named for that invocation wins over
`GSTACK_CLAUDE_MODEL` or `GSTACK_CODEX_MODEL`. That environment override wins over
the selected tier's configured model, which wins over the shipped catalog.
Generic coding-model settings do not silently select a cheaper plan reviewer;
choose `host` mode when that is the behavior you want.
In `host` mode, Codex keeps its existing native-config resolution, including
`CODEX_HOME`, active profiles and the `exec` versus native `review` distinction.
Claude can delegate model selection to its CLI without forcing `--model`; the
inspector reports that delegation rather than inventing an exact model ID.
Actual returned model usage remains separate from the requested model.
Implementation recommendations use a model explicitly requested for that job,
then the configured implementation tier and its provider override, then the
catalog. Outside-review environment overrides do not change implementation
recommendations. Known Claude Code and Codex hosts recommend their own provider;
an unknown or multi-provider host can show both choices.
Custom endpoints and partner platforms need an explicitly compatible model or
`host` mode. Gstack does not guess that public API IDs work with Bedrock, Vertex,
Foundry or a custom Codex provider. A model identifier passing syntax validation
is not proof of account availability. See [provider recovery](troubleshooting.md#model-policy-provider).
Detection is conservative, not a replacement for the CLI's settings engine.
Claude checks environment, user settings, project/shared/local settings, local
settings at the repository and worktree's main checkout, and local managed files.
Codex checks environment, user/active-profile settings and system configuration;
project-local Codex config is not inspected.
Any detected local custom-route signal requires an explicit model, even if
native trust or precedence rules might suppress it. Server-managed or MDM
policy and extra native CLI flags are not visible to this check; `--help` and
JSON output list these limits. Use an explicit compatible model for those cases,
or `host` mode for plan reviews. Implementation recommendations require an
explicit implementation model rather than delegating to a hidden native choice.
## Which workflows use the tiers?
The plan-review role applies to autoplan's CEO, design, DX and engineering
outside voices; standalone CEO, engineering, developer-experience and design
plan reviews; and the spec quality gate. Existing reasoning effort, access
restrictions, timeouts, review gates and `codex_reviews` consent remain in force.
The host selects the opposing provider independently from the tier. Claude Code
uses the Codex wrapper; Codex uses the Claude Code wrapper. A model's name does
not establish which harness is running.
Manual opposing-model wrappers can opt into `--role plan-review`. Ordinary
code reviews, live design reviews, design consultation, office-hours second
opinions and documentation reviews retain their existing model-selection
behavior. Historical no-role defaults, eval actors/judges, cheap helper models
and setup-time behavioral overlays do not follow this catalog.
Before the first affected paid plan review after upgrade, gstack explains the
independent frontier default and the winning source, with `smart` and `host`
alternatives. Inspection does not mark that notice as seen. Disabled outside
reviews stay disabled; an unavailable model is reported, never silently replaced.
## First completed outside review
Policy setup and a completed review are different milestones. Install and
authenticate the opposing CLI using the [host setup instructions](../README.md#install--about-30-seconds)
before invoking a paid review. From Claude Code, use `/codex consult --role
plan-review`. From Codex, use `/claude-code consult --role plan-review` (or its
installed namespaced name). Ask it to challenge the plan and supply the full plan
text. Consult is prompt-driven and works before there is a repository diff;
Challenge mode is for reviewing code changes.
Confirm the announced selection, the provider's actual result and the coverage
status. A resolved model, a successful availability probe or a native fallback
is not a completed outside review. An unusable explicit choice needs a repair
at the source named in the error; changing a lower-priority tier cannot override
an environment pin. See [selection recovery](troubleshooting.md#model-policy-selection).
## Machine-readable inspection
`gstack-models` with no arguments is `list`. Both `list` and `resolve` accept
`--json`. Plan-review resolution requires `--provider anthropic|openai`;
implementation resolution can omit the provider to return both recommendations.
```bash
"$GSTACK_ROOT/bin/gstack-models" resolve --role plan-review --provider anthropic --json
"$GSTACK_ROOT/bin/gstack-models" resolve --role implementation --json
```
JSON output is one document with `schemaVersion: 1`, a `selections` array and
structured errors on failure. Human diagnostics go to stderr. Exit `0` means
valid resolution, including host delegation; `1` means configuration or
selection failure; `2` means invalid usage. Neither inspection nor resolution
writes notice markers, probes a model or fetches vendor sources.
## Keeping shipped recommendations current
The model-policy freshness workflow runs weekly and supports manual dispatch.
It checks official Anthropic and OpenAI recommendation and deprecation sources,
without provider credentials or paid model calls. Normal PR checks use offline
fixtures; a vendor outage does not make unrelated PRs fail.
In GitHub Actions, select **Model-policy freshness** to run it manually. From a
gstack checkout, `bun scripts/model-policy-freshness.ts` performs the same public
source check without changing an issue; unlike `gstack-models`, it uses the
network. The workflow's guarded `--publish` path owns issue updates. Its JSON
report and job summary retain the source evidence.
The report's primary status distinguishes `current`, `update-candidate` and
`unknown/source-unavailable`. Its separate `evidenceStatus` marks retained
successful evidence `stale` after thirty days; a failed fetch remains unknown
in the headline even when that retained evidence is stale.
Newer or newly recommended models are candidates,
not automatic replacements. A support promise such as “not sooner than” is not
a scheduled retirement, and an incidental model mention is not a new default.
This evidence covers public API recommendations, not partner-platform entitlement.
One Actions-bot-owned tracking issue preserves the successful check's catalog/source/
parser identity and any unresolved lifecycle findings. Healthy state remains
readable when the issue is closed. A failed or partial check cannot erase a known
retirement, borrow a success from an older catalog, or pretend its evidence is
fresh. A catalog rollback reopens historical findings for a model put back in
use. Only current default-branch runs may update that issue; copied markers in
user-owned issues are not trusted state.
A documented retain decision preserves the current default, but does not make a
vendor mismatch `current`: the advisory issue stays open, or reopens, while that
candidate remains. Record retain rationales in issue comments, which the workflow
never edits, rather than racing a scheduled update to the issue body.
Thirty-day staleness means an absence of trustworthy freshness evidence, not
that a model was released thirty days ago. An unchanged recommendation does not
require a date-only catalog commit or a monthly paid benchmark. The recorded
next-check due date helps maintenance, but a workflow that never runs cannot
alert on its own absence.
### Maintainer update checklist
1. Check the latest freshness run and its due date. Resolve missing, corrupt or
source-unavailable evidence before declaring the catalog current.
2. Verify the official recommendation and lifecycle evidence. Triage announced
retirement/removal within one working day, stale/unavailable evidence within
seven days and discretionary update candidates within thirty days.
3. Open a catalog change with the source URLs, exact model IDs and qualification
evidence, or record why the current default should remain. Never edit users'
personal pins or silently substitute another provider.
4. Confirm CLI model/profile support and tool/reasoning compatibility. Keep the
pinned test runtimes and benchmark rulers stable. A required CLI upgrade is a
separate reviewed change, not an automatic side effect of discovery.
5. Predeclare any budgeted baseline/candidate comparison using existing eval
machinery, fixed prompts/panels and unchanged thresholds. Keep every attempt,
measure useful and unsupported findings, latency and available usage, and do
not treat vendor marketing or a creation timestamp as quality evidence.
6. Pass the affected free checks and selected workflow evaluations, review the
change, and ship it through the normal release process. Only unpinned
role-bearing defaults follow the new shipped catalog.
The workflow does not open an automatic model-upgrade PR, merge changes, switch
active sessions, alter benchmark models or schedule paid evaluations.
+9 -5
View File
@@ -1148,10 +1148,10 @@ This is my **second opinion mode**.
`/codex` brings OpenAI Codex CLI to review the same diff independently. It is available on every harness except Codex itself. External harnesses install it as `/gstack-codex`. Compare its findings with the native review to distinguish corroborated findings from issues only one reviewer caught.
gstack-owned Codex calls default to `gpt-6-astra`, including resumed consult
sessions. Set `GSTACK_CODEX_MODEL=<model>` to change the default, or name a
model in your request to override it for that invocation. Generated commands
pass the selection through `-c model=...`, overriding the CLI's configured model.
Without a role, gstack-owned Codex calls select a request-specific model, then
`GSTACK_CODEX_MODEL`, then Codex native settings, falling back to `gpt-6-astra`.
This also applies to resumed consultations. Generated commands pass the selection
through `-c model=...`, overriding the CLI's configured model for that call.
Native review also sets `-c review_model=...` to that selection, overriding any
separate review-model pin.
@@ -1161,6 +1161,10 @@ review, challenge, and consult calls preserve Claude's configured model.
sessions; a model named in your request takes precedence. Harness routing is
independent of model selection.
An explicit `--role plan-review` uses the configured plan-review tier instead;
request and environment overrides still win, and `host` mode restores native
selection. See [model policy](model-policy.md) for pins, precedence and recovery.
### Three modes
**Review** — run `codex review` against the current diff. Codex reads every changed file, classifies findings by severity (P1 critical, P2 high, P3 medium), and returns a PASS/FAIL verdict. Any P1 finding = FAIL. The review is fully independent — Codex doesn't see Claude's review.
@@ -1200,7 +1204,7 @@ Claude Code provides the outside reviewer when gstack runs in Codex. Other non-C
**Review** supplies the branch diff for a read-only pass/fail review. **Challenge** asks Claude Code to find concrete failure cases in the same diff. **Consult** supports read-only repository exploration and resumes the session saved in `.context/claude-session-id`. Review and challenge receive context from the parent workflow and run without tools; consultation can read and search files.
The Claude Code CLI must be installed and authenticated. Its existing model configuration and `GSTACK_CLAUDE_BIN` / `GSTACK_CLAUDE_BIN_ARGS` executable overrides are honored. Errors, timeouts, and invalid responses report missing outside coverage instead of a clean review. Automatic reviews start fresh; consult session continuity is explicit.
The Claude Code CLI must be installed and authenticated. Without a role, its existing model configuration is honored; explicit `--role plan-review` follows the [model policy](model-policy.md). `GSTACK_CLAUDE_BIN` / `GSTACK_CLAUDE_BIN_ARGS` executable overrides remain in force. Errors, timeouts, and invalid responses report missing outside coverage instead of a clean review. Automatic reviews start fresh; consult session continuity is explicit.
Outside-review routing follows the harness, independently of the configured model. Generic second-opinion requests choose `/claude-code` on Codex and `/codex` elsewhere; explicit provider requests keep that provider. The existing `codex_reviews` setting controls the selected automatic reviewer in workflows that already use that setting. Existing opt-in and skip controls still apply in office hours, design, and spec workflows.
+107
View File
@@ -34,6 +34,113 @@ no paid call: the Codex rows report the cached model check and its age, and
---
## Model policy
The [model-policy guide](model-policy.md) explains the two tiers, all six
settings, precedence and the read-only inspection command. A resolved selection
is not proof that the requested provider or model can run.
<a id="model-policy-config"></a>
### Model policy configuration is invalid or unreadable
**Meaning.** The new policy path could not read a coherent configuration or a
configured value is invalid. An unreadable file is not treated as an absent
setting: doing that could silently select the premium default.
**Fix.** Repair the file or key named by the error. `plan_review_tier` accepts
`frontier`, `smart` or `host`; `implementation_tier` accepts `frontier` or
`smart`. Use `gstack-config unset <key>` to restore a default, not an empty
value. A rejected `set` leaves the old value intact. Correct ownership/read
permissions rather than deleting an unrelated state directory.
For a malformed line such as `plan_review_tier = smart`, edit the reported file
and correct or remove that line first: `unset` only removes canonical `key:`
records and cannot repair a misspelled delimiter or quoted key.
**Expected result.** `gstack-models list` or the corresponding `resolve`
command explains the winning source. No outside model call ran on this error.
<a id="model-policy-provider"></a>
### A custom provider needs an explicit model or host mode
**Meaning.** Native settings or environment flags route the reviewer to a
custom endpoint or partner platform. A public API catalog identifier is not
necessarily a valid deployment identifier there. Malformed native provider
configuration also prevents gstack from assuming the public endpoint.
**Fix.** Supply a compatible model for that request, set the provider's gstack
environment override or tier-model setting, or choose
`gstack-config set plan_review_tier host`. Keep credentials in the provider's
normal authentication mechanism, never in a model setting. If the error names
a malformed native configuration, fix that source first.
**Expected result.** Inspection reports an explicit selection or honest
host-managed delegation. It does not certify account entitlement or send a
public default to an inferred custom endpoint.
<a id="model-policy-selection"></a>
### The selected review model could not be used
**Meaning.** The requested model was rejected or the provider could not
complete the review. A model may be nonexistent, unavailable to the account,
retired, or behind a misconfigured endpoint; preserve the provider's evidence
rather than treating all of these as the same diagnosis. No successful outside
review is implied, and gstack does not substitute a different model.
**Fix.** Change the source named in the error. For example, an environment
override beats a tier setting, so editing the tier cannot replace it. Unset or
correct that override, or explicitly name a supported model for the request.
Follow the existing auth, quota, sandbox or timeout repair when that is the
reported failure. Missing usage identity remains unknown, even if inspection
showed a requested ID.
**Expected result.** The next invocation announces its model and source,
checks that same selection and reports the actual provider result. A cached
probe or a native fallback is not a completed outside review.
<a id="model-policy-freshness"></a>
### Model-catalog freshness is unknown, stale or needs an update
**Meaning.** Official evidence changed, a required source could not be
checked, or the retained evidence does not match this catalog and parser.
`unknown/source-unavailable` is not current. A later source failure must not
erase an earlier retirement finding or close its maintenance issue.
**Fix.** Read the failed source and evidence details in the freshness workflow
report and tracking issue. Repair the source parser or publication access when
needed, then run the workflow manually against the default branch. If a model
was superseded or deprecated, follow the [maintainer checklist](model-policy.md#maintainer-update-checklist)
to qualify a replacement or record a supported mitigation. Do not clear the
issue just because a new release has not been adopted yet.
**Expected result.** Complete source checks produce evidence bound to the
current catalog, while unresolved lifecycle findings remain visible. Personal
pins and unrelated PRs are unchanged. Check the workflow's last run date too:
a scheduled job that never runs cannot report its own absence.
#### Repairing `recoveryRequired` state
This flag deliberately keeps the issue open until a maintainer repairs the
record. Another successful fetch alone cannot certify corrupted history.
1. Save the current issue body, including its retained recovery archive, and
download the latest trustworthy workflow report. Use its `report.state` as
the starting state; do not invent run IDs, catalog hashes or check dates.
2. Reconcile its `lifecycle` array with the issue's independent lifecycle block
and any retained archive. Preserve known retirements, deadlines and human
dispositions. A model currently in the catalog must not keep an old
`resolvedByCatalog` flag from a previous replacement.
3. In the issue editor, restore valid JSON between the `state:begin` and
`state:end` marker lines. Put that same lifecycle array between
`lifecycle:begin` and `lifecycle:end`. Restore the standalone ownership and
region markers, and leave human text outside the owned region intact.
4. Set `recoveryRequired` to `false` only after that reconciliation. If prior
successful evidence cannot be verified, set `lastSuccess` to `null` rather
than making up a fresh success. If lifecycle history cannot be recovered,
leave the flag set and the issue open for investigation.
5. Run **Model-policy freshness** against the default branch. Only complete,
current evidence with no unresolved findings can close the issue. A parsing
or source error leaves it open; do not clear history to manufacture a pass.
## Outside reviews (Codex and Claude Code)
Outside reviews send your diff, plan or question to a second AI provider. Their