mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-12 03:30:28 +01:00
feat: add configurable model tiers and advisory freshness
This commit is contained in:
garrytan committed
1 parent
db745675bd
commit
5ffc050659
78 files changed
+5106
-325
No files matched your search
@@ -0,0 +1,220 @@
|
||||
# Model policy
|
||||
|
||||
Use a frontier model to challenge a plan, then a smart model to build it.
|
||||
These are curated recommendations, not a claim that one model wins every task.
|
||||
User choices take precedence, and a recommendation never switches the running
|
||||
session or edits the host's native configuration.
|
||||
|
||||
| Tier | Default job | Anthropic | OpenAI |
|
||||
| --- | --- | --- | --- |
|
||||
| `frontier` | Independent plan review | `claude-fable-5-1` | `gpt-6-astra` |
|
||||
| `smart` | Implementation recommendation | `claude-opus-5-5` | `gpt-6.1-sol` |
|
||||
|
||||
## Inspect and change your policy
|
||||
|
||||
This quickstart assumes gstack is installed. It uses no network, credentials or
|
||||
paid model calls. Run it from the project's repository root in Bash, Git Bash or MSYS:
|
||||
automated outside reviews use that directory. Inspection from a subdirectory
|
||||
reflects that subdirectory's native settings and may differ. If your install is vendored
|
||||
or elsewhere, set `GSTACK_ROOT` to that installed gstack runtime instead.
|
||||
|
||||
```bash
|
||||
if [ -x "$HOME/.claude/skills/gstack/bin/gstack-models" ]; then
|
||||
GSTACK_ROOT="$HOME/.claude/skills/gstack"
|
||||
elif [ -x "${CODEX_HOME:-$HOME/.codex}/skills/gstack/bin/gstack-models" ]; then
|
||||
GSTACK_ROOT="${CODEX_HOME:-$HOME/.codex}/skills/gstack"
|
||||
else
|
||||
printf '%s\n' 'Upgrade the gstack install for your host, then rerun this quickstart.' >&2
|
||||
exit 1
|
||||
fi
|
||||
"$GSTACK_ROOT/bin/gstack-models" list
|
||||
"$GSTACK_ROOT/bin/gstack-models" resolve --role plan-review --provider openai
|
||||
```
|
||||
|
||||
The output explains the selected model and which setting won. It also shows
|
||||
catalog verification metadata. Offline selection does **not** prove that a CLI
|
||||
is installed, that your account can use the model, or that a vendor's current
|
||||
catalog has been checked.
|
||||
|
||||
Choose the less expensive review tier, then inspect the effective result:
|
||||
|
||||
```bash
|
||||
"$GSTACK_ROOT/bin/gstack-config" set plan_review_tier smart
|
||||
"$GSTACK_ROOT/bin/gstack-models" resolve --role plan-review --provider openai
|
||||
```
|
||||
|
||||
Pin a provider's model within a tier, or reset a pin to the shipped default:
|
||||
|
||||
```bash
|
||||
"$GSTACK_ROOT/bin/gstack-config" set model_frontier_openai gpt-6-astra
|
||||
"$GSTACK_ROOT/bin/gstack-config" set model_frontier_claude claude-fable-5-1
|
||||
"$GSTACK_ROOT/bin/gstack-config" unset model_frontier_openai
|
||||
"$GSTACK_ROOT/bin/gstack-config" unset plan_review_tier
|
||||
```
|
||||
|
||||
To keep using native CLI model settings for plan reviews:
|
||||
|
||||
```bash
|
||||
"$GSTACK_ROOT/bin/gstack-config" set plan_review_tier host
|
||||
```
|
||||
|
||||
Settings apply to the next invocation without rerendering skills. An invocation
|
||||
already being checked keeps its selected model through execution. Configuration
|
||||
uses the existing [gstack state root](state-root.md), not a new per-repository
|
||||
settings file. Invalid writes are rejected and leave the previous value intact;
|
||||
use `unset`, not an empty value, to restore a default.
|
||||
|
||||
## Settings and precedence
|
||||
|
||||
| Key | Default | Accepted value |
|
||||
| --- | --- | --- |
|
||||
| `plan_review_tier` | `frontier` | `frontier`, `smart`, `host` |
|
||||
| `implementation_tier` | `smart` | `frontier`, `smart` |
|
||||
| `model_frontier_claude` | Catalog's Anthropic frontier model | Explicit Anthropic model identifier |
|
||||
| `model_frontier_openai` | Catalog's OpenAI frontier model | Explicit OpenAI model identifier |
|
||||
| `model_smart_claude` | Catalog's Anthropic smart model | Explicit Anthropic model identifier |
|
||||
| `model_smart_openai` | Catalog's OpenAI smart model | Explicit OpenAI model identifier |
|
||||
|
||||
For plan reviews, a model named for that invocation wins over
|
||||
`GSTACK_CLAUDE_MODEL` or `GSTACK_CODEX_MODEL`. That environment override wins over
|
||||
the selected tier's configured model, which wins over the shipped catalog.
|
||||
Generic coding-model settings do not silently select a cheaper plan reviewer;
|
||||
choose `host` mode when that is the behavior you want.
|
||||
|
||||
In `host` mode, Codex keeps its existing native-config resolution, including
|
||||
`CODEX_HOME`, active profiles and the `exec` versus native `review` distinction.
|
||||
Claude can delegate model selection to its CLI without forcing `--model`; the
|
||||
inspector reports that delegation rather than inventing an exact model ID.
|
||||
Actual returned model usage remains separate from the requested model.
|
||||
|
||||
Implementation recommendations use a model explicitly requested for that job,
|
||||
then the configured implementation tier and its provider override, then the
|
||||
catalog. Outside-review environment overrides do not change implementation
|
||||
recommendations. Known Claude Code and Codex hosts recommend their own provider;
|
||||
an unknown or multi-provider host can show both choices.
|
||||
|
||||
Custom endpoints and partner platforms need an explicitly compatible model or
|
||||
`host` mode. Gstack does not guess that public API IDs work with Bedrock, Vertex,
|
||||
Foundry or a custom Codex provider. A model identifier passing syntax validation
|
||||
is not proof of account availability. See [provider recovery](troubleshooting.md#model-policy-provider).
|
||||
|
||||
Detection is conservative, not a replacement for the CLI's settings engine.
|
||||
Claude checks environment, user settings, project/shared/local settings, local
|
||||
settings at the repository and worktree's main checkout, and local managed files.
|
||||
Codex checks environment, user/active-profile settings and system configuration.
|
||||
Any detected local custom-route signal requires an explicit model, even if
|
||||
native trust or precedence rules might suppress it. Server-managed or MDM
|
||||
policy and extra native CLI flags are not visible to this check; `--help` and
|
||||
JSON output list these limits. Use an explicit compatible model for those cases,
|
||||
or `host` mode for plan reviews. Implementation recommendations require an
|
||||
explicit implementation model rather than delegating to a hidden native choice.
|
||||
|
||||
## Which workflows use the tiers?
|
||||
|
||||
The plan-review role applies to autoplan's CEO, design, DX and engineering
|
||||
outside voices; standalone CEO, engineering, developer-experience and design
|
||||
plan reviews; and the spec quality gate. Existing reasoning effort, access
|
||||
restrictions, timeouts, review gates and `codex_reviews` consent remain in force.
|
||||
|
||||
The host selects the opposing provider independently from the tier. Claude Code
|
||||
uses the Codex wrapper; Codex uses the Claude Code wrapper. A model's name does
|
||||
not establish which harness is running.
|
||||
|
||||
Manual opposing-model wrappers can opt into `--role plan-review`. Ordinary
|
||||
code reviews, live design reviews, design consultation, office-hours second
|
||||
opinions and documentation reviews retain their existing model-selection
|
||||
behavior. Historical no-role defaults, eval actors/judges, cheap helper models
|
||||
and setup-time behavioral overlays do not follow this catalog.
|
||||
|
||||
Before the first affected paid plan review after upgrade, gstack explains the
|
||||
independent frontier default and the winning source, with `smart` and `host`
|
||||
alternatives. Inspection does not mark that notice as seen. Disabled outside
|
||||
reviews stay disabled; an unavailable model is reported, never silently replaced.
|
||||
|
||||
## First completed outside review
|
||||
|
||||
Policy setup and a completed review are different milestones. Install and
|
||||
authenticate the opposing CLI using the [host setup instructions](../README.md#install--about-30-seconds)
|
||||
before invoking a paid review. From Claude Code, use `/codex challenge --role
|
||||
plan-review` with your plan. From Codex, use `/claude-code challenge --role
|
||||
plan-review` (or its installed namespaced name).
|
||||
|
||||
Confirm the announced selection, the provider's actual result and the coverage
|
||||
status. A resolved model, a successful availability probe or a native fallback
|
||||
is not a completed outside review. An unusable explicit choice needs a repair
|
||||
at the source named in the error; changing a lower-priority tier cannot override
|
||||
an environment pin. See [selection recovery](troubleshooting.md#model-policy-selection).
|
||||
|
||||
## Machine-readable inspection
|
||||
|
||||
`gstack-models` with no arguments is `list`. Both `list` and `resolve` accept
|
||||
`--json`. Plan-review resolution requires `--provider anthropic|openai`;
|
||||
implementation resolution can omit the provider to return both recommendations.
|
||||
|
||||
```bash
|
||||
"$GSTACK_ROOT/bin/gstack-models" resolve --role plan-review --provider anthropic --json
|
||||
"$GSTACK_ROOT/bin/gstack-models" resolve --role implementation --json
|
||||
```
|
||||
|
||||
JSON output is one document with `schemaVersion: 1`, a `selections` array and
|
||||
structured errors on failure. Human diagnostics go to stderr. Exit `0` means
|
||||
valid resolution, including host delegation; `1` means configuration or
|
||||
selection failure; `2` means invalid usage. Neither inspection nor resolution
|
||||
writes notice markers, probes a model or fetches vendor sources.
|
||||
|
||||
## Keeping shipped recommendations current
|
||||
|
||||
The model-policy freshness workflow runs weekly and supports manual dispatch.
|
||||
It checks official Anthropic and OpenAI recommendation and deprecation sources,
|
||||
without provider credentials or paid model calls. Normal PR checks use offline
|
||||
fixtures; a vendor outage does not make unrelated PRs fail.
|
||||
|
||||
In GitHub Actions, select **Model-policy freshness** to run it manually. From a
|
||||
gstack checkout, `bun scripts/model-policy-freshness.ts` performs the same public
|
||||
source check without changing an issue; unlike `gstack-models`, it uses the
|
||||
network. The workflow's guarded `--publish` path owns issue updates. Its JSON
|
||||
report and job summary retain the source evidence.
|
||||
|
||||
The report distinguishes `current`, `update-candidate`, `stale` and
|
||||
`unknown/source-unavailable`. Newer or newly recommended models are candidates,
|
||||
not automatic replacements. A support promise such as “not sooner than” is not
|
||||
a scheduled retirement, and an incidental model mention is not a new default.
|
||||
This evidence covers public API recommendations, not partner-platform entitlement.
|
||||
|
||||
One Actions-bot-owned tracking issue preserves the successful check's catalog/source/
|
||||
parser identity and any unresolved lifecycle findings. Healthy state remains
|
||||
readable when the issue is closed. A failed or partial check cannot erase a known
|
||||
retirement, borrow a success from an older catalog, or pretend its evidence is
|
||||
fresh. A catalog rollback reopens historical findings for a model put back in
|
||||
use. Only current default-branch runs may update that issue; copied markers in
|
||||
user-owned issues are not trusted state.
|
||||
|
||||
Thirty-day staleness means an absence of trustworthy freshness evidence, not
|
||||
that a model was released thirty days ago. An unchanged recommendation does not
|
||||
require a date-only catalog commit or a monthly paid benchmark. The recorded
|
||||
next-check due date helps maintenance, but a workflow that never runs cannot
|
||||
alert on its own absence.
|
||||
|
||||
### Maintainer update checklist
|
||||
|
||||
1. Check the latest freshness run and its due date. Resolve missing, corrupt or
|
||||
source-unavailable evidence before declaring the catalog current.
|
||||
2. Verify the official recommendation and lifecycle evidence. Triage announced
|
||||
retirement/removal within one working day, stale/unavailable evidence within
|
||||
seven days and discretionary update candidates within thirty days.
|
||||
3. Open a catalog change with the source URLs, exact model IDs and qualification
|
||||
evidence, or record why the current default should remain. Never edit users'
|
||||
personal pins or silently substitute another provider.
|
||||
4. Confirm CLI model/profile support and tool/reasoning compatibility. Keep the
|
||||
pinned test runtimes and benchmark rulers stable. A required CLI upgrade is a
|
||||
separate reviewed change, not an automatic side effect of discovery.
|
||||
5. Predeclare any budgeted baseline/candidate comparison using existing eval
|
||||
machinery, fixed prompts/panels and unchanged thresholds. Keep every attempt,
|
||||
measure useful and unsupported findings, latency and available usage, and do
|
||||
not treat vendor marketing or a creation timestamp as quality evidence.
|
||||
6. Pass the affected free checks and selected workflow evaluations, review the
|
||||
change, and ship it through the normal release process. Only unpinned
|
||||
role-bearing defaults follow the new shipped catalog.
|
||||
|
||||
The workflow does not open an automatic model-upgrade PR, merge changes, switch
|
||||
active sessions, alter benchmark models or schedule paid evaluations.
|
||||
@@ -22,6 +22,110 @@ section has a stable link anchor; the reason codes and anchors come from
|
||||
|
||||
---
|
||||
|
||||
## Model policy
|
||||
|
||||
The [model-policy guide](model-policy.md) explains the two tiers, all six
|
||||
settings, precedence and the read-only inspection command. A resolved selection
|
||||
is not proof that the requested provider or model can run.
|
||||
|
||||
<a id="model-policy-config"></a>
|
||||
### Model policy configuration is invalid or unreadable
|
||||
|
||||
**Meaning.** The new policy path could not read a coherent configuration or a
|
||||
configured value is invalid. An unreadable file is not treated as an absent
|
||||
setting: doing that could silently select the premium default.
|
||||
|
||||
**Fix.** Repair the file or key named by the error. `plan_review_tier` accepts
|
||||
`frontier`, `smart` or `host`; `implementation_tier` accepts `frontier` or
|
||||
`smart`. Use `gstack-config unset <key>` to restore a default, not an empty
|
||||
value. A rejected `set` leaves the old value intact. Correct ownership/read
|
||||
permissions rather than deleting an unrelated state directory.
|
||||
|
||||
**Expected result.** `gstack-models list` or the corresponding `resolve`
|
||||
command explains the winning source. No outside model call ran on this error.
|
||||
|
||||
<a id="model-policy-provider"></a>
|
||||
### A custom provider needs an explicit model or host mode
|
||||
|
||||
**Meaning.** Native settings or environment flags route the reviewer to a
|
||||
custom endpoint or partner platform. A public API catalog identifier is not
|
||||
necessarily a valid deployment identifier there. Malformed native provider
|
||||
configuration also prevents gstack from assuming the public endpoint.
|
||||
|
||||
**Fix.** Supply a compatible model for that request, set the provider's gstack
|
||||
environment override or tier-model setting, or choose
|
||||
`gstack-config set plan_review_tier host`. Keep credentials in the provider's
|
||||
normal authentication mechanism, never in a model setting. If the error names
|
||||
a malformed native configuration, fix that source first.
|
||||
|
||||
**Expected result.** Inspection reports an explicit selection or honest
|
||||
host-managed delegation. It does not certify account entitlement or send a
|
||||
public default to an inferred custom endpoint.
|
||||
|
||||
<a id="model-policy-selection"></a>
|
||||
### The selected review model could not be used
|
||||
|
||||
**Meaning.** The requested model was rejected or the provider could not
|
||||
complete the review. A model may be nonexistent, unavailable to the account,
|
||||
retired, or behind a misconfigured endpoint; preserve the provider's evidence
|
||||
rather than treating all of these as the same diagnosis. No successful outside
|
||||
review is implied, and gstack does not substitute a different model.
|
||||
|
||||
**Fix.** Change the source named in the error. For example, an environment
|
||||
override beats a tier setting, so editing the tier cannot replace it. Unset or
|
||||
correct that override, or explicitly name a supported model for the request.
|
||||
Follow the existing auth, quota, sandbox or timeout repair when that is the
|
||||
reported failure. Missing usage identity remains unknown, even if inspection
|
||||
showed a requested ID.
|
||||
|
||||
**Expected result.** The next invocation announces its model and source,
|
||||
checks that same selection and reports the actual provider result. A cached
|
||||
probe or a native fallback is not a completed outside review.
|
||||
|
||||
<a id="model-policy-freshness"></a>
|
||||
### Model-catalog freshness is unknown, stale or needs an update
|
||||
|
||||
**Meaning.** Official evidence changed, a required source could not be
|
||||
checked, or the retained evidence does not match this catalog and parser.
|
||||
`unknown/source-unavailable` is not current. A later source failure must not
|
||||
erase an earlier retirement finding or close its maintenance issue.
|
||||
|
||||
**Fix.** Read the failed source and evidence details in the freshness workflow
|
||||
report and tracking issue. Repair the source parser or publication access when
|
||||
needed, then run the workflow manually against the default branch. If a model
|
||||
was superseded or deprecated, follow the [maintainer checklist](model-policy.md#maintainer-update-checklist)
|
||||
to qualify a replacement or record a supported mitigation. Do not clear the
|
||||
issue just because a new release has not been adopted yet.
|
||||
|
||||
**Expected result.** Complete source checks produce evidence bound to the
|
||||
current catalog, while unresolved lifecycle findings remain visible. Personal
|
||||
pins and unrelated PRs are unchanged. Check the workflow's last run date too:
|
||||
a scheduled job that never runs cannot report its own absence.
|
||||
|
||||
#### Repairing `recoveryRequired` state
|
||||
|
||||
This flag deliberately keeps the issue open until a maintainer repairs the
|
||||
record. Another successful fetch alone cannot certify corrupted history.
|
||||
|
||||
1. Save the current issue body, including its retained recovery archive, and
|
||||
download the latest trustworthy workflow report. Use its `report.state` as
|
||||
the starting state; do not invent run IDs, catalog hashes or check dates.
|
||||
2. Reconcile its `lifecycle` array with the issue's independent lifecycle block
|
||||
and any retained archive. Preserve known retirements, deadlines and human
|
||||
dispositions. A model currently in the catalog must not keep an old
|
||||
`resolvedByCatalog` flag from a previous replacement.
|
||||
3. In the issue editor, restore valid JSON between the `state:begin` and
|
||||
`state:end` marker lines. Put that same lifecycle array between
|
||||
`lifecycle:begin` and `lifecycle:end`. Restore the standalone ownership and
|
||||
region markers, and leave human text outside the owned region intact.
|
||||
4. Set `recoveryRequired` to `false` only after that reconciliation. If prior
|
||||
successful evidence cannot be verified, set `lastSuccess` to `null` rather
|
||||
than making up a fresh success. If lifecycle history cannot be recovered,
|
||||
leave the flag set and the issue open for investigation.
|
||||
5. Run **Model-policy freshness** against the default branch. Only complete,
|
||||
current evidence with no unresolved findings can close the issue. A parsing
|
||||
or source error leaves it open; do not clear history to manufacture a pass.
|
||||
|
||||
## Outside reviews (Codex and Claude Code)
|
||||
|
||||
Outside reviews send your diff, plan or question to a second AI provider. Their
|
||||
|
||||
Reference in new issue
Block a user