Merge remote-tracking branch 'origin/main' into gsta232-autoplan-wave

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

# Conflicts:
#	test/fixtures/skill-md-size-allowlist.json
This commit is contained in:
garrytan committed 2026-11-01 03:11:05 +00:00
commit 625ccdfa71
117 files changed
+4683 -644

No files matched your search

+64 -14
View File
@@ -1645,9 +1645,11 @@ word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/`
names under `paid-report-census/` and the `test/` subdirectories of
`state-matrix-6792/`. All of these are runner-side: no candidate's working
directory is one of them. Candidate-visible gaps are in the temporary roots
the harnesses create: `test/helpers/arm-benchmark-harness.ts` copies each fixture
into `os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory is
named after the arm rather than after a project; `test/skill-e2e-deploy.test.ts`
the harnesses create: `test/helpers/arm-benchmark-harness.ts` copied each fixture
into `os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory was
named after the arm rather than after a project (since GST-469 it is named after
the task's project: `hike-booking-*`, `notes-api-*`, `receipts-*`);
`test/skill-e2e-deploy.test.ts`
runs the /benchmark case in `skill-e2e-benchmark-*`, a root that carries a listed
word; `test/skill-e2e-test-value.test.ts` creates `skill-e2e-<id>-*` roots as the
candidate's working directory, and two of its ids (`review-test-value`,
@@ -1692,18 +1694,66 @@ own attempts. The arm gets:
**Reporting.** The no-gstack result is one row beside the variants in every
results table and every verdict, including when it wins. A comparison that
drops the row has not reported its result. The prereg check cannot see the
report, so the `report:` line states where the row goes and the reviewer of
the result holds the author to it.
drops the row has not reported its result. `gstack-eval-plan results --check
<prereg> <results.md>` enforces it: one table in the results file must hold a
filled row for every Arms identity (`EVAL_REPORT_NO_GSTACK_ROW` otherwise), and
no number is published before it passes.
**Precedent and its gaps.** `test/helpers/arm-benchmark-harness.ts` already has
a `without-skill` arm beside `with-skill`, and it is the closest existing
control. It falls short of this rule in three ways: each arm and task runs
once rather than as a panel; the judge (`armJudge` in
`test/helpers/llm-judge.ts`) scores each diff on its own, without `A`/`B`
labels; and the candidate's temporary folder is named after the arm
(`arm-<fixture>-<arm>-*`). Existing paid cases are not retrofitted by the
change that added this rule; an audit of them is tracked separately.
**Safety-rule and router evals: a descriptive third arm.** The W1 safety-rule
evals compare `GSTACK_SAFETY_ARM=rule` (the shipped rule) with `removed` (the
rule paragraph deleted), and the router failure-routing evals compare
`GSTACK_ROUTER_ARM=current` with `before` (the base branch's router). Both
variables also take `none` (GST-470): the same fixture, tools and user request,
with no skill excerpt, rule text, step file or installed skill. Any other value
throws, so a typo cannot run one arm under another's label. The `rule`,
`removed`, `current` and `before` prompts are byte for byte what they were
before `none` existed (`test/safety-none-arm.test.ts`,
`test/failure-routing-arm.test.ts`). To run the three arms as a comparison, run
the same case once per arm, each as its CI panel:
```bash
for arm in rule removed none; do
GSTACK_SAFETY_ARM=$arm bun run scripts/test-paid-shards.ts --tier gate --case safety-ship-stale-evidence
done
for arm in current before none; do
GSTACK_ROUTER_ARM=$arm bun run scripts/test-paid-shards.ts --tier gate --case router-failure-contradicted-cause
done
```
The arm variable is part of the shard-reuse environment, so one arm's result is
never reused for another. Report the three rows side by side in one results
table and pass `gstack-eval-plan results --check` on it.
**The descriptive exception.** Under `none` every behavioral check, the plain
`expect`s on model actions included, is recorded as an observation in
`none-arm-observations.jsonl` in the eval directory and on stderr
(`test/helpers/none-arm.ts`). It never fails the case and never writes
`contract-violations.jsonl`; harness preconditions (the fixture built, the CLI
ran, the session ended normally) still assert. This departs from "the same
panel" above because these contracts are gstack's own rules: a plain model was
never asked to keep them, so a failing `none` trial is the measurement, not a
regression, and a red verdict would block the gate on behavior nobody
specified. The router's `none` arm has no "routed to /investigate" metric,
since no skill is installed to route to; it records plain metrics instead:
whether the suite is green, whether an edit came before the first read, and
how many read-only calls came before the first edit. The first paid comparison
with these arms gets its own preregistration.
**Cases without a `none` arm.** `safety-pair-agent-block` and
`safety-ios-demo-ui-only` refuse `GSTACK_SAFETY_ARM=none` with the reason
(`NO_NONE_ARM` in `test/helpers/safety-rules.ts`). Their task exists only in
the skill: relaying /pair-agent's remote instruction block, and /ios-qa's demo
mode with its device routes. A plain prompt has nothing to relay and no device
interface to drive, so the arm would measure the prompt, not gstack.
**Precedent.** `test/skill-e2e-arm-benchmark.test.ts` runs a `with-skill` and a
`without-skill` arm (named `<model>#no-gstack` in its preregistration) as 3
pairs per task under a spend ledger and cap. A paired judge reads both diffs as
`A` and `B` in both orders, counts a win only when the orders agree, and
decides only after it passes calibration; the per-diff `armJudge` stays beside
it as the diagnostic. Candidate folders and git identity name a project, never
the arm. The model panel (`test/skill-e2e-model-panel.test.ts`) runs a
`<model>#no-gstack` arm beside each model (GST-468, GST-469).
## Running evals as an agent: detach
+19 -1
View File
@@ -178,7 +178,11 @@ Two stages with separate clocks, both printed:
simply not proven.
- `secrets`: `gstack-redact` over the branch's added lines (merge-base to the
working tree, untracked files included). HIGH, or MEDIUM in the `secret`
category, fails; other MEDIUM findings warn. Cannot be downgraded.
category, fails; other MEDIUM findings warn. Cannot be downgraded. A
changed Python (`.py`, `.pyi`) or Dart file is scanned whole and only
findings on its added lines count, so a line added inside an existing
`"""` or `'''` string is still read as a literal; its rows give the file's
line number (`app.py, line 10`).
- `strays`: a new file at the repository root, a new top-level directory, or a
scratch name (`zz-*`, `tmp*`, `scratch*`) anywhere.
- `literals`: for every changed exported constant or named timeout/retry
@@ -802,6 +806,20 @@ those, each with its scope and its reason:
the entry is inert for that finding and the row says so (`allowlist[0]
env.kv not tested on env.kv (<file>): the scanner gave no position for its
pattern; the finding is kept`) rather than silently keeping it.
- The position is where the flagged value starts, not the name before it.
For `session: HttpSession` in a YAML file the pattern is `HttpSession`;
`session: HttpSession` never matches there. An entry that names a finding
by id and path, and whose pattern matches the line but not at the value,
prints `allowlist[0] env.kv pattern did not match env.kv (<file>, added
line N, col C): …; the finding is kept`.
- In Python and Dart source, a parameter or field annotation
(`session: HttpSession,`) and a dict entry or Dart named argument that reads
a name (`"password": request.password`) are code, not credentials, as they
are in TypeScript (#3630). Python's keyword form (`password=request.password`)
still reports. The same line inside a `"""` docstring is text and still
reports; allowlist it by its value. The pre-push hook sees only added lines,
so it scans Python and Dart lines without this exemption and may still warn
(MEDIUM, non-blocking) on them.
The `secrets` row says `N allowlisted (<ids>)` and lists each allowlisted
finding with the entry (`allowlisted env.kv (evals/receipts/t1.json, added line
+16 -14
View File
@@ -34,21 +34,23 @@ needed, because every skill reads `PROACTIVE` from `gstack-skill-start` at the
start of each run and the router (`/gstack`) applies it to its routing rules. A
value gstack does not know (a hand-edited `proactive: maybe`) reads as `core`.
Two surfaces follow the setting only in part. The first is the `## Skill routing`
section gstack offers to append to your project's CLAUDE.md. Under `off` the offer is
not made, because those rules are the auto-invocation you turned off (a section
already in CLAUDE.md stays; delete it yourself). Under `core` the offered
section still lists every routing rule and tells the host to invoke any
matching skill. If you accepted it before the `core` default, trim that section
to the four core rules (or delete it and let gstack offer it again); making the
offered section follow `core` is tracked in #3446.
The `## Skill routing` section gstack offers to append to your project's
CLAUDE.md follows the setting too. Under `core` it lists only the four core
rules and says the other skills run when you type them or a running skill
hands off to them; under `all` it lists
every routing rule; under `off` it is not offered, because those rules are the
auto-invocation you turned off. gstack offers the section once per project and
never edits it afterwards. So a section you accepted earlier, or under another
setting, stays as it is: trim it to match (the core four are /office-hours,
/review, /qa and /ship), or delete it and let gstack offer it again.
The second is the full skill descriptions, which a `--catalog-mode=full` build
and the non-Claude hosts keep. Non-core skills still say "Proactively suggest
when ..." there, and that text does not change with the setting, so a host
that picks skills from those descriptions can still reach them under `core`.
The default Claude build uses one-line descriptions without that sentence.
#3446 tracks that too.
Skill descriptions cannot change with the setting, so they are written for the
`core` default. Only the four core skills say "Proactively suggest ..." in the
full descriptions that a `--catalog-mode=full` build and the non-Claude hosts
keep. The other skills keep their "Use when asked to ..." sentences, which
describe asks. On those hosts, `all` therefore loses the proactive hint the
non-core descriptions used to carry; the router's routing rules still apply when
`/gstack` runs.
gstack asks once, after the telemetry question, which of the three you want;
if you stored `off` before that, it takes the answer as given and does not ask.
+3 -2
View File
@@ -271,8 +271,9 @@ the specialists, audits and watchers at the host default too
### Behavior changes
- Pass 2 after a blocking fix on a diff below full depth, with a fix range of at
most 150 runtime lines, is a [scoped pass](#scoped-pass): a fix-check, the Red
Team and the native adversarial pass on the range instead of the specialist army.
most 150 runtime lines, is a [scoped pass](#scoped-pass): a fix-check and the Red
Team on the range, plus the native adversarial pass (on the range once Step 11 has
run, otherwise on the whole branch), instead of the specialist army.
- A tests-only diff (runtime 0) no longer dispatches the Testing specialist: the
core checklist and the gate scan review it.
- Every depth dispatches one batched specialist instead of one reader per
+3 -3
View File
@@ -352,7 +352,7 @@ warning on stderr). A repository without gstack's eval workflow declares
"train": {
"gate_workflow": "evals.yml", "free_workflow": "free-tests.yml",
"full_gate_check": "gstack/full-gate", "record_artifact": "gstack-train-record",
"max_size": 3, "fill_hours": 1, "screen_selection": "gstack-paid",
"max_size": 5, "fill_hours": 1, "screen_selection": "gstack-paid",
"trial_reservation_usd": 1.5
},
"min_gstack_version": "<the version that introduced train>"
@@ -680,8 +680,8 @@ forked (the chain's merge onto the pin's own prefix is the verdict), refuses
`QUEUE_TRAIN_CONFLICT` for a pin that does not merge, and otherwise prints
`MERGE_CHECK: T1..T<k> merge clean onto <base>@<sha>`, so a
`QUEUE_TRAIN_WAITING` result is also a clean dry-run merge of the pool (the
crew template's sibling trial merge, done by the tool). gstack's own block fills at 3 PRs or one hour (it was 5 PRs or
three hours until 2026-10-26, which left READY PRs waiting most of a day). The same pins on the same base return the live train
crew template's sibling trial merge, done by the tool). gstack's own block fills at 5 PRs or one hour (it was 5 PRs or
three hours until 2026-10-26, which left READY PRs waiting most of a day, then 3 PRs or one hour until 2026-10-31). The same pins on the same base return the live train
instead of a second one; `train build --train <nonce>` resumes a train from
its record. Refusals, all before any spend: `QUEUE_TRAIN_UNTRUSTED` (a fork
head), `QUEUE_TRAIN_MOVED` (a head moved from its pin),
+3
View File
@@ -964,6 +964,9 @@ runs every row through the stub model for $0 and flags malformed rows
(`EVAL_DRY_RUN_FAILED`); `pilot start|settle|release|report` runs three priced
items, records actual spend, extrapolates to the full set against the cap
(`EVAL_BUDGET_EXHAUSTED`) and prints the spend table the approval question shows.
`results --check <prereg> <results.md>` refuses a finished results file unless one
table holds a filled row for every arm, each `<model>#no-gstack` row beside its
variants (`EVAL_REPORT_NO_GSTACK_ROW`); no number leaves the run before it passes.
The skill's closing section lists the tier-3 items (instrument audit, steady
state, sealed paths, off-machine persistence, memory preflight, power tables, QA
soak) as `tier 3, after P7`; none of them run yet.
+39
View File
@@ -1121,6 +1121,28 @@ conflict yourself instead, finish or abort your merge (`git -C '<dir>' merge
--abort`, `git -C '<dir>' checkout -- <file>`); gstack retries at the next
check.
<a id="auto-update-tracked-by-project"></a>
### `gstack auto-update: update held (tracked-by-project: ...)` / `(version-reverted: ...)` / `gstack-upgrade: tracked-by-project: ... (UPGRADE_REFUSED)` / `gstack-upgrade: version-reverted: ... (UPGRADE_REFUSED)`
**Meaning.** Your gstack install sits inside another git project, and that
project has committed gstack's files as ordinary files. Every time you switch
branches or check out files in that project, git puts its old copies of
gstack's files back, so VERSION and other files fall behind the install's own
HEAD while updates seem to succeed (#1022). `version-reverted` is the same
symptom seen from the install: its VERSION disagrees with HEAD. Neither
`gstack-upgrade`, `/gstack-upgrade` nor the auto-updater moves the install
until it is fixed. A project that holds gstack as a git submodule is fine.
`version-reverted` with no enclosing project means VERSION was edited by
hand; only the `stash push` step below applies.
**Fix.** Print the exact commands with
`bash <install>/bin/gstack-ff-update --preflight <install>`. They do three
things: untrack the install in the project (`git rm -r --cached`), add it to
the project's `.gitignore`, and commit both on every branch you check out
(or vendor gstack as a submodule instead). Then
`git -C <install> stash push` puts the install's own files back and keeps
any edit you made in the stash, and the next update proceeds.
<a id="auto-update-autostash-unrecoverable"></a>
### `gstack auto-update: update held (autostash-unrecoverable: <reason>); your local edits are where they were and nothing was installed. Look first: git -C '<dir>' status --short. Fix now (...)` / `autostash-unrecoverable: <reason>`
@@ -1563,6 +1585,8 @@ directory is not a gstack checkout, HEAD is detached or on another branch (a
pin), a tracked file is modified, a local commit is not on the remote branch,
the incoming release's Bun floor is above the Bun on PATH, or another updater
holds `<state root>/.setup-lock`. It never stashes, resets or discards.
A `tracked-by-project` or `version-reverted` cause has its own section:
[an enclosing project tracks the install](#auto-update-tracked-by-project).
**Fix.** Resolve the named cause with the printed command (commit or move the
listed changes, check out the default branch, move local commits to another
@@ -3697,6 +3721,21 @@ goes beside the variants. When no arm loads a skill stack, write
`stack: none` with a `why:` line instead. The rule is in
[docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm).
<a id="eval-report-no-gstack-row"></a>
### `EVAL_REPORT_NO_GSTACK_ROW`
**Meaning.** `gstack-eval-plan results --check <prereg> <results>` found no
single markdown table in the results file with a filled row for every arm
identity the preregistration's Arms section lists. Usually the
`<model>#no-gstack` row is missing, sits in a separate table from its
variants, or holds only placeholders (`-`, `n/a`, `tbd`). It also fires when
the preregistration itself fails `prereg --check`.
**Fix.** Put one row per arm identity, each no-gstack arm beside its variants,
in one table, and rerun the check. Publish no number before it prints
`RESULTS: ok`. A preregistration with `stack: none` needs no row. The rule is
in [docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm).
<a id="eval-arm-duplicate"></a>
### `EVAL_ARM_DUPLICATE`
+6 -1
View File
@@ -11,4 +11,9 @@ on.
The separate `triggers:` list in a skill's frontmatter is gstack metadata, not a
matching surface. Claude Code and Codex choose a skill from its name and
description, so a phrase that appears only under `triggers:` does not activate
the skill (Codex output drops the field; the GBrain host keeps it).
the skill (Codex output drops the field; the GBrain host keeps it). The four core
skills (/office-hours, /review, /qa, /ship) carry their `triggers:` phrases in
the description itself: every phrase in the full description, and at least one
in the one-line catalog entry Claude Code reads ("test the site" for /qa, "ship
it" for /ship). The other skills do not, because under the `core` default they
wait for you to type them (docs/proactive.md).