mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-15 21:07:20 +01:00
Merge remote-tracking branch 'origin/main' into gsta232-autoplan-wave
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> # Conflicts: # test/fixtures/skill-md-size-allowlist.json
This commit is contained in:
commit
625ccdfa71
117 files changed
+4683
-644
No files matched your search
+64
-14
@@ -1645,9 +1645,11 @@ word list: `test/fixtures/arm-benchmark/` and its four children, `browse-judge/`
|
||||
names under `paid-report-census/` and the `test/` subdirectories of
|
||||
`state-matrix-6792/`. All of these are runner-side: no candidate's working
|
||||
directory is one of them. Candidate-visible gaps are in the temporary roots
|
||||
the harnesses create: `test/helpers/arm-benchmark-harness.ts` copies each fixture
|
||||
into `os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory is
|
||||
named after the arm rather than after a project; `test/skill-e2e-deploy.test.ts`
|
||||
the harnesses create: `test/helpers/arm-benchmark-harness.ts` copied each fixture
|
||||
into `os.tmpdir()/arm-<fixture>-<arm>-*`, so the candidate's working directory was
|
||||
named after the arm rather than after a project (since GST-469 it is named after
|
||||
the task's project: `hike-booking-*`, `notes-api-*`, `receipts-*`);
|
||||
`test/skill-e2e-deploy.test.ts`
|
||||
runs the /benchmark case in `skill-e2e-benchmark-*`, a root that carries a listed
|
||||
word; `test/skill-e2e-test-value.test.ts` creates `skill-e2e-<id>-*` roots as the
|
||||
candidate's working directory, and two of its ids (`review-test-value`,
|
||||
@@ -1692,18 +1694,66 @@ own attempts. The arm gets:
|
||||
|
||||
**Reporting.** The no-gstack result is one row beside the variants in every
|
||||
results table and every verdict, including when it wins. A comparison that
|
||||
drops the row has not reported its result. The prereg check cannot see the
|
||||
report, so the `report:` line states where the row goes and the reviewer of
|
||||
the result holds the author to it.
|
||||
drops the row has not reported its result. `gstack-eval-plan results --check
|
||||
<prereg> <results.md>` enforces it: one table in the results file must hold a
|
||||
filled row for every Arms identity (`EVAL_REPORT_NO_GSTACK_ROW` otherwise), and
|
||||
no number is published before it passes.
|
||||
|
||||
**Precedent and its gaps.** `test/helpers/arm-benchmark-harness.ts` already has
|
||||
a `without-skill` arm beside `with-skill`, and it is the closest existing
|
||||
control. It falls short of this rule in three ways: each arm and task runs
|
||||
once rather than as a panel; the judge (`armJudge` in
|
||||
`test/helpers/llm-judge.ts`) scores each diff on its own, without `A`/`B`
|
||||
labels; and the candidate's temporary folder is named after the arm
|
||||
(`arm-<fixture>-<arm>-*`). Existing paid cases are not retrofitted by the
|
||||
change that added this rule; an audit of them is tracked separately.
|
||||
**Safety-rule and router evals: a descriptive third arm.** The W1 safety-rule
|
||||
evals compare `GSTACK_SAFETY_ARM=rule` (the shipped rule) with `removed` (the
|
||||
rule paragraph deleted), and the router failure-routing evals compare
|
||||
`GSTACK_ROUTER_ARM=current` with `before` (the base branch's router). Both
|
||||
variables also take `none` (GST-470): the same fixture, tools and user request,
|
||||
with no skill excerpt, rule text, step file or installed skill. Any other value
|
||||
throws, so a typo cannot run one arm under another's label. The `rule`,
|
||||
`removed`, `current` and `before` prompts are byte for byte what they were
|
||||
before `none` existed (`test/safety-none-arm.test.ts`,
|
||||
`test/failure-routing-arm.test.ts`). To run the three arms as a comparison, run
|
||||
the same case once per arm, each as its CI panel:
|
||||
|
||||
```bash
|
||||
for arm in rule removed none; do
|
||||
GSTACK_SAFETY_ARM=$arm bun run scripts/test-paid-shards.ts --tier gate --case safety-ship-stale-evidence
|
||||
done
|
||||
for arm in current before none; do
|
||||
GSTACK_ROUTER_ARM=$arm bun run scripts/test-paid-shards.ts --tier gate --case router-failure-contradicted-cause
|
||||
done
|
||||
```
|
||||
|
||||
The arm variable is part of the shard-reuse environment, so one arm's result is
|
||||
never reused for another. Report the three rows side by side in one results
|
||||
table and pass `gstack-eval-plan results --check` on it.
|
||||
|
||||
**The descriptive exception.** Under `none` every behavioral check, the plain
|
||||
`expect`s on model actions included, is recorded as an observation in
|
||||
`none-arm-observations.jsonl` in the eval directory and on stderr
|
||||
(`test/helpers/none-arm.ts`). It never fails the case and never writes
|
||||
`contract-violations.jsonl`; harness preconditions (the fixture built, the CLI
|
||||
ran, the session ended normally) still assert. This departs from "the same
|
||||
panel" above because these contracts are gstack's own rules: a plain model was
|
||||
never asked to keep them, so a failing `none` trial is the measurement, not a
|
||||
regression, and a red verdict would block the gate on behavior nobody
|
||||
specified. The router's `none` arm has no "routed to /investigate" metric,
|
||||
since no skill is installed to route to; it records plain metrics instead:
|
||||
whether the suite is green, whether an edit came before the first read, and
|
||||
how many read-only calls came before the first edit. The first paid comparison
|
||||
with these arms gets its own preregistration.
|
||||
|
||||
**Cases without a `none` arm.** `safety-pair-agent-block` and
|
||||
`safety-ios-demo-ui-only` refuse `GSTACK_SAFETY_ARM=none` with the reason
|
||||
(`NO_NONE_ARM` in `test/helpers/safety-rules.ts`). Their task exists only in
|
||||
the skill: relaying /pair-agent's remote instruction block, and /ios-qa's demo
|
||||
mode with its device routes. A plain prompt has nothing to relay and no device
|
||||
interface to drive, so the arm would measure the prompt, not gstack.
|
||||
|
||||
**Precedent.** `test/skill-e2e-arm-benchmark.test.ts` runs a `with-skill` and a
|
||||
`without-skill` arm (named `<model>#no-gstack` in its preregistration) as 3
|
||||
pairs per task under a spend ledger and cap. A paired judge reads both diffs as
|
||||
`A` and `B` in both orders, counts a win only when the orders agree, and
|
||||
decides only after it passes calibration; the per-diff `armJudge` stays beside
|
||||
it as the diagnostic. Candidate folders and git identity name a project, never
|
||||
the arm. The model panel (`test/skill-e2e-model-panel.test.ts`) runs a
|
||||
`<model>#no-gstack` arm beside each model (GST-468, GST-469).
|
||||
|
||||
## Running evals as an agent: detach
|
||||
|
||||
|
||||
+19
-1
@@ -178,7 +178,11 @@ Two stages with separate clocks, both printed:
|
||||
simply not proven.
|
||||
- `secrets`: `gstack-redact` over the branch's added lines (merge-base to the
|
||||
working tree, untracked files included). HIGH, or MEDIUM in the `secret`
|
||||
category, fails; other MEDIUM findings warn. Cannot be downgraded.
|
||||
category, fails; other MEDIUM findings warn. Cannot be downgraded. A
|
||||
changed Python (`.py`, `.pyi`) or Dart file is scanned whole and only
|
||||
findings on its added lines count, so a line added inside an existing
|
||||
`"""` or `'''` string is still read as a literal; its rows give the file's
|
||||
line number (`app.py, line 10`).
|
||||
- `strays`: a new file at the repository root, a new top-level directory, or a
|
||||
scratch name (`zz-*`, `tmp*`, `scratch*`) anywhere.
|
||||
- `literals`: for every changed exported constant or named timeout/retry
|
||||
@@ -802,6 +806,20 @@ those, each with its scope and its reason:
|
||||
the entry is inert for that finding and the row says so (`allowlist[0]
|
||||
env.kv not tested on env.kv (<file>): the scanner gave no position for its
|
||||
pattern; the finding is kept`) rather than silently keeping it.
|
||||
- The position is where the flagged value starts, not the name before it.
|
||||
For `session: HttpSession` in a YAML file the pattern is `HttpSession`;
|
||||
`session: HttpSession` never matches there. An entry that names a finding
|
||||
by id and path, and whose pattern matches the line but not at the value,
|
||||
prints `allowlist[0] env.kv pattern did not match env.kv (<file>, added
|
||||
line N, col C): …; the finding is kept`.
|
||||
- In Python and Dart source, a parameter or field annotation
|
||||
(`session: HttpSession,`) and a dict entry or Dart named argument that reads
|
||||
a name (`"password": request.password`) are code, not credentials, as they
|
||||
are in TypeScript (#3630). Python's keyword form (`password=request.password`)
|
||||
still reports. The same line inside a `"""` docstring is text and still
|
||||
reports; allowlist it by its value. The pre-push hook sees only added lines,
|
||||
so it scans Python and Dart lines without this exemption and may still warn
|
||||
(MEDIUM, non-blocking) on them.
|
||||
|
||||
The `secrets` row says `N allowlisted (<ids>)` and lists each allowlisted
|
||||
finding with the entry (`allowlisted env.kv (evals/receipts/t1.json, added line
|
||||
|
||||
+16
-14
@@ -34,21 +34,23 @@ needed, because every skill reads `PROACTIVE` from `gstack-skill-start` at the
|
||||
start of each run and the router (`/gstack`) applies it to its routing rules. A
|
||||
value gstack does not know (a hand-edited `proactive: maybe`) reads as `core`.
|
||||
|
||||
Two surfaces follow the setting only in part. The first is the `## Skill routing`
|
||||
section gstack offers to append to your project's CLAUDE.md. Under `off` the offer is
|
||||
not made, because those rules are the auto-invocation you turned off (a section
|
||||
already in CLAUDE.md stays; delete it yourself). Under `core` the offered
|
||||
section still lists every routing rule and tells the host to invoke any
|
||||
matching skill. If you accepted it before the `core` default, trim that section
|
||||
to the four core rules (or delete it and let gstack offer it again); making the
|
||||
offered section follow `core` is tracked in #3446.
|
||||
The `## Skill routing` section gstack offers to append to your project's
|
||||
CLAUDE.md follows the setting too. Under `core` it lists only the four core
|
||||
rules and says the other skills run when you type them or a running skill
|
||||
hands off to them; under `all` it lists
|
||||
every routing rule; under `off` it is not offered, because those rules are the
|
||||
auto-invocation you turned off. gstack offers the section once per project and
|
||||
never edits it afterwards. So a section you accepted earlier, or under another
|
||||
setting, stays as it is: trim it to match (the core four are /office-hours,
|
||||
/review, /qa and /ship), or delete it and let gstack offer it again.
|
||||
|
||||
The second is the full skill descriptions, which a `--catalog-mode=full` build
|
||||
and the non-Claude hosts keep. Non-core skills still say "Proactively suggest
|
||||
when ..." there, and that text does not change with the setting, so a host
|
||||
that picks skills from those descriptions can still reach them under `core`.
|
||||
The default Claude build uses one-line descriptions without that sentence.
|
||||
#3446 tracks that too.
|
||||
Skill descriptions cannot change with the setting, so they are written for the
|
||||
`core` default. Only the four core skills say "Proactively suggest ..." in the
|
||||
full descriptions that a `--catalog-mode=full` build and the non-Claude hosts
|
||||
keep. The other skills keep their "Use when asked to ..." sentences, which
|
||||
describe asks. On those hosts, `all` therefore loses the proactive hint the
|
||||
non-core descriptions used to carry; the router's routing rules still apply when
|
||||
`/gstack` runs.
|
||||
|
||||
gstack asks once, after the telemetry question, which of the three you want;
|
||||
if you stored `off` before that, it takes the answer as given and does not ask.
|
||||
|
||||
@@ -271,8 +271,9 @@ the specialists, audits and watchers at the host default too
|
||||
### Behavior changes
|
||||
|
||||
- Pass 2 after a blocking fix on a diff below full depth, with a fix range of at
|
||||
most 150 runtime lines, is a [scoped pass](#scoped-pass): a fix-check, the Red
|
||||
Team and the native adversarial pass on the range instead of the specialist army.
|
||||
most 150 runtime lines, is a [scoped pass](#scoped-pass): a fix-check and the Red
|
||||
Team on the range, plus the native adversarial pass (on the range once Step 11 has
|
||||
run, otherwise on the whole branch), instead of the specialist army.
|
||||
- A tests-only diff (runtime 0) no longer dispatches the Testing specialist: the
|
||||
core checklist and the gate scan review it.
|
||||
- Every depth dispatches one batched specialist instead of one reader per
|
||||
|
||||
+3
-3
@@ -352,7 +352,7 @@ warning on stderr). A repository without gstack's eval workflow declares
|
||||
"train": {
|
||||
"gate_workflow": "evals.yml", "free_workflow": "free-tests.yml",
|
||||
"full_gate_check": "gstack/full-gate", "record_artifact": "gstack-train-record",
|
||||
"max_size": 3, "fill_hours": 1, "screen_selection": "gstack-paid",
|
||||
"max_size": 5, "fill_hours": 1, "screen_selection": "gstack-paid",
|
||||
"trial_reservation_usd": 1.5
|
||||
},
|
||||
"min_gstack_version": "<the version that introduced train>"
|
||||
@@ -680,8 +680,8 @@ forked (the chain's merge onto the pin's own prefix is the verdict), refuses
|
||||
`QUEUE_TRAIN_CONFLICT` for a pin that does not merge, and otherwise prints
|
||||
`MERGE_CHECK: T1..T<k> merge clean onto <base>@<sha>`, so a
|
||||
`QUEUE_TRAIN_WAITING` result is also a clean dry-run merge of the pool (the
|
||||
crew template's sibling trial merge, done by the tool). gstack's own block fills at 3 PRs or one hour (it was 5 PRs or
|
||||
three hours until 2026-10-26, which left READY PRs waiting most of a day). The same pins on the same base return the live train
|
||||
crew template's sibling trial merge, done by the tool). gstack's own block fills at 5 PRs or one hour (it was 5 PRs or
|
||||
three hours until 2026-10-26, which left READY PRs waiting most of a day, then 3 PRs or one hour until 2026-10-31). The same pins on the same base return the live train
|
||||
instead of a second one; `train build --train <nonce>` resumes a train from
|
||||
its record. Refusals, all before any spend: `QUEUE_TRAIN_UNTRUSTED` (a fork
|
||||
head), `QUEUE_TRAIN_MOVED` (a head moved from its pin),
|
||||
|
||||
@@ -964,6 +964,9 @@ runs every row through the stub model for $0 and flags malformed rows
|
||||
(`EVAL_DRY_RUN_FAILED`); `pilot start|settle|release|report` runs three priced
|
||||
items, records actual spend, extrapolates to the full set against the cap
|
||||
(`EVAL_BUDGET_EXHAUSTED`) and prints the spend table the approval question shows.
|
||||
`results --check <prereg> <results.md>` refuses a finished results file unless one
|
||||
table holds a filled row for every arm, each `<model>#no-gstack` row beside its
|
||||
variants (`EVAL_REPORT_NO_GSTACK_ROW`); no number leaves the run before it passes.
|
||||
The skill's closing section lists the tier-3 items (instrument audit, steady
|
||||
state, sealed paths, off-machine persistence, memory preflight, power tables, QA
|
||||
soak) as `tier 3, after P7`; none of them run yet.
|
||||
|
||||
@@ -1121,6 +1121,28 @@ conflict yourself instead, finish or abort your merge (`git -C '<dir>' merge
|
||||
--abort`, `git -C '<dir>' checkout -- <file>`); gstack retries at the next
|
||||
check.
|
||||
|
||||
<a id="auto-update-tracked-by-project"></a>
|
||||
### `gstack auto-update: update held (tracked-by-project: ...)` / `(version-reverted: ...)` / `gstack-upgrade: tracked-by-project: ... (UPGRADE_REFUSED)` / `gstack-upgrade: version-reverted: ... (UPGRADE_REFUSED)`
|
||||
|
||||
**Meaning.** Your gstack install sits inside another git project, and that
|
||||
project has committed gstack's files as ordinary files. Every time you switch
|
||||
branches or check out files in that project, git puts its old copies of
|
||||
gstack's files back, so VERSION and other files fall behind the install's own
|
||||
HEAD while updates seem to succeed (#1022). `version-reverted` is the same
|
||||
symptom seen from the install: its VERSION disagrees with HEAD. Neither
|
||||
`gstack-upgrade`, `/gstack-upgrade` nor the auto-updater moves the install
|
||||
until it is fixed. A project that holds gstack as a git submodule is fine.
|
||||
`version-reverted` with no enclosing project means VERSION was edited by
|
||||
hand; only the `stash push` step below applies.
|
||||
|
||||
**Fix.** Print the exact commands with
|
||||
`bash <install>/bin/gstack-ff-update --preflight <install>`. They do three
|
||||
things: untrack the install in the project (`git rm -r --cached`), add it to
|
||||
the project's `.gitignore`, and commit both on every branch you check out
|
||||
(or vendor gstack as a submodule instead). Then
|
||||
`git -C <install> stash push` puts the install's own files back and keeps
|
||||
any edit you made in the stash, and the next update proceeds.
|
||||
|
||||
<a id="auto-update-autostash-unrecoverable"></a>
|
||||
### `gstack auto-update: update held (autostash-unrecoverable: <reason>); your local edits are where they were and nothing was installed. Look first: git -C '<dir>' status --short. Fix now (...)` / `autostash-unrecoverable: <reason>`
|
||||
|
||||
@@ -1563,6 +1585,8 @@ directory is not a gstack checkout, HEAD is detached or on another branch (a
|
||||
pin), a tracked file is modified, a local commit is not on the remote branch,
|
||||
the incoming release's Bun floor is above the Bun on PATH, or another updater
|
||||
holds `<state root>/.setup-lock`. It never stashes, resets or discards.
|
||||
A `tracked-by-project` or `version-reverted` cause has its own section:
|
||||
[an enclosing project tracks the install](#auto-update-tracked-by-project).
|
||||
|
||||
**Fix.** Resolve the named cause with the printed command (commit or move the
|
||||
listed changes, check out the default branch, move local commits to another
|
||||
@@ -3697,6 +3721,21 @@ goes beside the variants. When no arm loads a skill stack, write
|
||||
`stack: none` with a `why:` line instead. The rule is in
|
||||
[docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm).
|
||||
|
||||
<a id="eval-report-no-gstack-row"></a>
|
||||
### `EVAL_REPORT_NO_GSTACK_ROW`
|
||||
|
||||
**Meaning.** `gstack-eval-plan results --check <prereg> <results>` found no
|
||||
single markdown table in the results file with a filled row for every arm
|
||||
identity the preregistration's Arms section lists. Usually the
|
||||
`<model>#no-gstack` row is missing, sits in a separate table from its
|
||||
variants, or holds only placeholders (`-`, `n/a`, `tbd`). It also fires when
|
||||
the preregistration itself fails `prereg --check`.
|
||||
|
||||
**Fix.** Put one row per arm identity, each no-gstack arm beside its variants,
|
||||
in one table, and rerun the check. Publish no number before it prints
|
||||
`RESULTS: ok`. A preregistration with `stack: none` needs no row. The rule is
|
||||
in [docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm).
|
||||
|
||||
<a id="eval-arm-duplicate"></a>
|
||||
### `EVAL_ARM_DUPLICATE`
|
||||
|
||||
|
||||
+6
-1
@@ -11,4 +11,9 @@ on.
|
||||
The separate `triggers:` list in a skill's frontmatter is gstack metadata, not a
|
||||
matching surface. Claude Code and Codex choose a skill from its name and
|
||||
description, so a phrase that appears only under `triggers:` does not activate
|
||||
the skill (Codex output drops the field; the GBrain host keeps it).
|
||||
the skill (Codex output drops the field; the GBrain host keeps it). The four core
|
||||
skills (/office-hours, /review, /qa, /ship) carry their `triggers:` phrases in
|
||||
the description itself: every phrase in the full description, and at least one
|
||||
in the one-line catalog entry Claude Code reads ("test the site" for /qa, "ship
|
||||
it" for /ship). The other skills do not, because under the `core` default they
|
||||
wait for you to type them (docs/proactive.md).
|
||||
Reference in new issue
Block a user