mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-18 14:14:37 +01:00
Stop giving bug reports a special rule in gstack's router (GST-2873) (#4442)
gstack's router no longer gives bug reports a special check-the-cause, fix-it-test-first rule, and its description no longer lists debugging, so the model fixes a bug the way it would without gstack. This keeps gstack to the parts that measurably help, and on 26 hard real bugs the rule gave no clear gain. Nothing to do. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> Review-Finding: p1/red-team#1 reader=red-team
This commit is contained in:
1 parent
b79843c5d7
commit
f000b6e0e3
19 files changed
+25
-368
No files matched your search
@@ -2012,25 +2012,19 @@ appears only when it is part of the lesson, and datasets or tools are opened
|
||||
only with a stated reason. The rules and their honesty bars are in
|
||||
[ETHOS.md, Explain the Number](../ETHOS.md#8-explain-the-number).
|
||||
|
||||
**Safety-rule and router evals: a descriptive third arm.** The W1 safety-rule
|
||||
**Safety-rule evals: a descriptive third arm.** The W1 safety-rule
|
||||
evals compare `GSTACK_SAFETY_ARM=rule` (the shipped rule) with `removed` (the
|
||||
rule paragraph deleted), and the router failure-routing evals compare
|
||||
`GSTACK_ROUTER_ARM=current` with `before` (the base branch's router). Both
|
||||
variables also take `none` (GST-470): the same fixture, tools and user request,
|
||||
with no skill excerpt, rule text, step file or installed skill. Any other value
|
||||
throws, so a typo cannot run one arm under another's label. The `rule`,
|
||||
`removed`, `current` and `before` prompts are byte for byte what they were
|
||||
before `none` existed (`test/safety-none-arm.test.ts`,
|
||||
`test/failure-routing-arm.test.ts`). To run the three arms as a comparison, run
|
||||
the same case once per arm, each as its CI panel:
|
||||
rule paragraph deleted). The variable also takes `none` (GST-470): the same
|
||||
fixture, tools and user request, with no skill excerpt, rule text, step file or
|
||||
installed skill. Any other value throws, so a typo cannot run one arm under
|
||||
another's label. The `rule` and `removed` prompts are byte for byte what they
|
||||
were before `none` existed (`test/safety-none-arm.test.ts`). To run the three
|
||||
arms as a comparison, run the same case once per arm, each as its CI panel:
|
||||
|
||||
```bash
|
||||
for arm in rule removed none; do
|
||||
GSTACK_SAFETY_ARM=$arm bun run scripts/test-paid-shards.ts --tier gate --case safety-ship-stale-evidence
|
||||
done
|
||||
for arm in current before none; do
|
||||
GSTACK_ROUTER_ARM=$arm bun run scripts/test-paid-shards.ts --tier gate --case router-failure-verified-cause
|
||||
done
|
||||
```
|
||||
|
||||
The arm variable is part of the shard-reuse environment, so one arm's result is
|
||||
@@ -2046,11 +2040,8 @@ ran, the session ended normally) still assert. This departs from "the same
|
||||
panel" above because these contracts are gstack's own rules: a plain model was
|
||||
never asked to keep them, so a failing `none` trial is the measurement, not a
|
||||
regression, and a red verdict would block the gate on behavior nobody
|
||||
specified. The router's `none` arm has no repro-first metric,
|
||||
since it never sees the router's rules; it records plain metrics instead:
|
||||
whether the suite is green, whether an edit came before the first read, and
|
||||
how many read-only calls came before the first edit. The first paid comparison
|
||||
with these arms gets its own preregistration.
|
||||
specified. The first paid comparison with these arms gets its own
|
||||
preregistration.
|
||||
|
||||
**Cases without a `none` arm.** `safety-pair-agent-block` and
|
||||
`safety-ios-demo-ui-only` refuse `GSTACK_SAFETY_ARM=none` with the reason
|
||||
|
||||
+1
-1
@@ -660,7 +660,7 @@ I want the model imagining the production incident before it happens.
|
||||
|
||||
`/investigate` was cut on 2026-11-07 (GST-1717) after a preregistered eval found no gain. On 18 real bug-fix tasks that plain Opus 5.5 solves about half the time, 20 trials each, Opus told to use /investigate passed 50.0% of trials against 47.2% with no skill: +2.8 points (95% CI -5.8 to +11.4), below the 10-point lift the run could detect. A passing fix cost about 43% more with the skill. The evidence is GEVA-380's R3c result, [gstack-evals c90ab277, wiki/board/2026-11-07-geva-380-r3c-results.md](https://github.com/garrytan/gstack-evals/blob/c90ab277/wiki/board/2026-11-07-geva-380-r3c-results.md).
|
||||
|
||||
The skill no longer installs, and `./setup` removes an existing gstack-owned `investigate` / `gstack-investigate` entry on every host, along with the opt-in /investigate gate's hooks. For a bug, fix it directly: check the cause once, and when it is still unclear, write a focused test that fails on the bug before any edit, then fix the root cause and rerun the test. `/verify` runs the repo's checks while you build, and `/qa` tests a running site, API or CLI and fixes what it finds.
|
||||
The skill no longer installs, and `./setup` removes an existing gstack-owned `investigate` / `gstack-investigate` entry on every host, along with the opt-in /investigate gate's hooks. For a bug, just ask the model to fix it. gstack's router also stopped giving bug reports a fix-it-test-first rule on 2026-11-17 (GST-2873), because a test on 26 hard real bugs found no clear gain from it: +3.8 points, with a 95% interval of -11.5 to +19.2. The optional routing block gstack offers for your CLAUDE.md still has its own bugs line (check the cause once, or write a failing test first when it is unclear); this change leaves it as it is. `/verify` runs the repo's checks while you build, and `/qa` tests a running site, API or CLI and fixes what it finds.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user