Stop giving bug reports a special rule in gstack's router (GST-2873) (#4442)

gstack's router no longer gives bug reports a special check-the-cause,
fix-it-test-first rule, and its description no longer lists debugging,
so the model fixes a bug the way it would without gstack. This keeps
gstack to the parts that measurably help, and on 26 hard real bugs the
rule gave no clear gain.

Nothing to do.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Review-Finding: p1/red-team#1 reader=red-team
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-17 14:37:34 +00:00
1 parent b79843c5d7
commit f000b6e0e3
19 files changed
+25 -368

No files matched your search

+9 -18
View File
@@ -2012,25 +2012,19 @@ appears only when it is part of the lesson, and datasets or tools are opened
only with a stated reason. The rules and their honesty bars are in
[ETHOS.md, Explain the Number](../ETHOS.md#8-explain-the-number).
**Safety-rule and router evals: a descriptive third arm.** The W1 safety-rule
**Safety-rule evals: a descriptive third arm.** The W1 safety-rule
evals compare `GSTACK_SAFETY_ARM=rule` (the shipped rule) with `removed` (the
rule paragraph deleted), and the router failure-routing evals compare
`GSTACK_ROUTER_ARM=current` with `before` (the base branch's router). Both
variables also take `none` (GST-470): the same fixture, tools and user request,
with no skill excerpt, rule text, step file or installed skill. Any other value
throws, so a typo cannot run one arm under another's label. The `rule`,
`removed`, `current` and `before` prompts are byte for byte what they were
before `none` existed (`test/safety-none-arm.test.ts`,
`test/failure-routing-arm.test.ts`). To run the three arms as a comparison, run
the same case once per arm, each as its CI panel:
rule paragraph deleted). The variable also takes `none` (GST-470): the same
fixture, tools and user request, with no skill excerpt, rule text, step file or
installed skill. Any other value throws, so a typo cannot run one arm under
another's label. The `rule` and `removed` prompts are byte for byte what they
were before `none` existed (`test/safety-none-arm.test.ts`). To run the three
arms as a comparison, run the same case once per arm, each as its CI panel:
```bash
for arm in rule removed none; do
GSTACK_SAFETY_ARM=$arm bun run scripts/test-paid-shards.ts --tier gate --case safety-ship-stale-evidence
done
for arm in current before none; do
GSTACK_ROUTER_ARM=$arm bun run scripts/test-paid-shards.ts --tier gate --case router-failure-verified-cause
done
```
The arm variable is part of the shard-reuse environment, so one arm's result is
@@ -2046,11 +2040,8 @@ ran, the session ended normally) still assert. This departs from "the same
panel" above because these contracts are gstack's own rules: a plain model was
never asked to keep them, so a failing `none` trial is the measurement, not a
regression, and a red verdict would block the gate on behavior nobody
specified. The router's `none` arm has no repro-first metric,
since it never sees the router's rules; it records plain metrics instead:
whether the suite is green, whether an edit came before the first read, and
how many read-only calls came before the first edit. The first paid comparison
with these arms gets its own preregistration.
specified. The first paid comparison with these arms gets its own
preregistration.
**Cases without a `none` arm.** `safety-pair-agent-block` and
`safety-ios-demo-ui-only` refuse `GSTACK_SAFETY_ARM=none` with the reason
+1 -1
View File
@@ -660,7 +660,7 @@ I want the model imagining the production incident before it happens.
`/investigate` was cut on 2026-11-07 (GST-1717) after a preregistered eval found no gain. On 18 real bug-fix tasks that plain Opus 5.5 solves about half the time, 20 trials each, Opus told to use /investigate passed 50.0% of trials against 47.2% with no skill: +2.8 points (95% CI -5.8 to +11.4), below the 10-point lift the run could detect. A passing fix cost about 43% more with the skill. The evidence is GEVA-380's R3c result, [gstack-evals c90ab277, wiki/board/2026-11-07-geva-380-r3c-results.md](https://github.com/garrytan/gstack-evals/blob/c90ab277/wiki/board/2026-11-07-geva-380-r3c-results.md).
The skill no longer installs, and `./setup` removes an existing gstack-owned `investigate` / `gstack-investigate` entry on every host, along with the opt-in /investigate gate's hooks. For a bug, fix it directly: check the cause once, and when it is still unclear, write a focused test that fails on the bug before any edit, then fix the root cause and rerun the test. `/verify` runs the repo's checks while you build, and `/qa` tests a running site, API or CLI and fixes what it finds.
The skill no longer installs, and `./setup` removes an existing gstack-owned `investigate` / `gstack-investigate` entry on every host, along with the opt-in /investigate gate's hooks. For a bug, just ask the model to fix it. gstack's router also stopped giving bug reports a fix-it-test-first rule on 2026-11-17 (GST-2873), because a test on 26 hard real bugs found no clear gain from it: +3.8 points, with a 95% interval of -11.5 to +19.2. The optional routing block gstack offers for your CLAUDE.md still has its own bugs line (check the cause once, or write a failing test first when it is unclear); this change leaves it as it is. `/verify` runs the repo's checks while you build, and `/qa` tests a running site, API or CLI and fixes what it finds.
---