mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-05 16:34:00 +01:00
docs(inventory): add the development-suite limit to the /review caveats (GST-1370)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
e23dee4aa5
commit
56c1fb045e
2 files changed
+3
-2
No files matched your search
@@ -1 +1 @@
|
||||
- **The evidence ledger records /review as helps, with its two caveats beside the status.** R2 at the after pin (gstack-evals #207, superseded by #211) measured /review with the contract Stop hook on Opus 5.5 at gstack 699a41cb: blocking recall 26.2% against 3.6% for plain Opus 5.5 with no gstack, +22.6 points (95% CI +11.0 to +33.7). The `review` row in `docs/inventory/evidence.json` now says helps, with its sources, scope and R48 KEEP. A new `caveats` field, allowed only on helps, no-effect and hurts rows, holds the result's limits, and `docs/inventory.md` and `gstack-skill-bill` print them beside the status: /review and the hook count as one unit (42 of 66 sessions ran the readers only after the hook blocked), and 41 of the suite's 42 bugs came from gstack-family reviews. The earlier review results stay under `prior`.
|
||||
- **The evidence ledger records /review as helps, with its three caveats beside the status.** R2 at the after pin (gstack-evals #207, superseded by #211) measured /review with the contract Stop hook on Opus 5.5 at gstack 699a41cb: blocking recall 26.2% against 3.6% for plain Opus 5.5 with no gstack, +22.6 points (95% CI +11.0 to +33.7). The `review` row in `docs/inventory/evidence.json` now says helps, with its sources, scope and R48 KEEP. A new `caveats` field, allowed only on helps, no-effect and hurts rows, holds the result's limits, and `docs/inventory.md` and `gstack-skill-bill` print them beside the status: /review and the hook count as one unit (42 of 66 sessions ran the readers only after the hook blocked), 41 of the suite's 42 bugs came from gstack-family reviews, and stacks-r1 is a development suite, so the result makes no release claim. The earlier review results stay under `prior`.
|
||||
@@ -438,7 +438,8 @@
|
||||
},
|
||||
"caveats": [
|
||||
"one unit: /review and the contract Stop hook were measured together, and neither gets credit alone; 42 of 66 gstack sessions ran the review readers only after the hook blocked them (24 unprompted), and that split is not randomized",
|
||||
"home-field bias in gstack's favor: 41 of the 42 blocking bugs in stacks-r1 came from gstack-family reviews, so +22.6 may run high; a suite seeded from other reviewers' bugs would remove the bias (GEVA-2 lists it as future work)"
|
||||
"home-field bias in gstack's favor: 41 of the 42 blocking bugs in stacks-r1 came from gstack-family reviews, so +22.6 may run high; a suite seeded from other reviewers' bugs would remove the bias (GEVA-2 lists it as future work)",
|
||||
"development suite: stacks-r1 is a development suite, so the result makes no release claim and holds only for these 33 units at these settings"
|
||||
],
|
||||
"prior": [
|
||||
{
|
||||
|
||||
Reference in new issue
Block a user