mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-17 13:44:32 +01:00
* Count review rounds per pull request across /ship runs (GST-2673) /ship's two-round review cap was counted inside each run, so a pull request shipped many times was reviewed again at every head. Step 1 now reads the rounds the PR body's receipt history records, passes them to the review plan, and sends later heads to the scoped re-check, with a lane or Captain round grant for an extra round. Adds the no-loss replay preregistration. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Cap full review rounds per pull request and keep fix-checks to each run The plan counts a PR's earlier full rounds (--prior-full) against the cap of two, as the approved plan says, and prints every earlier round as pr_rounds=. Renames the troubleshooting anchor to pr-round-cap, lists <SESSION_ID> as a reviewed placeholder, and leaves ship/DRIVER.md at its size cap. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Point the ship driver's review pass at Step 9's plan flags, which carry the PR round count Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Give a pull request's final head a full review and make the diff-only re-check advice only (GST-2673) A scoped re-check that reads only what changed since the last reviewed head caught 22 of 24 planted faults but none of 8 real faults that full rounds had caught, so it no longer carries a /ship receipt. A later head whose own lines changed now gets a fresh full review, the PR's last before merge; a PR already at two full rounds gets one read-only full review with no fix round, whose blocker goes to the lane's Captain. carry prints RECHECK: advisory and refuses --recheck. The drift threshold, the re-check blocker count and the design-change grant kind are removed with the route they fed. Repins lib/review-depth.ts and ship/sections/review-army.md under replay GST-2673 (docs/evals/pr-round-count-results.md; deterministic grid against main: 196,608 of 196,608 plans identical with no prior rounds). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Keep a review round grant usable through its fix round and after a read-only review A grant now holds after the fix round's commits move HEAD past it, counts only the rounds earlier heads spent, and a receipt written without a run keeps other runs' stubs. Review-Finding: p1/specialist-batched#1 reader=state-machines Review-Finding: p1/red-team#2 reader=red-team Review-Finding: p1/red-team#3 reader=red-team Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Count every review round at a granted head again, so a grant can't reset the count Reverts the history change that skipped rounds recorded at a round grant's head: the run that hit the cap is recorded at that same head, so skipping it wiped the count. A grant recorded after a read-only review still doesn't buy a fix round (fails closed). Review-Finding: p2/adversarial-native#1 reader=adversarial-native Review-Finding: p2/outside-voice#1 reader=outside-voice Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
634 lines
27 KiB
JSON
634 lines
27 KiB
JSON
{
|
|
"schema_version": 1,
|
|
"note": "R56 evidence ledger (ETHOS.md: 9. Earn Its Place). One row per inventory key (docs/inventory.md lists them); bin/gstack-inventory --check validates it and test/inventory.test.ts runs that check. Statuses and their required fields: helps, no-effect, hurts need sources, models (frontier per R54), prompt: preset, scope and masked: true, and no-effect and hurts also r48 {verdict, ref}; caveats lists a verdict's limits, shown beside its status; untested needs plan (a reference, or none only when grandfathered), and a new part also reason, owner and measure_by; safety needs reason. prior lists earlier results that do not count yet, with why. Seeded 2026-11-02 from GEVA-2's table (org-gstack-gbrain/research/gstack-part-evidence.json, gstack dac71371a); every row present at that base is grandfathered.",
|
|
"seed_source": "org-gstack-gbrain/research/gstack-part-evidence.json (GEVA-156 for GST-915, 2026-11-02, gstack-evals main 409ff47)",
|
|
"parts": {
|
|
"autoplan": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-3 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: autoplan: untested (never measured)"
|
|
},
|
|
"benchmark": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: benchmark: untested (never measured)"
|
|
},
|
|
"benchmark-models": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: benchmark-models: untested (never measured)"
|
|
},
|
|
"browse": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: browse: untested (never measured)"
|
|
},
|
|
"canary": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: canary: untested (never measured)"
|
|
},
|
|
"captain": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: captain: untested (never measured)"
|
|
},
|
|
"careful": {
|
|
"status": "safety",
|
|
"grandfathered": true,
|
|
"reason": "safety guard: warns before destructive commands (rm -rf, DROP TABLE, force-push); kept on that reason, an average score cannot see the rare bad run it stops",
|
|
"seed": "GEVA-2, 2026-11-02: careful: exempt (safety guard)"
|
|
},
|
|
"claude-code": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: claude-code: untested (never measured)"
|
|
},
|
|
"codex": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "not in GEVA-2's table (2026-11-02); never measured"
|
|
},
|
|
"context-restore": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: context-restore: untested (never measured)"
|
|
},
|
|
"context-save": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: context-save: untested (never measured)"
|
|
},
|
|
"cso": {
|
|
"status": "safety",
|
|
"grandfathered": true,
|
|
"reason": "security: supported security findings with explicit coverage; kept on its security reason (GEVA-2 marks it exempt; GStack Minder may reclassify it)",
|
|
"seed": "GEVA-2, 2026-11-02: cso: exempt (security audit skill, treated as exempt; GStack Minder may reclassify)"
|
|
},
|
|
"design-consultation": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-10 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: design-consultation: untested (never measured)"
|
|
},
|
|
"design-html": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: design-html: untested (never measured)"
|
|
},
|
|
"design-motion": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: design-motion: untested (never measured)"
|
|
},
|
|
"design-review": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-9 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: design-review: untested (never measured)"
|
|
},
|
|
"design-shotgun": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: design-shotgun: untested (never measured)"
|
|
},
|
|
"deslop-shared-libs": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: deslop-shared-libs: untested (never measured)"
|
|
},
|
|
"devex-review": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-11 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: devex-review: untested (never measured)"
|
|
},
|
|
"diagram": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: diagram: untested (never measured)"
|
|
},
|
|
"document-generate": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: document-generate: untested (never measured)"
|
|
},
|
|
"document-release": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: document-release: untested (never measured)"
|
|
},
|
|
"eval-plan": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: eval-plan: untested (never measured)"
|
|
},
|
|
"freeze": {
|
|
"status": "safety",
|
|
"grandfathered": true,
|
|
"reason": "safety guard: hard-blocks edits outside one directory; kept on that reason",
|
|
"seed": "GEVA-2, 2026-11-02: freeze: exempt (safety guard)"
|
|
},
|
|
"gstack": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: gstack (root skill): untested (never measured)",
|
|
"aliases": [
|
|
"gstack (root skill)"
|
|
]
|
|
},
|
|
"gstack-upgrade": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: upgrade: untested (never measured)",
|
|
"aliases": [
|
|
"upgrade"
|
|
]
|
|
},
|
|
"guard": {
|
|
"status": "safety",
|
|
"grandfathered": true,
|
|
"reason": "safety guard: careful and freeze together; kept on that reason",
|
|
"seed": "GEVA-2, 2026-11-02: guard: exempt (safety guard)"
|
|
},
|
|
"health": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: health: untested (never measured)"
|
|
},
|
|
"ios-clean": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: ios-clean: untested (never measured)"
|
|
},
|
|
"ios-design-review": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: ios-design-review: untested (never measured)"
|
|
},
|
|
"ios-fix": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: ios-fix: untested (never measured)"
|
|
},
|
|
"ios-qa": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: ios-qa: untested (never measured)"
|
|
},
|
|
"ios-sync": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: ios-sync: untested (never measured)"
|
|
},
|
|
"land-and-deploy": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-8 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: land-and-deploy: untested (never measured)"
|
|
},
|
|
"landing-report": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: landing-report: untested (never measured)"
|
|
},
|
|
"learn": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"prior": [
|
|
{
|
|
"ref": "GSTA-13, GEVA-54 and GEVA-65 Lane A runs (wiki/board/2026-10-28-gsta-13-quick-look-3-results and the GEVA-54 and GEVA-65 posts)",
|
|
"why": [
|
|
"stub",
|
|
"no-true-no-gstack-arm",
|
|
"ceiling"
|
|
],
|
|
"note": "Opus contract tasks 27/30 vs 27/30"
|
|
},
|
|
{
|
|
"ref": "GEVA-65 Lane A rerun row 5 (GPT-6.1 Sol in Codex)",
|
|
"why": [
|
|
"not-claude-code",
|
|
"no-true-no-gstack-arm"
|
|
],
|
|
"note": "Sol failed the no-code-change safety task in 5 of 9 skill trials vs 0 of 3 without; small n, a lead to check, not a hurts verdict"
|
|
}
|
|
],
|
|
"seed": "GEVA-2, 2026-11-02: learn: untested (unfair test: SDK stub; no true no-gstack arm; Opus at ceiling; Sol ran in Codex, not Claude Code)"
|
|
},
|
|
"make-pdf": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: make-pdf: untested (never measured)"
|
|
},
|
|
"modernize": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: modernize: untested (never measured)"
|
|
},
|
|
"office-hours": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"prior": [
|
|
{
|
|
"ref": "GSTA-13 quick look 3 Lane A, wiki/board/2026-10-28-gsta-13-quick-look-3-results",
|
|
"why": [
|
|
"stub",
|
|
"no-true-no-gstack-arm"
|
|
],
|
|
"note": "contract tasks on Opus: gstack 74% vs none 28%; the tasks grade the skill's own contract, a manipulation check, not lift"
|
|
},
|
|
{
|
|
"ref": "GEVA-65 Lane A rerun, wiki/board/2026-10-31-geva-65-lanea-shakedown-rerun-results",
|
|
"why": [
|
|
"stub",
|
|
"no-true-no-gstack-arm"
|
|
],
|
|
"note": "Opus before 0.67, after 0.76, none 0.36"
|
|
}
|
|
],
|
|
"seed": "GEVA-2, 2026-11-02: office-hours: untested (unfair test: SDK stub; no true no-gstack arm; tasks grade the skill's own contract, a manipulation check, not lift)"
|
|
},
|
|
"open-gstack-browser": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: open-gstack-browser: untested (never measured)"
|
|
},
|
|
"pair-agent": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: pair-agent: untested (never measured)"
|
|
},
|
|
"plan-ceo-review": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-5 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"prior": [
|
|
{
|
|
"ref": "CEO panel v2 admission (GEVA-142), wiki/board/2026-11-02-geva-142-ceo-panel-v2-admission-results (open PR #131)",
|
|
"why": [
|
|
"ceiling",
|
|
"unmasked"
|
|
],
|
|
"note": "preset, Opus 5.5 only: must-fix recall 1.00 in all three arms; plain invocation ran the procedure in 0 of 16"
|
|
},
|
|
{
|
|
"ref": "Planning R1a pilot (GEVA-102), wiki/board/2026-11-01-geva-102-planning-r1a-pilot-stop (#114)",
|
|
"why": [
|
|
"stub",
|
|
"ceiling",
|
|
"unmasked"
|
|
],
|
|
"note": "none-opus 8/8; gstack-opus read the sections file in 0 of 4"
|
|
}
|
|
],
|
|
"r48": {
|
|
"verdict": "FIX-METHOD",
|
|
"ref": "CEO panel v2 admission (GEVA-142, open PR #131): FIX-METHOD on the panel, a harder panel is planned"
|
|
},
|
|
"seed": "GEVA-2, 2026-11-02: plan-ceo-review: untested (unfair test: ceiling, control 12 of 12 so no MDD; plain invocation ran the procedure in 0 of 16)"
|
|
},
|
|
"plan-design-review": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-7 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: plan-design-review: untested (never measured)"
|
|
},
|
|
"plan-devex-review": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-6 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: plan-devex-review: untested (never measured)"
|
|
},
|
|
"plan-eng-review": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-2 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: plan-eng-review: untested (never measured)"
|
|
},
|
|
"plan-tune": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: plan-tune: untested (never measured)"
|
|
},
|
|
"qa": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: qa: untested (never measured)"
|
|
},
|
|
"qa-only": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: qa-only: untested (never measured)"
|
|
},
|
|
"retro": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-12 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: retro: untested (never measured)"
|
|
},
|
|
"review": {
|
|
"status": "helps",
|
|
"grandfathered": true,
|
|
"sources": [
|
|
"gstack-evals #207 (baa599a9) wiki/board/2026-11-04-geva-271-r2-after-pin-results",
|
|
"gstack-evals #211 (81daa79) wiki/board/2026-11-04-geva-281-r2-after-pin-results-home-field"
|
|
],
|
|
"models": [
|
|
"claude-opus-5-5"
|
|
],
|
|
"prompt": "preset",
|
|
"scope": "/review as a Claude Code user installs it at gstack 699a41cb (./setup --host claude), contract Stop hook included, told /review, against plain Opus 5.5 with no gstack; Claude Code CLI 2.1.314 on the claude_code preset; stacks-r1, 33 units, 42 blocking bugs, 2 passes: blocking recall 22/84 (26.2%) vs 3/84 (3.6%), +22.6 points [95% PR-clustered CI +11.0, +33.7], MDD 24.5",
|
|
"masked": true,
|
|
"r48": {
|
|
"verdict": "KEEP",
|
|
"ref": "gstack-evals #211, wiki/board/2026-11-04-geva-281-r2-after-pin-results-home-field (What we found: R48 KEEP, rr-decide.py KEEP)"
|
|
},
|
|
"caveats": [
|
|
"one unit: /review and the contract Stop hook were measured together, and neither gets credit alone; 42 of 66 gstack sessions ran the review readers only after the hook blocked them (24 unprompted), and that split is not randomized",
|
|
"home-field bias in gstack's favor: 41 of the 42 blocking bugs in stacks-r1 came from gstack-family reviews, so +22.6 may run high; a suite seeded from other reviewers' bugs would remove the bias (GEVA-2 lists it as future work)",
|
|
"development suite: stacks-r1 is a development suite, so the result makes no release claim and holds only for these 33 units at these settings"
|
|
],
|
|
"prior": [
|
|
{
|
|
"ref": "gstack-evals #108 stacks round 1 review (GEVA-89), wiki/board/2026-11-01-geva-89-stacks-round-1-review-results",
|
|
"why": [
|
|
"stub",
|
|
"no-exposure",
|
|
"unmasked"
|
|
],
|
|
"note": "Opus 5.5 blocking recall 9.5% vs 7.1%, +2.4 points [-6.4, +11.1]; specialists and adversarial pass ran in 0 of 66 sessions"
|
|
},
|
|
{
|
|
"ref": "gstack-evals #120 R1 blind re-judge (GEVA-113), wiki/board/2026-11-01-geva-113-r1-blind-rejudge-results",
|
|
"why": [
|
|
"stub",
|
|
"unmasked"
|
|
],
|
|
"note": "gstack-opus minus none-opus +3.5 points [-9.6, +16.7]"
|
|
},
|
|
{
|
|
"ref": "GSTA-11 quick look 1 review replay, wiki/board/2026-10-28-gsta-11-review-replay-results",
|
|
"why": [
|
|
"not-claude-code",
|
|
"unmasked"
|
|
],
|
|
"note": "/review's prompts through the Messages API; Opus recall 44% vs 39% plain, 12 bugs"
|
|
}
|
|
],
|
|
"seed": "GEVA-2, 2026-11-02: review: measurement under way (R2)",
|
|
"note": "R2 at the after pin: GEVA-271 under prereg #194 (amending #154), runs review-units-20261104T070239Z-cbff (gstack-full) and review-units-20261104T123631Z-3fb3 (none-rr); #211 (GEVA-281) supersedes #207's post with every number unchanged and adds the home-field limit. Until then, this row was untested with R48 FIX-METHOD (org-gstack-gbrain/research/why-null-stacks-r1.md, GEVA-100), plan R2 (gstack-evals #129). R2's first two attempts gave no outcome: GEVA-175 (procedure gate 10 of 19) and GEVA-229 (exposure smoke 2 of 3). masked: the no-gstack arm ran in the rig's Docker sandbox with no gstack install, and #207's read audit of its 66 transcripts found no read of ~/.capy/drive or ~/.local/share/gstack. Only Opus 5.5 was measured; GPT-6.1 Sol was not."
|
|
},
|
|
"scrape": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: scrape: untested (never measured)"
|
|
},
|
|
"setup-browser-cookies": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: setup-browser-cookies: untested (never measured)"
|
|
},
|
|
"setup-deploy": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: setup-deploy: untested (never measured)"
|
|
},
|
|
"setup-gbrain": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: setup-gbrain: untested (never measured)"
|
|
},
|
|
"ship": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-4 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"seed": "GEVA-2, 2026-11-02: ship: untested (never measured)"
|
|
},
|
|
"skillify": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: skillify: untested (never measured)"
|
|
},
|
|
"spec": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: spec: untested (never measured)"
|
|
},
|
|
"sync-gbrain": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: sync-gbrain: untested (never measured)"
|
|
},
|
|
"test-audit": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: test-audit: untested (never measured)"
|
|
},
|
|
"ubicloud": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: ubicloud: untested (never measured)"
|
|
},
|
|
"unfreeze": {
|
|
"status": "safety",
|
|
"grandfathered": true,
|
|
"reason": "safety guard: the off switch for freeze's edit restriction; kept with the guard it releases",
|
|
"seed": "GEVA-2, 2026-11-02: unfreeze: exempt (safety guard (undoes freeze))"
|
|
},
|
|
"runaway-stopper": {
|
|
"status": "untested",
|
|
"plan": "docs/evals/runaway-stopper-prereg.md",
|
|
"reason": "new skill (GST-1279): its detector is tested offline against real thread readings; whether the skill's actions and prevent defaults cut spend is not measured yet",
|
|
"owner": "GStack Minder (GSTA-36)",
|
|
"measure_by": "2026-11-24"
|
|
},
|
|
"verify": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "not in GEVA-2's table (2026-11-02); never measured"
|
|
},
|
|
"write-skill": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: write-skill: untested (never measured)"
|
|
},
|
|
"preamble": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "I1 ceremony cut (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-108-ceremony-cut-prereg (#122, main 409ff47) with its G3 amendment (#128), preset, Opus 5.5 and particle-1",
|
|
"prior": [
|
|
{
|
|
"ref": "GEVA-100 exposure counts, org-gstack-gbrain/research/why-null-stacks-r1.md (GEVA-100)",
|
|
"why": [
|
|
"stub",
|
|
"no-exposure"
|
|
],
|
|
"note": "ceremony sections ran in 0 of 66 Opus review sessions and 6 of 111 Opus TB2 trials; no effect estimate"
|
|
}
|
|
],
|
|
"r48": {
|
|
"verdict": "CUT-CANDIDATE",
|
|
"ref": "org-gstack-gbrain/research/why-null-stacks-r1.md (GEVA-100); taken by Garry as the I1 ceremony-cut eval"
|
|
},
|
|
"seed": "GEVA-2, 2026-11-02: preamble (ceremony): measurement under way (I1, ceremony-cut eval)",
|
|
"aliases": [
|
|
"preamble (ceremony)"
|
|
]
|
|
},
|
|
"catalog": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "Run D (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-125-untold-discovery-prereg (#125, main 9820b14), untold gstack vs placebo vs no gstack, preset vs stub, Opus 5.5, GPT-6.1 Sol and particle-1; also measures /investigate's description",
|
|
"prior": [
|
|
{
|
|
"ref": "GSTA-12 Lane B quick look 1, wiki/board/2026-10-29-gsta-12-laneb-quick-results",
|
|
"why": [
|
|
"stub",
|
|
"no-exposure",
|
|
"unmasked"
|
|
],
|
|
"note": "untold skill opened in 0 of 180 gstack-arm trials; Opus gstack minus none +5 [-3, +17]"
|
|
}
|
|
],
|
|
"seed": "GEVA-2, 2026-11-02: always-on skill catalog (descriptions): measurement under way (Run D)",
|
|
"aliases": [
|
|
"always-on skill catalog (descriptions)"
|
|
],
|
|
"note": "L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, pins 82db89ab → 699a41cb): the description rewrite alone had no effect (off-u, routing and contracts off with the rewritten descriptions kept: 0/80 untold /investigate opens). The rewrite helps only inside the untold routing package; see the investigate row's caveats and this ledger's unmapped rows."
|
|
},
|
|
"capy-skill": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: capy/skill/gstack/SKILL.md (Capy context): untested (never measured)",
|
|
"aliases": [
|
|
"capy/skill/gstack/SKILL.md (Capy context)"
|
|
],
|
|
"note": "loads only in Capy sessions, which the claude_code preset fair test cannot reach yet; needs a Captain call on how to test it"
|
|
},
|
|
"capy-factory": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "none",
|
|
"seed": "GEVA-2, 2026-11-02: capy/skill/gstack/factory.md (Capy context): untested (never measured)",
|
|
"aliases": [
|
|
"capy/skill/gstack/factory.md (Capy context)"
|
|
],
|
|
"note": "loads only in Capy sessions, which the claude_code preset fair test cannot reach yet; needs a Captain call on how to test it"
|
|
},
|
|
"ship-review-panel": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "GEVA-2 registry row R56-1 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
|
"prior": [
|
|
{
|
|
"ref": "gstack-evals #120 R1 blind re-judge (GEVA-113), wiki/board/2026-11-01-geva-113-r1-blind-rejudge-results",
|
|
"why": [
|
|
"stub",
|
|
"not-claude-code",
|
|
"unmasked"
|
|
],
|
|
"note": "arm C minus none +56.9 points [40.5, 72.2]; the none arm ran on the stub, arm C in Capy at another gstack version, on a cohort mined from gstack's own catches"
|
|
},
|
|
{
|
|
"ref": "GSTA-48 review no-loss arm D (#77), wiki/board/2026-10-30-gsta-48-review-noloss-arm-how-to",
|
|
"why": [
|
|
"variant-only",
|
|
"not-claude-code"
|
|
],
|
|
"note": "arm D vs pooled arm C, no no-gstack arm"
|
|
}
|
|
],
|
|
"seed": "GEVA-2, 2026-11-02: ship: review panel (specialist, adversarial, targeted, outside voice, red team): untested (unfair test: none arm on the SDK stub; arms from different harnesses and gstack versions; cohort mined from gstack's own catches)",
|
|
"aliases": [
|
|
"ship: review panel (specialist, adversarial, targeted, outside voice, red team)"
|
|
],
|
|
"note": "GST-2673 adds the per-PR round count to Step 9 (docs/review-policy.md#rounds-per-pull-request); its replay (docs/evals/pr-round-count-prereg.md) found a diff-only re-check caught 0 of 8 real faults, so re-checks are advice only and the merge head gets a full review. Owner GStack Minder (GSTA-36); measure by 2026-12-08"
|
|
},
|
|
"ship-docs-audit": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "folded into GEVA-2 registry row R56-4 (ship); proposed, not yet preregistered",
|
|
"seed": "GEVA-2, 2026-11-02: ship: docs audit (Step 14.5): untested (never measured)",
|
|
"aliases": [
|
|
"ship: docs audit (Step 14.5)"
|
|
]
|
|
},
|
|
"autoplan-phases": {
|
|
"status": "untested",
|
|
"grandfathered": true,
|
|
"plan": "folded into GEVA-2 registry row R56-3 (autoplan); proposed, not yet preregistered",
|
|
"seed": "GEVA-2, 2026-11-02: autoplan: CEO, design, DX and eng phases, outside voice: untested (never measured)",
|
|
"aliases": [
|
|
"autoplan: CEO, design, DX and eng phases, outside voice"
|
|
]
|
|
}
|
|
},
|
|
"unmapped": [
|
|
{
|
|
"row": "review: Review Army and adversarial pass",
|
|
"why": "a section inside /review, not its own key; R2 measures it as part of the review row's plan"
|
|
},
|
|
{
|
|
"row": "ship: QA probes (Step 9.2.1)",
|
|
"why": "a step inside /ship with no separate size; folded into the ship row (GEVA-2 registry row R56-4)"
|
|
},
|
|
{
|
|
"row": "gstack as a whole (installed, any skill)",
|
|
"why": "not a single text part; GEVA-2's real-world outcome lane measures it (gstack-evals plan #111), not a measurement plan for any one part"
|
|
},
|
|
{
|
|
"row": "untold routing package (route hook, contracts and the description rewrite)",
|
|
"why": "two Claude Code hooks and four description lines, not one text part with a key. L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb): helps, untold /investigate opens 17/80 vs 0/80, B1 +21.3 [+12.5, +30.0], R48 KEEP; it needs routing or contracts on, and hook vs contracts aren't split; the description rewrite alone had no effect (off-u 0/80). The investigate row carries it as a caveat"
|
|
}
|
|
]
|
|
}
|