mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
* v1.91.79.0 feat(eval-plan): /eval-plan tier 1, lane-check, evidence verify, coordinator contract, owner brief, contributor-PR mode, banned terms (multi-agent wave PR E) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(spec,review): fail closed when the coordinator contract cannot be read; restore the omitted-Army sentence The spec --execute spawn captures `gstack-autoplan contract` before piping the archive, so a lane never starts without the contract; the C9 fixture installs its gstack-like checkout (bin, lib, a contract stub) at every host's global root. Review Step 5.8 keeps the pinned 'without claiming specialist coverage' wording. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
10 KiB
10 KiB
Multi-agent wave: incident-to-test map
Each incident from the gbrain feedback (docs/designs/MULTI_AGENT_WAVE_2026_10_10.md, Validation 1) becomes a fixture or scripted scenario here, named by plan section and incident, and a test shows the tool catching it or producing the artifact. Later PRs of the wave append their rows; a row is never removed.
| Incident | Plan item | Fixture / scenario | Test |
|---|---|---|---|
| p0-2-hand-written-artifacts: the plan's own run wrote its artifacts by hand because no validator existed | B0 | reference-run/ (the plan branch's artifacts, run-dir-relative paths, status: complete) |
test/headless-artifacts.test.ts "reference run validates" |
| p0-2-interrupted-run: a parent consumed a run whose reviewer never returned | B0 | reference-run/ with status: running / a voice pending |
test/headless-artifacts.test.ts "interrupted run" |
| p0-2-omitted-finding: a count that matched hid an omitted finding | B0 | reference-run/ with a dropped row and an unchanged count, then a changed hash |
test/headless-artifacts.test.ts "count mismatch", "stale artifact" |
| p0-2-missing-reviewer: a phase reported CONFIRMED with one voice absent | B0 | reference-run/ with phases.eng.outside removed |
test/headless-artifacts.test.ts "missing reviewer" |
| p0-2-path-escape: an artifact list naming a file outside the run directory | B0 | symlink out of a temp run dir | test/headless-artifacts.test.ts "path escape" |
p0-2-claimed-clean: status: clean hand-written beside 176 open findings |
B7 | findings.jsonl with open rows and a claimed-clean row | test/review-log-findings.test.ts |
p0-2-headless-collision: headless redefinition would change eval semantics |
B1 | GSTACK_HEADLESS=1 beside GSTACK_SESSION_KIND=unattended |
test/gstack-session-kind.test.ts "unattended" |
| p0-2-unattended-egress: an unattended run synced data nobody approved | B1 | stubbed egress bins on PATH | test/gstack-skill-start.test.ts "unattended … egress" |
p0-2-gate-grammar: d10 yes, 1a 2a, a 33-decision approve-all |
B9 | reply strings | test/gate-list.test.ts |
| p0-2-timing: no wall-time record per phase, no pre-run estimate | B10 | analytics history | test/autoplan-timing.test.ts |
| p0-2-urgent-block: a security finding buried in a 400-line record | B12 | findings with urgent: true |
test/headless-artifacts.test.ts "URGENT block" |
| p1-5-unverified-premise: the wave's load-bearing premise ("13 of 28 threads ran nothing") had no receipt | D1 | reality-rows.json premise-table (positive from ceo-native.md:11-37, receipt-less decoy) |
test/plan-reality.test.ts "positive fixture", "decoy" |
| p1-5-already-done-rebuilt: a fix re-planned after main had merged it | D1 | reality-rows.json already-done (positive from eng-native.md:9, tier 1 n/a decoy) |
test/plan-reality.test.ts "decoy", "tier 1 n/a is missing" |
p1-5-surface-miss: bun upgrade --version assumed; --help lists only --canary |
D1 | reality-rows.json surface-check (positive from eng-native.md:191, prose-only decoy) |
test/plan-reality.test.ts "prose has every row missing" |
| p1-5-overruled-binding-decision: an owner decision undone by a project default two phases later | D1 | reality-rows.json binding-decisions (lookalike row id decoy) |
test/plan-reality.test.ts "unknown row ids are reported, not counted" |
| p1-5-uncredible-numbers: an effort table with no basis credited until re-counted | D1 | reality-rows.json numbers (positive from the plan's Effort table, receipt-less decoy) |
test/plan-reality.test.ts "decoy" |
| p1-5-deferred-asks-unlisted: two of six issue asks found only after merge | D1 | reality-rows.json deferred-asks (positive from "Not in this wave") |
test/plan-reality.test.ts "positive fixture" |
| p1-5-pr-count-unstated: five PRs became seven with no queue cost stated | D1 | reality-rows.json pr-count (positive from eng-native.md:23) |
test/plan-reality.test.ts "positive fixture" |
| p1-5-row-missing-is-a-warning: an unattended phase closed with a required row missing | D1 | six of seven rows under GSTACK_SESSION_KIND=unattended |
test/plan-reality.test.ts "verdict=incomplete", bin "check" |
| p1-6-cache-skips-guard: a new cache in front of a guarded call skipped the policy check on a hit | D2 | review/checklist.md Sideways sweep (cache bullet) |
test/sideways-review.test.ts "every shape the plan names" |
| p1-6-sibling-ingress-downgraded: a defect fixed at one ingress and rated low while its sibling stayed | D2 | review Step 4 sideways sweep, forced-CRITICAL list; cso Phase 12 sibling sweep | test/sideways-review.test.ts "Step 4", "cso" |
| p1-6-undeclared-behavior-change: a diff changed a default the PR body never mentioned and the audit said CLEAN | D2 | review-mode Scope Check Diff read / Undeclared behavior changes lines |
test/sideways-review.test.ts "Scope Check" |
| p1-7-flake-loosened: a flaky test was "fixed" with a wider timeout and no probe | D3 | investigate/sections/flake.md (never loosen, P(fail | regression), ## Flake evidence) |
test/investigate-recipes.test.ts "flake" |
| p1-7-polluter: a test green alone, red in its shard, rerun until green | D3 | investigate/sections/polluter.md (ship-measure free --files bisect) |
test/investigate-recipes.test.ts "polluter" |
| p1-7-backlog-closed-unreproduced: an issue closed from a triage that never ran the repro | D3 | investigate/sections/backlog.md (repro required for still_open, read-only, no closing) |
test/investigate-recipes.test.ts "backlog" |
| p1-7-stale-index-timing: a benchmark in a fresh worktree charged git's index refresh to the first measurement | D3 | benchmark/SKILL.md.tmpl and investigate Phase 1 git update-index --refresh |
test/investigate-recipes.test.ts "Phase 1 additions" |
| p0-4-stale-goldens: a generated file (ship goldens, agents digest) hand-merged or left stale, caught only by CI | C4, C8 regen |
test/helpers/pregate-fixture.ts repo: src/version.js bumped, generated/digest.md unchanged; registry on origin/main |
test/gstack-regen.test.ts "stale"; test/pregate-selection.test.ts "p0-4-stale-goldens" |
p0-4-scratch-script: zz-probe.sh committed at the repository root |
C8 strays |
fixture repo with zz-probe.sh, notes.txt, a new top-level directory and a conventional CONTRIBUTING.md |
test/pregate-selection.test.ts "p0-4-scratch-script" |
| p0-4-heavy-old-constant: a nightly-only test kept asserting the old timeout literal | C8 literals |
test/nightly/retry-heavy.test.js asserting TIMEOUT_MS 5000 after lib/retry.js moves to 9000 |
test/pregate-selection.test.ts "p0-4-heavy-old-constant" |
| p0-4-windows-lane-unrun: a Windows-only test changed on a Linux box merged with that lane unrun | C8 lanes |
test/windows-paths.test.js touched; windows.yml/windows-shard is a planner-computed matrix with a selection manifest |
test/pregate-selection.test.ts "p0-4-windows-lane-unrun"; test/pregate-invalidation.test.ts "uncleared requires-remote" |
p0-4-pipe-hidden-failure: a failing runner behind | tail reported green; a zero-run lane reported green |
C8 lanes (pipefail) |
runner node --test {files} 2>&1 | tail -n 1 with a broken lib/util.js; runner that prints # tests 0 |
test/pregate-selection.test.ts "p0-4-pipe-hidden-failure" |
| p0-4-stale-pregate: a pre-gate record reused after the final stamp moved the tree | C8 publication | pregate.json recorded, then README.md edited |
test/pregate-invalidation.test.ts "PREGATE_STALE for both" |
p0-4-dropped-vendored-hunk: a patchedDependencies hunk silently dropped on regeneration |
C8 patches (tier 2, deferred) |
— | — (follow-up pre-gate PR) |
| p1-8-unpriced-arm: a paid eval ran arms whose price the table did not know, so the admission estimate was $0 | E1 arms, dry-run |
resolveArms with a made-up model; dryRun with an unpriced arm |
test/pricing-eval-plan.test.ts "EVAL_PRICE_MISSING", "price_table.missing" |
| p1-8-blank-prereg: bars and stop rules were written after the first priced call | E1 prereg --check |
preregTemplate() with every section empty |
test/pricing-eval-plan.test.ts "empty section fails the check" |
| p1-8-malformed-rows: rows without a prompt field reached the paid run; the retry prompt was never read | E1 dry-run |
rows.jsonl with a non-JSON line and a row missing prompt; no retry prompt; a 100-token output budget |
test/pricing-eval-plan.test.ts "flags malformed rows" |
| p1-8-pilot-overrun: a run kept reserving calls past its cap; unknown charges were counted as zero | E1 pilot |
cap $0.0001; one attempt settled unknown |
test/pricing-eval-plan.test.ts "EVAL_BUDGET_EXHAUSTED", "verdict unknown" |
| p2-9-overlapping-lanes: two lanes edited the same planned file and the second merge rewrote the first | E2 gstack-lane-check |
fixture repo with lane-a (PR #7) and lane-b both pushed; planned src/a.ts, docs/** |
test/lane-coordination.test.ts "LANE_CONFLICT" |
| p2-9-borrowed-evidence: a bundle produced on another machine was cited as a pass here | E2 gstack-evidence verify |
bundle with no matching ledger record; --rerun on a moved tree |
test/lane-coordination.test.ts "stays unverified", "moved tree refuses" |
| p2-9-contract-drift: lane prompts paraphrased the merge conditions and spend caps differently | E2 {{COORDINATOR_CONTRACT}} |
resolver, gstack-autoplan contract, both reviewer prompts, spec's spawn pipe |
test/lane-coordination.test.ts "same bytes" |
| p2-10-buried-gate: a 400-line final report hid the decisions and a critical finding | E3 gstack-owner-brief render |
nine-item gate with one urgent finding | test/owner-brief-contributor-banned.test.ts "one page" |
p2-10-draft-left-standing: draft direction stands until the owner decides survived the owner's answer |
E3 gstack-owner-brief close |
plan with three draft lines, one still pending | test/owner-brief-contributor-banned.test.ts "close rewrites" |
| p2-10-outside-pr-as-code: an outside PR was reviewed as code first and credited without carried-over evidence | E3 gstack-contributor-mode |
explicit maintainers, read vs write permission, gh offline |
test/owner-brief-contributor-banned.test.ts "contributor mode" |
| p2-10-banned-term: a renamed product's old name landed in README while the glossary was allowed | E3 gstack-banned-terms |
diff touching README, docs/migration/**, the glossary and the term file itself |
test/owner-brief-contributor-banned.test.ts "added lines are scanned" |