test: cut the last two behavior-guessing LLM judges; check review-bot history facts for free (GST-1403) (#4014)

* test: cut the qa anti-refusal and voice-tone judges; check review-bot history facts for free (GST-1403)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: count assertions follow the deleted judge file (GST-1403)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-05 02:30:04 +00:00
1 parent 6bdafb2cf5
commit 55637db835
136 files changed
+189 -5267

No files matched your search

-1
View File
@@ -46,7 +46,6 @@ gstack/
│ ├── skill-validation.test.ts # Tier 1: static validation (free, <1s)
│ ├── gen-skill-docs.test.ts # Tier 1: generator quality (free, <1s)
│ ├── setup-*.test.ts, relink.test.ts, hook-scripts.test.ts # Tier 1: setup linker ownership, retired-skill prune, browser hint, rebuild check + Chromium bootstrap (anchor-sliced from setup), gstack-relink, PreToolUse hooks (free)
│ ├── skill-llm-eval.test.ts # Tier 3: LLM-as-judge (~$0.15/run)
│ └── skill-e2e-*.test.ts # Tier 2: E2E via claude -p (~$3.85/run, split by category)
├── qa/, qa-only/ # Surface-aware browser/functional QA; /qa-only reports and proposes tests without product edits
├── plan-design-review/ # /plan-design-review skill (report-only design audit)
+2 -5
View File
@@ -326,8 +326,7 @@ runs the broad gate. This deliberately moves some defect detection later.
`scripts/eval-input-cache.ts` accepts only complete, clean, first-attempt passes
with matching before/after inputs. Missing/unknown inputs force execution. Receipts
are scoped to the same repository and PR, expire after 24 hours, and contain
public scores and provenance rather than prompts or secrets. The quality judges in
`test/skill-llm-eval.test.ts` never reuse a receipt and always run fresh.
public scores and provenance rather than prompts or secrets.
CI supplies the scoped cache/runtime configuration; local runs are fresh by
default. Cached scores must
pass current assertions; reused records retain their original source and time
@@ -837,8 +836,6 @@ cleanup. No per-case budget grows. Overlay wrappers
have a 1,830-second minimum shard wall and run without Bun retries; see the
[overlay contract](OVERLAY_BENCHMARK_CONTRACT.md) for their unchanged work budget.
The quality file reserves its whole-file wall (480 seconds) for every case run
once, plus cleanup. Each of its three judges has a 120-second Bun timeout.
When a judge's abort signal cancels its request, `callJudge`
(`test/helpers/llm-judge.ts`) prints one `llm-judge-aborted` JSON line to
stderr: elapsed time, wrapper `attempts` and `retryWaitsMs` (429 waits),
@@ -1438,7 +1435,7 @@ once and reused from there. It prints:
Screening figures are labelled `screen-provisional`: the rule's preamble
detector is a line-commonality heuristic, not T2's hunk classifier, so they
stay heuristic until T2 lands. A `package.json` change that only moves the
version is neither selected nor a canary trigger, and all 3 judges always run.
version is neither selected nor a canary trigger.
The selectors are this checkout's, so an old merge is replayed against today's
maps. A shallow checkout says which `git fetch --shallow-since` to run. Logic:
`lib/gate-cost.ts`; tests: `test/gate-cost.test.ts`.
+1 -1
View File
@@ -15,7 +15,7 @@ different things even when they mention the same skill.
| --- | --- | --- |
| Selection, budgets, process ownership, cleanup, native-event parsing and recorded failure controls | Free runner and fixture regressions | A live model choosing or completing the right workflow |
| Generated files, host parity, section manifests and complete input identity | Free generation and structural tests | An agent actually loading the required section |
| Targeted prompt properties (QA anti-refusal, cross-skill consistency, voice) | The registered quality judge for that prompt | Tool execution, native acknowledgments or a finished report |
| Facts several skills must state alike (review-bot history paths, columns, record types, categories) | Free cross-file fact tests (`test/review-bot-history-facts.test.ts`) | An agent acting on those facts in a live run |
| Native first-question format and substance | Live SDK/PTY question capture | An answered question or a completed workflow; receipts explicitly say `workflowCompleted: false` |
| Stochastic consistency and verbose/carved comparison | Independent captures, with separate stability and A/B oracles | One successful sample reused as three trials, or one prompt version standing in for the other |
| Decisions, findings and report completion | Per-skill native workflow fixtures | The first question alone, screen text without native evidence, or a generic question count |