mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-11 03:00:26 +01:00
test: cut the last two behavior-guessing LLM judges; check review-bot history facts for free (GST-1403) (#4014)
* test: cut the qa anti-refusal and voice-tone judges; check review-bot history facts for free (GST-1403) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: count assertions follow the deleted judge file (GST-1403) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
6bdafb2cf5
commit
55637db835
136 files changed
+189
-5267
No files matched your search
@@ -46,7 +46,6 @@ gstack/
|
||||
│ ├── skill-validation.test.ts # Tier 1: static validation (free, <1s)
|
||||
│ ├── gen-skill-docs.test.ts # Tier 1: generator quality (free, <1s)
|
||||
│ ├── setup-*.test.ts, relink.test.ts, hook-scripts.test.ts # Tier 1: setup linker ownership, retired-skill prune, browser hint, rebuild check + Chromium bootstrap (anchor-sliced from setup), gstack-relink, PreToolUse hooks (free)
|
||||
│ ├── skill-llm-eval.test.ts # Tier 3: LLM-as-judge (~$0.15/run)
|
||||
│ └── skill-e2e-*.test.ts # Tier 2: E2E via claude -p (~$3.85/run, split by category)
|
||||
├── qa/, qa-only/ # Surface-aware browser/functional QA; /qa-only reports and proposes tests without product edits
|
||||
├── plan-design-review/ # /plan-design-review skill (report-only design audit)
|
||||
|
||||
@@ -326,8 +326,7 @@ runs the broad gate. This deliberately moves some defect detection later.
|
||||
`scripts/eval-input-cache.ts` accepts only complete, clean, first-attempt passes
|
||||
with matching before/after inputs. Missing/unknown inputs force execution. Receipts
|
||||
are scoped to the same repository and PR, expire after 24 hours, and contain
|
||||
public scores and provenance rather than prompts or secrets. The quality judges in
|
||||
`test/skill-llm-eval.test.ts` never reuse a receipt and always run fresh.
|
||||
public scores and provenance rather than prompts or secrets.
|
||||
CI supplies the scoped cache/runtime configuration; local runs are fresh by
|
||||
default. Cached scores must
|
||||
pass current assertions; reused records retain their original source and time
|
||||
@@ -837,8 +836,6 @@ cleanup. No per-case budget grows. Overlay wrappers
|
||||
have a 1,830-second minimum shard wall and run without Bun retries; see the
|
||||
[overlay contract](OVERLAY_BENCHMARK_CONTRACT.md) for their unchanged work budget.
|
||||
|
||||
The quality file reserves its whole-file wall (480 seconds) for every case run
|
||||
once, plus cleanup. Each of its three judges has a 120-second Bun timeout.
|
||||
When a judge's abort signal cancels its request, `callJudge`
|
||||
(`test/helpers/llm-judge.ts`) prints one `llm-judge-aborted` JSON line to
|
||||
stderr: elapsed time, wrapper `attempts` and `retryWaitsMs` (429 waits),
|
||||
@@ -1438,7 +1435,7 @@ once and reused from there. It prints:
|
||||
Screening figures are labelled `screen-provisional`: the rule's preamble
|
||||
detector is a line-commonality heuristic, not T2's hunk classifier, so they
|
||||
stay heuristic until T2 lands. A `package.json` change that only moves the
|
||||
version is neither selected nor a canary trigger, and all 3 judges always run.
|
||||
version is neither selected nor a canary trigger.
|
||||
The selectors are this checkout's, so an old merge is replayed against today's
|
||||
maps. A shallow checkout says which `git fetch --shallow-since` to run. Logic:
|
||||
`lib/gate-cost.ts`; tests: `test/gate-cost.test.ts`.
|
||||
|
||||
@@ -15,7 +15,7 @@ different things even when they mention the same skill.
|
||||
| --- | --- | --- |
|
||||
| Selection, budgets, process ownership, cleanup, native-event parsing and recorded failure controls | Free runner and fixture regressions | A live model choosing or completing the right workflow |
|
||||
| Generated files, host parity, section manifests and complete input identity | Free generation and structural tests | An agent actually loading the required section |
|
||||
| Targeted prompt properties (QA anti-refusal, cross-skill consistency, voice) | The registered quality judge for that prompt | Tool execution, native acknowledgments or a finished report |
|
||||
| Facts several skills must state alike (review-bot history paths, columns, record types, categories) | Free cross-file fact tests (`test/review-bot-history-facts.test.ts`) | An agent acting on those facts in a live run |
|
||||
| Native first-question format and substance | Live SDK/PTY question capture | An answered question or a completed workflow; receipts explicitly say `workflowCompleted: false` |
|
||||
| Stochastic consistency and verbose/carved comparison | Independent captures, with separate stability and A/B oracles | One successful sample reused as three trials, or one prompt version standing in for the other |
|
||||
| Decisions, findings and report completion | Per-skill native workflow fixtures | The first question alone, screen text without native evidence, or a generic question count |
|
||||
|
||||
Reference in new issue
Block a user