mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-17 13:44:32 +01:00
test: remove the AI scorer that grades skill prose for clarity (freeze repair, GST-1371) (#4009)
* test: remove the skill-prose clarity LLM judges (GST-1371) Removes the 21 eval cases where an LLM scored a skill's instruction text for clarity, completeness and actionability, with their registrations, input builders, response cache, calibration corpora, the cookie-workflow manual-review approval and scripts/judge-controls.ts. The three remaining skill-llm-eval judges keep their prompts, thresholds and samples; the file's supervision wall shrinks to fit them. Outcome judges are untouched. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: pin the retired judge-control script and cookie manual-review record as deleted (GST-1371) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: register the manual-review fixture as free (GST-1371) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: keep the orphaned clarity-judge calibration corpora for a follow-up deletion (GST-1371) Deleting the four corpus directories alone puts the diff over the /ship review ceiling; they serve no configuration now and go in a follow-up. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(todos): file the clarity-judge removal follow-ups (GST-1371) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: keep skill-bundle section paths POSIX so references dedupe on Windows (GST-1371) windows-scoped-shard (run 41918796523) failed 'cross-skill references retain complete bytes once': path.join gave the section entry a backslash path, so a reference to the same section was not recognized as already bundled. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
622e8b1f76
commit
781f928ccf
55 files changed
+209
-3000
No files matched your search
+10
-18
@@ -324,13 +324,10 @@ changes are explicitly deferred, not counted as PR passes. Weekly/manual runs of
|
||||
runs the broad gate. This deliberately moves some defect detection later.
|
||||
|
||||
`scripts/eval-input-cache.ts` accepts only complete, clean, first-attempt passes
|
||||
with matching before/after inputs. The audited workflow-judge adapter hashes the
|
||||
actual expanded prompt, source/fixture/rubric/runner closure, installed SDK,
|
||||
model parameters and runtime. Missing/unknown inputs force execution. Receipts
|
||||
with matching before/after inputs. Missing/unknown inputs force execution. Receipts
|
||||
are scoped to the same repository and PR, expire after 24 hours, and contain
|
||||
public scores and provenance rather than prompts or secrets. Of the 17 cases
|
||||
using `runWorkflowJudge`, 16 are eligible; the cookie workflow's custom input
|
||||
does not match the cache adapter and stays fresh, as do the other 11 quality cases.
|
||||
public scores and provenance rather than prompts or secrets. The quality judges in
|
||||
`test/skill-llm-eval.test.ts` never reuse a receipt and always run fresh.
|
||||
CI supplies the scoped cache/runtime configuration; local runs are fresh by
|
||||
default. Cached scores must
|
||||
pass current assertions; reused records retain their original source and time
|
||||
@@ -455,7 +452,7 @@ and the kind fixes its trials before the run:
|
||||
non-JSON, a malformed field) fails the panel and is never resampled; a
|
||||
refusal counts as an unscored panel only when every sample refused.
|
||||
`callJudge`'s 429 backoff happens before any model output and is transport,
|
||||
not a verdict retry. The workflow-judge cache stores whole panels only.
|
||||
not a verdict retry.
|
||||
The `rule` case auq-matrix uses the same panel inside its one execution: each
|
||||
skill's single native capture has its recommendation substance scored by 3
|
||||
judge samples, gated on their median against the unchanged minimum of 4, and a
|
||||
@@ -840,11 +837,8 @@ cleanup. No per-case budget grows. Overlay wrappers
|
||||
have a 1,830-second minimum shard wall and run without Bun retries; see the
|
||||
[overlay contract](OVERLAY_BENCHMARK_CONTRACT.md) for their unchanged work budget.
|
||||
|
||||
The quality file reserves its whole-file wall (3,170 seconds) for every case run
|
||||
once, plus cleanup. Each still has 120 seconds of model work. Its 17 workflow
|
||||
judges own their deadline and abort signal, with five seconds for terminal
|
||||
recording inside a ten-second Bun grace; the other 11 retain their existing
|
||||
120-second Bun timeout. Late responses cannot create records or cache passes.
|
||||
The quality file reserves its whole-file wall (480 seconds) for every case run
|
||||
once, plus cleanup. Each of its three judges has a 120-second Bun timeout.
|
||||
When a judge's abort signal cancels its request, `callJudge`
|
||||
(`test/helpers/llm-judge.ts`) prints one `llm-judge-aborted` JSON line to
|
||||
stderr: elapsed time, wrapper `attempts` and `retryWaitsMs` (429 waits),
|
||||
@@ -1444,7 +1438,7 @@ once and reused from there. It prints:
|
||||
Screening figures are labelled `screen-provisional`: the rule's preamble
|
||||
detector is a line-commonality heuristic, not T2's hunk classifier, so they
|
||||
stay heuristic until T2 lands. A `package.json` change that only moves the
|
||||
version is neither selected nor a canary trigger, and all 24 judges always run.
|
||||
version is neither selected nor a canary trigger, and all 3 judges always run.
|
||||
The selectors are this checkout's, so an old merge is replayed against today's
|
||||
maps. A shallow checkout says which `git fetch --shallow-since` to run. Logic:
|
||||
`lib/gate-cost.ts`; tests: `test/gate-cost.test.ts`.
|
||||
@@ -1470,11 +1464,9 @@ integration tests, the Aside contract pins, and the render-wrapper pins.
|
||||
`bun run test:pr` runs the selected short live behaviors and quality judges.
|
||||
It reports deferred broad coverage; unknown dependencies restore the full gate,
|
||||
and an unmapped prompt without registered coverage blocks planning. Full free
|
||||
acceptance and required PR checks must pass before publishing. CI can reuse the
|
||||
16 workflow-judge passes for 24 hours when their complete consumed inputs and
|
||||
runtime match; records preserve original provenance. The cookie workflow's custom
|
||||
input, the other 11 judge cases, dynamic agent tests, and local runs without
|
||||
scoped cache configuration stay fresh.
|
||||
acceptance and required PR checks must pass before publishing. The quality
|
||||
judges, dynamic agent tests, and local runs without scoped cache configuration
|
||||
stay fresh.
|
||||
Scheduled/manual full coverage and `test:release` always run fresh.
|
||||
See [testing policy](../CONTRIBUTING.md#test-tiers) for commands and measured targets.
|
||||
Anything that needs Aside
|
||||
|
||||
@@ -15,7 +15,7 @@ different things even when they mention the same skill.
|
||||
| --- | --- | --- |
|
||||
| Selection, budgets, process ownership, cleanup, native-event parsing and recorded failure controls | Free runner and fixture regressions | A live model choosing or completing the right workflow |
|
||||
| Generated files, host parity, section manifests and complete input identity | Free generation and structural tests | An agent actually loading the required section |
|
||||
| Prompt clarity and rubric quality | The registered quality judge for that complete prompt | Tool execution, native acknowledgments or a finished report |
|
||||
| Targeted prompt properties (QA anti-refusal, cross-skill consistency, voice) | The registered quality judge for that prompt | Tool execution, native acknowledgments or a finished report |
|
||||
| Native first-question format and substance | Live SDK/PTY question capture | An answered question or a completed workflow; receipts explicitly say `workflowCompleted: false` |
|
||||
| Stochastic consistency and verbose/carved comparison | Independent captures, with separate stability and A/B oracles | One successful sample reused as three trials, or one prompt version standing in for the other |
|
||||
| Decisions, findings and report completion | Per-skill native workflow fixtures | The first question alone, screen text without native evidence, or a generic question count |
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Workflow-judge control bundles (C1), pre-registered 2026-10-04
|
||||
|
||||
> Historical record. The workflow clarity judges and `scripts/judge-controls.ts` were removed in GST-1371 (2026-11-04); the evals that check real outcomes replace them.
|
||||
|
||||
Purpose: learn whether the workflow judges that produced low samples in the eight
|
||||
wave censuses (37174266054 … 37198445662) can separate a sound workflow from a
|
||||
broken one. Score compression at 4 is a hypothesis; these controls test it.
|
||||
|
||||
Reference in new issue
Block a user