test: remove the AI scorer that grades skill prose for clarity (freeze repair, GST-1371) (#4009)

* test: remove the skill-prose clarity LLM judges (GST-1371)

Removes the 21 eval cases where an LLM scored a skill's instruction text for
clarity, completeness and actionability, with their registrations, input
builders, response cache, calibration corpora, the cookie-workflow manual-review
approval and scripts/judge-controls.ts. The three remaining skill-llm-eval
judges keep their prompts, thresholds and samples; the file's supervision wall
shrinks to fit them. Outcome judges are untouched.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: pin the retired judge-control script and cookie manual-review record as deleted (GST-1371)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: register the manual-review fixture as free (GST-1371)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep the orphaned clarity-judge calibration corpora for a follow-up deletion (GST-1371)

Deleting the four corpus directories alone puts the diff over the /ship review
ceiling; they serve no configuration now and go in a follow-up.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(todos): file the clarity-judge removal follow-ups (GST-1371)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep skill-bundle section paths POSIX so references dedupe on Windows (GST-1371)

windows-scoped-shard (run 41918796523) failed 'cross-skill references retain
complete bytes once': path.join gave the section entry a backslash path, so a
reference to the same section was not recognized as already bundled.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-04 22:19:08 +00:00
1 parent 622e8b1f76
commit 781f928ccf
55 files changed
+209 -3000

No files matched your search

@@ -1,13 +0,0 @@
{
"schema_version": 1,
"test_name": "setup-browser-cookies/SKILL.md workflow",
"prompt_sha256": "7f7f76f49912818d51f25c86d686aab6ad2c95ccdea60ac912c1e4c8b6936e5f",
"prompt_bytes": 10159,
"model": "claude-fable-5-1",
"max_tokens": 8192,
"thresholds": { "clarity": 4, "completeness": 3, "actionability": 4 },
"approved_by": "garrytan",
"approved_at": "2026-09-24",
"approval_url": "https://github.com/garrytan/gstack/pull/2964#issuecomment-5822514476",
"reason": "Maintainer-approved manual review of this exact cookie workflow after empty provider refusals. No automated quality score or pass credit; other errors and changed inputs remain blocking."
}
-22
View File
@@ -1,22 +0,0 @@
# Cookie workflow manual-review exception
`cookie-workflow-manual-review.json` records the maintainer's approval for one
exact cookie-workflow judge request. It is not generated content. Do not update
its hash, model, budget, thresholds, or provenance just to make a changed test
pass; a changed request needs a new explicit review and approval.
The ordinary judge request still runs. Only an explicit provider refusal with
complete request/response identifiers, zero output tokens, and no text blocks
can use this approval. Low scores, malformed output or evidence, other errors,
timeouts, and late or superseded attempts remain failures. Every other case
remains subject to its existing gate.
Manual acceptance is first-attempt-only: a retry refusal cannot erase an earlier
scored failure, timeout, or other error. Normal configured retries are unchanged.
The collector preserves `passed: false`, `execution: executed`, the refusal,
and the manual approval, without a score or score-cache receipt. Reports count
manual acceptance separately from automated passes and failures; “executed”
counts the actual provider request, not a completed scored evaluation. Historical
records retain that distinction. CI also checks manual claims against the
current source and committed approval before accepting them.