mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-07 17:28:23 +01:00
test: remove the AI scorer that grades skill prose for clarity (freeze repair, GST-1371) (#4009)
* test: remove the skill-prose clarity LLM judges (GST-1371) Removes the 21 eval cases where an LLM scored a skill's instruction text for clarity, completeness and actionability, with their registrations, input builders, response cache, calibration corpora, the cookie-workflow manual-review approval and scripts/judge-controls.ts. The three remaining skill-llm-eval judges keep their prompts, thresholds and samples; the file's supervision wall shrinks to fit them. Outcome judges are untouched. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: pin the retired judge-control script and cookie manual-review record as deleted (GST-1371) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: register the manual-review fixture as free (GST-1371) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: keep the orphaned clarity-judge calibration corpora for a follow-up deletion (GST-1371) Deleting the four corpus directories alone puts the diff over the /ship review ceiling; they serve no configuration now and go in a follow-up. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(todos): file the clarity-judge removal follow-ups (GST-1371) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: keep skill-bundle section paths POSIX so references dedupe on Windows (GST-1371) windows-scoped-shard (run 41918796523) failed 'cross-skill references retain complete bytes once': path.join gave the section entry a backslash path, so a reference to the same section was not recognized as already bundled. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
622e8b1f76
commit
781f928ccf
55 files changed
+209
-3000
No files matched your search
@@ -1,13 +0,0 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"test_name": "setup-browser-cookies/SKILL.md workflow",
|
||||
"prompt_sha256": "7f7f76f49912818d51f25c86d686aab6ad2c95ccdea60ac912c1e4c8b6936e5f",
|
||||
"prompt_bytes": 10159,
|
||||
"model": "claude-fable-5-1",
|
||||
"max_tokens": 8192,
|
||||
"thresholds": { "clarity": 4, "completeness": 3, "actionability": 4 },
|
||||
"approved_by": "garrytan",
|
||||
"approved_at": "2026-09-24",
|
||||
"approval_url": "https://github.com/garrytan/gstack/pull/2964#issuecomment-5822514476",
|
||||
"reason": "Maintainer-approved manual review of this exact cookie workflow after empty provider refusals. No automated quality score or pass credit; other errors and changed inputs remain blocking."
|
||||
}
|
||||
@@ -1,22 +0,0 @@
|
||||
# Cookie workflow manual-review exception
|
||||
|
||||
`cookie-workflow-manual-review.json` records the maintainer's approval for one
|
||||
exact cookie-workflow judge request. It is not generated content. Do not update
|
||||
its hash, model, budget, thresholds, or provenance just to make a changed test
|
||||
pass; a changed request needs a new explicit review and approval.
|
||||
|
||||
The ordinary judge request still runs. Only an explicit provider refusal with
|
||||
complete request/response identifiers, zero output tokens, and no text blocks
|
||||
can use this approval. Low scores, malformed output or evidence, other errors,
|
||||
timeouts, and late or superseded attempts remain failures. Every other case
|
||||
remains subject to its existing gate.
|
||||
|
||||
Manual acceptance is first-attempt-only: a retry refusal cannot erase an earlier
|
||||
scored failure, timeout, or other error. Normal configured retries are unchanged.
|
||||
|
||||
The collector preserves `passed: false`, `execution: executed`, the refusal,
|
||||
and the manual approval, without a score or score-cache receipt. Reports count
|
||||
manual acceptance separately from automated passes and failures; “executed”
|
||||
counts the actual provider request, not a completed scored evaluation. Historical
|
||||
records retain that distinction. CI also checks manual claims against the
|
||||
current source and committed approval before accepting them.
|
||||
Reference in new issue
Block a user