diff --git a/.github/cookie-workflow-manual-review.json b/.github/cookie-workflow-manual-review.json deleted file mode 100644 index 61d80dbde..000000000 --- a/.github/cookie-workflow-manual-review.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "schema_version": 1, - "test_name": "setup-browser-cookies/SKILL.md workflow", - "prompt_sha256": "7f7f76f49912818d51f25c86d686aab6ad2c95ccdea60ac912c1e4c8b6936e5f", - "prompt_bytes": 10159, - "model": "claude-fable-5-1", - "max_tokens": 8192, - "thresholds": { "clarity": 4, "completeness": 3, "actionability": 4 }, - "approved_by": "garrytan", - "approved_at": "2026-09-24", - "approval_url": "https://github.com/garrytan/gstack/pull/2964#issuecomment-5822514476", - "reason": "Maintainer-approved manual review of this exact cookie workflow after empty provider refusals. No automated quality score or pass credit; other errors and changed inputs remain blocking." -} diff --git a/.github/cookie-workflow-manual-review.md b/.github/cookie-workflow-manual-review.md deleted file mode 100644 index 96f8611d6..000000000 --- a/.github/cookie-workflow-manual-review.md +++ /dev/null @@ -1,22 +0,0 @@ -# Cookie workflow manual-review exception - -`cookie-workflow-manual-review.json` records the maintainer's approval for one -exact cookie-workflow judge request. It is not generated content. Do not update -its hash, model, budget, thresholds, or provenance just to make a changed test -pass; a changed request needs a new explicit review and approval. - -The ordinary judge request still runs. Only an explicit provider refusal with -complete request/response identifiers, zero output tokens, and no text blocks -can use this approval. Low scores, malformed output or evidence, other errors, -timeouts, and late or superseded attempts remain failures. Every other case -remains subject to its existing gate. - -Manual acceptance is first-attempt-only: a retry refusal cannot erase an earlier -scored failure, timeout, or other error. Normal configured retries are unchanged. - -The collector preserves `passed: false`, `execution: executed`, the refusal, -and the manual approval, without a score or score-cache receipt. Reports count -manual acceptance separately from automated passes and failures; “executed” -counts the actual provider request, not a completed scored evaluation. Historical -records retain that distinction. CI also checks manual claims against the -current source and committed approval before accepting them. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index aa74a6704..d0f9dc987 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -319,9 +319,7 @@ then spend one paid run with the line's `after a repair:` command. A red census on `main` is reported on the weekly tracking issue; a census dispatched on a branch writes the same report to its run summary and `census-report` artifact. -CI enables verified first-attempt reuse for 16 workflow quality judges for -24 hours within the same PR. The cookie workflow's custom input and the other 11 -quality cases stay fresh. PR-profile E2E shards that run once (no retry, so the +Quality judges always run fresh. PR-profile E2E shards that run once (no retry, so the pass is provably a first attempt) reuse a pass from the same PR when every consumed input is byte-identical: the test's import closure, every tracked file its registered cases' touchfiles and the global touchfiles match, the runner and diff --git a/TODOS.md b/TODOS.md index 4f97a9c57..0142b7da1 100644 --- a/TODOS.md +++ b/TODOS.md @@ -2,6 +2,11 @@ ## NEXT PRIORITY +### P3: clarity-judge removal follow-ups (filed 2026-11-04, GST-1371) + +- **Delete the four orphaned judge-calibration corpora.** `test/fixtures/judge-calibration/{qa-workflow,qa-health-rubric,workflow-default,cookie-workflow}/` serve no configuration since the skill-prose clarity judges were removed; deleting them in the removal PR put its diff over the /ship review ceiling. Delete them alone once main is unfrozen (or after `review_exclude_paths` lands), and drop their rows from the README's results tables if the record is no longer wanted. **Effort:** S. **Priority:** P3. +- **Refresh the judge counts the removal left stale.** `docs/TESTING_INTERNALS.md`'s all-green probability table counts 24 PR-gate judges and `scripts/gate-cost-replay.ts`'s header says complete full-gate runs hold 24; there are 3 now. Recompute the table from current rates rather than editing the count alone. `scripts/paid-test-durations.json` still records the 24-judge `test/skill-llm-eval.test.ts` duration until the next paid report rewrites it. **Effort:** S. **Priority:** P3. + ### P3: screen verdict reuse lookback (filed 2026-11-02, GST-1027) - **Screen reuse sees only the newest 50 evals.yml dispatches.** `listGateRuns` (`scripts/recover-receipts.ts`) lists dispatched screens as one unfiltered page (`event=workflow_dispatch`, `per_page=50`), shared with every PR's screens and every train. At week-B traffic (220 dispatches from 2026-10-25 to 11-02, about 27 a day) that is under two days, and nothing says when a PR's earlier screen fell off the page: `gate-reuse find` prints "no earlier executed gate on this PR bound a verdict" and the full screen runs. No miss yet: the longest gap among the week-B re-screens was 44 dispatches (#3607). Page the listing until runs are older than the PR's oldest candidate head (or filter by `created>=`), and print a truncation line when the last page is full. A missed run only costs a screen, never a wrong verdict. **Effort:** S. **Priority:** P3. @@ -285,15 +290,6 @@ Deferred from the approved reliability follow-ups plan (docs/designs/RELIABILITY runs (none of 20 without pty-workspace-trust, not significant). Next step: move the git calls in shared fixture helpers to async spawn with a hard timeout, or take a Bun release that fixes #34069. **Effort:** M. **Priority:** P2. -- **Judge rubric anchors (C1, needs Garry: EVAL_POLICY)** — the pre-registered - control panels (`docs/evals/judge-controls-2026-10.md`, 36 calls) showed - every workflow judge separates gross breakage, but the qa workflow judge - passed a /qa bundle with Phase 10 Report removed (no sample noticed) and the - review judge passed /review with Step 3 removed (two of three samples noticed - it and rated it minor); no positive sample scored 5 on any dimension. - Missing-step coverage now lives in `test/workflow-required-steps.test.ts`. - Proposal: anchor completeness/actionability 4 and 5 to named deliverables - and inputs, as a pre-registered EVAL_POLICY change. **Effort:** M. **Priority:** P2. - **auto-decide-preserved: the CEO handoff line no longer depends on the model** — periodic run 37272185151 went red because the model led its chat with a paraphrase; a reminder beside the line measured 1/10. The Oct 6 wave shows @@ -629,16 +625,6 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 `rule` but gates on one judge sample. Decide under EVAL_POLICY whether it is a `judge` case (3 samples, median) and supply pass-rate evidence before changing the kind. Effort S. **Priority:** P2. -- **qa/SKILL.md workflow judge dips below its actionability floor** — census - 37186854666 (b24b6d8) scored actionability 3.67 < 4 (clarity 3, completeness - 4) on the same qa bundle bytes that passed in census 37182865432 (44ef18c); - nothing under qa/, qa-only/ or scripts/resolvers changed between them. Red in - 2 of the 17 earlier censuses (36920606897, 36776104571). Judge kind, 3 - samples, unchanged threshold. The qa-only workflow judge did the same in - census 37198445662 (fd6854b): 3.67/4.33/3.67, one 3/4/3 sample citing density, - on qa-only bytes unchanged since 44ef18c, where it passed in four censuses. - Find which bundle step the low samples call unactionable before rewording. - Effort S. **Priority:** P3. - **shared-libs-review-revalidation stalls mid-stream** — gate census 4 of 37186854666: the `filtered` scenario's stream stopped after a thinking block at 07:51:21 and emitted nothing until the 657 s abort; the other three @@ -780,11 +766,6 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 by re-running `./setup` from it. Iterate the registry's render column instead. **Effort:** S. **Priority:** P3. -- **Judge actionability sits exactly at the 4.0 gate** (P2, from the PR #3014 eval - census) — all 24 workflow judges score actionability at 4.0 in nearly every run, so one - low sample of three fails the case (qa-only 9/78, qa 6/78, plan-ceo modes 5/78, plan-eng - sections 4/78, review 4/70 over 09-27..10-02). The repair is clearer skill text per case - (as done for qa in v1.91.14.0), not a lower threshold. Effort M. - **`--case` cannot select loop-registered paid cases** (P3) — coverage audits, TPA and plan-mode-no-op register their cases in a loop, so `test-paid-shards.ts --case ` runs nothing; teach `fileCaseRegistration` the loop pattern. Effort S. @@ -809,25 +790,11 @@ Each item was deferred with a reason during the `/claude-api prompt-audit` clean other models with no runner, and the `.agents` render has no Codex runner for this eval. Narrow the rewording to the design-review render, or add per-host runners. Priority P2. Effort M. -- **Judge prose and rubric cleanup (W2 deferral)** — the arm, qa health-rubric, - qa anti-refusal, cross-skill and voice judges, and the default workflow judge, - now send a JSON schema; their prompt-audit s11 M1-M7 prose cleanup is still - open, as is the qa anti-refusal rubric's "always … regardless" wording. Three - prose steps were triggered and not run: the qa workflow judge (its schema - comparison held: 4/24 new flips vs 2/24 old, one new false pass on a06, zero - format errors, so it stays on today's request), the default workflow judge - (landed, but one provider refusal on corpus item f03 under the new prompt, - matched by one under the old prompt) and the cookie workflow judge (GSTA-91: - held on one new false pass on a07, zero new-request format errors; the hold was - released on GStack Minder's 2026-10-29 ruling and it now sends the schema; today's request itself false-passes a07 in `old-b` and on - both valid `old-a` samples, and a08 in `old-a`, so the prose step targets those). Reuse `test/fixtures/judge-calibration` - and `scripts/judge-calibration.ts`. Priority P3. Effort M. -- **Uncalibrated workflow-judge configurations** — build a corpus and calibrate - schema transport for the frontier configuration (review/SKILL.md workflow); it - stays on today's request. The cookie configuration - (setup-browser-cookies/SKILL.md workflow) has a corpus and a held comparison-1 - (GSTA-91); it moved to the schema on GStack Minder's 2026-10-29 release ruling. After any transport change, - re-run the review and cookie workflow judges. Priority P3. Effort M. +- **Judge prose and rubric cleanup (W2 deferral)** — the arm, qa anti-refusal, + cross-skill and voice judges now send a JSON schema; their prompt-audit s11 + M1-M7 prose cleanup is still open, as is the qa anti-refusal rubric's + "always … regardless" wording. Reuse `test/fixtures/judge-calibration` and + `scripts/judge-calibration.ts`. Priority P3. Effort M. - **Claude model auto-detection for the skill overlay** — when no `claude_overlay_model` is set, detect the Claude Code model (`ANTHROPIC_MODEL`, user and managed settings `model`) in `./setup` and render its pinned overlay. @@ -1688,8 +1655,7 @@ separately, so the change alters a paid verdict and needs its own paid run. **What:** GSTA-49 TM3 (2026-10-18) moved every remaining old-model paid pin to the current generation: the `claude-opus-4-7` pins in `skill-e2e-{plan,plan-prosons,design,office-hours-phase4,qa-bugs}` follow -`resolveEvalModel('capture')`, the overlay fixtures pin `claude-opus-5-5` and `claude-sonnet-5-5`, the cookie workflow -judge pins `claude-sonnet-5-5`, and `codex-sol-scope-termination` runs on `gpt-6.1-sol` (still rendering the +`resolveEvalModel('capture')`, the overlay fixtures pin `claude-opus-5-5` and `claude-sonnet-5-5`, and `codex-sol-scope-termination` runs on `gpt-6.1-sol` (still rendering the `gpt-5.6-sol` overlay). Three things remain: - The base `claude-dedicated-tools-vs-bash` fixture now duplicates `claude-dedicated-tools-vs-bash-opus-5-5`. diff --git a/changelog.d/capy-gst-1371-remove-clarity-judges.md b/changelog.d/capy-gst-1371-remove-clarity-judges.md new file mode 100644 index 000000000..5dd90a11d --- /dev/null +++ b/changelog.d/capy-gst-1371-remove-clarity-judges.md @@ -0,0 +1 @@ +- **The AI scorer that graded skill instructions for clarity is gone, so main's backstop no longer goes red on a model's opinion of /ship's prose.** The `ship/SKILL.md workflow` judge failed main's full run twice (runs 41817566885 and 41884510834), freezing merges. Garry's call: the evals that check real outcomes are the real eval. The 21 cases that had an LLM score a skill's instruction text for clarity, completeness and actionability are removed, along with their input builders, response cache, calibration configurations, the cookie-workflow manual-review approval and the control-bundle script. The three remaining judges in `test/skill-llm-eval.test.ts` (QA anti-refusal, cross-skill greptile consistency, voice directive) keep their thresholds and samples, and the judge file's supervision wall shrinks from 3,170 to 480 seconds to match. The paid report now rejects any manual-review claim. (GST-1371) diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 205427b23..ff5939113 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -324,13 +324,10 @@ changes are explicitly deferred, not counted as PR passes. Weekly/manual runs of runs the broad gate. This deliberately moves some defect detection later. `scripts/eval-input-cache.ts` accepts only complete, clean, first-attempt passes -with matching before/after inputs. The audited workflow-judge adapter hashes the -actual expanded prompt, source/fixture/rubric/runner closure, installed SDK, -model parameters and runtime. Missing/unknown inputs force execution. Receipts +with matching before/after inputs. Missing/unknown inputs force execution. Receipts are scoped to the same repository and PR, expire after 24 hours, and contain -public scores and provenance rather than prompts or secrets. Of the 17 cases -using `runWorkflowJudge`, 16 are eligible; the cookie workflow's custom input -does not match the cache adapter and stays fresh, as do the other 11 quality cases. +public scores and provenance rather than prompts or secrets. The quality judges in +`test/skill-llm-eval.test.ts` never reuse a receipt and always run fresh. CI supplies the scoped cache/runtime configuration; local runs are fresh by default. Cached scores must pass current assertions; reused records retain their original source and time @@ -455,7 +452,7 @@ and the kind fixes its trials before the run: non-JSON, a malformed field) fails the panel and is never resampled; a refusal counts as an unscored panel only when every sample refused. `callJudge`'s 429 backoff happens before any model output and is transport, - not a verdict retry. The workflow-judge cache stores whole panels only. + not a verdict retry. The `rule` case auq-matrix uses the same panel inside its one execution: each skill's single native capture has its recommendation substance scored by 3 judge samples, gated on their median against the unchanged minimum of 4, and a @@ -840,11 +837,8 @@ cleanup. No per-case budget grows. Overlay wrappers have a 1,830-second minimum shard wall and run without Bun retries; see the [overlay contract](OVERLAY_BENCHMARK_CONTRACT.md) for their unchanged work budget. -The quality file reserves its whole-file wall (3,170 seconds) for every case run -once, plus cleanup. Each still has 120 seconds of model work. Its 17 workflow -judges own their deadline and abort signal, with five seconds for terminal -recording inside a ten-second Bun grace; the other 11 retain their existing -120-second Bun timeout. Late responses cannot create records or cache passes. +The quality file reserves its whole-file wall (480 seconds) for every case run +once, plus cleanup. Each of its three judges has a 120-second Bun timeout. When a judge's abort signal cancels its request, `callJudge` (`test/helpers/llm-judge.ts`) prints one `llm-judge-aborted` JSON line to stderr: elapsed time, wrapper `attempts` and `retryWaitsMs` (429 waits), @@ -1444,7 +1438,7 @@ once and reused from there. It prints: Screening figures are labelled `screen-provisional`: the rule's preamble detector is a line-commonality heuristic, not T2's hunk classifier, so they stay heuristic until T2 lands. A `package.json` change that only moves the -version is neither selected nor a canary trigger, and all 24 judges always run. +version is neither selected nor a canary trigger, and all 3 judges always run. The selectors are this checkout's, so an old merge is replayed against today's maps. A shallow checkout says which `git fetch --shallow-since` to run. Logic: `lib/gate-cost.ts`; tests: `test/gate-cost.test.ts`. @@ -1470,11 +1464,9 @@ integration tests, the Aside contract pins, and the render-wrapper pins. `bun run test:pr` runs the selected short live behaviors and quality judges. It reports deferred broad coverage; unknown dependencies restore the full gate, and an unmapped prompt without registered coverage blocks planning. Full free -acceptance and required PR checks must pass before publishing. CI can reuse the -16 workflow-judge passes for 24 hours when their complete consumed inputs and -runtime match; records preserve original provenance. The cookie workflow's custom -input, the other 11 judge cases, dynamic agent tests, and local runs without -scoped cache configuration stay fresh. +acceptance and required PR checks must pass before publishing. The quality +judges, dynamic agent tests, and local runs without scoped cache configuration +stay fresh. Scheduled/manual full coverage and `test:release` always run fresh. See [testing policy](../CONTRIBUTING.md#test-tiers) for commands and measured targets. Anything that needs Aside diff --git a/docs/TEST_PORTFOLIO.md b/docs/TEST_PORTFOLIO.md index 78510a41e..e082f52b2 100644 --- a/docs/TEST_PORTFOLIO.md +++ b/docs/TEST_PORTFOLIO.md @@ -15,7 +15,7 @@ different things even when they mention the same skill. | --- | --- | --- | | Selection, budgets, process ownership, cleanup, native-event parsing and recorded failure controls | Free runner and fixture regressions | A live model choosing or completing the right workflow | | Generated files, host parity, section manifests and complete input identity | Free generation and structural tests | An agent actually loading the required section | -| Prompt clarity and rubric quality | The registered quality judge for that complete prompt | Tool execution, native acknowledgments or a finished report | +| Targeted prompt properties (QA anti-refusal, cross-skill consistency, voice) | The registered quality judge for that prompt | Tool execution, native acknowledgments or a finished report | | Native first-question format and substance | Live SDK/PTY question capture | An answered question or a completed workflow; receipts explicitly say `workflowCompleted: false` | | Stochastic consistency and verbose/carved comparison | Independent captures, with separate stability and A/B oracles | One successful sample reused as three trials, or one prompt version standing in for the other | | Decisions, findings and report completion | Per-skill native workflow fixtures | The first question alone, screen text without native evidence, or a generic question count | diff --git a/docs/evals/judge-controls-2026-10.md b/docs/evals/judge-controls-2026-10.md index 584eb2d73..5957c3c10 100644 --- a/docs/evals/judge-controls-2026-10.md +++ b/docs/evals/judge-controls-2026-10.md @@ -1,5 +1,7 @@ # Workflow-judge control bundles (C1), pre-registered 2026-10-04 +> Historical record. The workflow clarity judges and `scripts/judge-controls.ts` were removed in GST-1371 (2026-11-04); the evals that check real outcomes replace them. + Purpose: learn whether the workflow judges that produced low samples in the eight wave censuses (37174266054 … 37198445662) can separate a sound workflow from a broken one. Score compression at 4 is a hypothesis; these controls test it. diff --git a/scripts/e2e-shard-reuse.ts b/scripts/e2e-shard-reuse.ts index e6ffcc176..fcd4aa9e8 100644 --- a/scripts/e2e-shard-reuse.ts +++ b/scripts/e2e-shard-reuse.ts @@ -1,6 +1,6 @@ /** - * Verified first-attempt reuse for PR-lane E2E shards, on the same receipts as - * the workflow-judge reuse (scripts/eval-input-cache.ts). + * Verified first-attempt reuse for PR-lane E2E shards, on the receipts of + * scripts/eval-input-cache.ts. * * A PR-profile shard (a file, or one case of a case-sharded file) is reused * only when every consumed input is byte-identical to a fresh pass recorded in @@ -309,8 +309,8 @@ export function compareReceiptSources(a: { runId: string; completedAt: number }, /** * Planner-side selection: copy `from` into `to`, dropping every pass or panel * receipt that has a same-or-newer negative receipt for its identity, and - * every panel receipt that is not a whole PASS panel. Workflow-judge and - * other receipts pass through for their own validation at lookup. + * every panel receipt that is not a whole PASS panel. Other receipts pass + * through for their own validation at lookup. */ export function selectPlanReceipts(from: string, to: string, now = Date.now()): { shipped: number; blocked: string[] } { fs.mkdirSync(to, { recursive: true }); diff --git a/scripts/free-test-durations-windows.json b/scripts/free-test-durations-windows.json index e09083bad..e0c7ebc68 100644 --- a/scripts/free-test-durations-windows.json +++ b/scripts/free-test-durations-windows.json @@ -302,8 +302,6 @@ "test/context-budget-ratchet.test.ts": 586, "test/context-remaining-work-provenance.test.ts": 67, "test/context-restore-fragment-verdict.test.ts": 109, - "test/cookie-workflow-judge-input.test.ts": 4466, - "test/cookie-workflow-manual-review.test.ts": 256, "test/copilot-windows-bash.test.ts": 63, "test/cost-known.test.ts": 201, "test/coverage-audit.test.ts": 9537, @@ -819,7 +817,6 @@ "test/workflow-concurrency.test.ts": 72, "test/workflow-excerpt.test.ts": 328, "test/workflow-gh-token.test.ts": 73, - "test/workflow-judge-cache.test.ts": 17402, "test/workflow-required-steps.test.ts": 101, "test/workflow-shell-lint.test.ts": 107, "test/worktree-isolated-shapes.test.ts": 116, diff --git a/scripts/free-test-durations.json b/scripts/free-test-durations.json index 0ef0e54ad..8e8879ff8 100644 --- a/scripts/free-test-durations.json +++ b/scripts/free-test-durations.json @@ -441,8 +441,6 @@ "test/context-restore-head.test.ts": 1363, "test/context-restore-transcripts.test.ts": 409, "test/context-save-hardening.test.ts": 410, - "test/cookie-workflow-judge-input.test.ts": 1960, - "test/cookie-workflow-manual-review.test.ts": 74, "test/copilot-windows-bash.test.ts": 12, "test/cost-known.test.ts": 62, "test/coverage-audit-evidence.test.ts": 193, @@ -1463,7 +1461,6 @@ "test/workflow-concurrency.test.ts": 17, "test/workflow-excerpt.test.ts": 88, "test/workflow-gh-token.test.ts": 22, - "test/workflow-judge-cache.test.ts": 1128, "test/workflow-judge-input.test.ts": 34, "test/workflow-required-steps.test.ts": 17, "test/workflow-shell-lint.test.ts": 26, diff --git a/scripts/harness-pins/scripts/lib/paid-report.ts.json b/scripts/harness-pins/scripts/lib/paid-report.ts.json index 643e38909..c961930f7 100644 --- a/scripts/harness-pins/scripts/lib/paid-report.ts.json +++ b/scripts/harness-pins/scripts/lib/paid-report.ts.json @@ -1 +1 @@ -{"blob":"e0d18da4ea02ac443835a68992abeea2b47f22fc","since":6,"nonBehavioral":"the headline adds only streamed session cost; a billed session's cost is already in its collector total; cache writes without a 5m key fall back to the total minus 1h; cost fields only"} +{"blob":"31355b5a97706943e5fccb2946adeba5fa17a08c","since":6,"nonBehavioral":"GST-1371 removes the 21 skill-prose clarity judges: llm-judge.ts drops judge() and two prompt builders no remaining case calls, paid-report.ts rejects manual-review claims that only the removed cookie judge made, and eval-budgets.ts shrinks the skill-llm-eval file wall to the three kept cases, whose own JUDGE_MS timeouts are unchanged; no kept case's prompt, threshold, samples or verdict can change"} diff --git a/scripts/harness-pins/test/helpers/eval-budgets.ts.json b/scripts/harness-pins/test/helpers/eval-budgets.ts.json index 914ab65fe..dba050060 100644 --- a/scripts/harness-pins/test/helpers/eval-budgets.ts.json +++ b/scripts/harness-pins/test/helpers/eval-budgets.ts.json @@ -1 +1 @@ -{"blob":"667d0e09b20fe5d7366f57b77b5fb9b04baa37ef","since":6} +{"blob":"847a2225d7296280571377ea15ede19f24ac3a62","since":6,"nonBehavioral":"GST-1371 removes the 21 skill-prose clarity judges: llm-judge.ts drops judge() and two prompt builders no remaining case calls, paid-report.ts rejects manual-review claims that only the removed cookie judge made, and eval-budgets.ts shrinks the skill-llm-eval file wall to the three kept cases, whose own JUDGE_MS timeouts are unchanged; no kept case's prompt, threshold, samples or verdict can change"} diff --git a/scripts/harness-pins/test/helpers/llm-judge.ts.json b/scripts/harness-pins/test/helpers/llm-judge.ts.json index c33f39919..cf5c71377 100644 --- a/scripts/harness-pins/test/helpers/llm-judge.ts.json +++ b/scripts/harness-pins/test/helpers/llm-judge.ts.json @@ -1 +1 @@ -{"blob":"ff32d03f32e4af76c6c7314a79cd1b35263ed0dd","since":6,"nonBehavioral":"llm-judge.ts adds a pass-through byte counter (bytesReceived, lastByteMs) to the aborted-judge diagnostic; the request, response bytes, deadline and scoring are unchanged. GSTA-224 B2 (#3122): skill-fixture.ts follows review's renamed Step 2.5 heading so the review E2E cases extract the same section (now the skill's own reworded bytes); llm-judge.ts changes one bullet of the cross-skill consistency judge's intended architecture to name review-bot-history.md beside the older greptile-history.md name, so its calibration corpus (greptile-history inputs) stays valid. No other case's input or verdict can change; the cross-skill judge's judged files changed in this PR, so its owned inputs start its own series"} +{"blob":"b804a4585e45b0e5c84bd90b1ede0191b66283cc","since":6,"nonBehavioral":"GST-1371 removes the 21 skill-prose clarity judges: llm-judge.ts drops judge() and two prompt builders no remaining case calls, paid-report.ts rejects manual-review claims that only the removed cookie judge made, and eval-budgets.ts shrinks the skill-llm-eval file wall to the three kept cases, whose own JUDGE_MS timeouts are unchanged; no kept case's prompt, threshold, samples or verdict can change"} diff --git a/scripts/judge-calibration-configs.ts b/scripts/judge-calibration-configs.ts index 926bdd415..181da172b 100644 --- a/scripts/judge-calibration-configs.ts +++ b/scripts/judge-calibration-configs.ts @@ -6,14 +6,11 @@ */ import { ARM_JUDGE_MODEL, ARM_JUDGE_SCHEMA, buildArmJudgePrompt, parseArmJudgeResponse, - buildQaWorkflowJudgePrompt, buildQaHealthRubricJudgePrompt, buildQaAntiRefusalJudgePrompt, - buildCrossSkillConsistencyJudgePrompt, buildVoiceDirectiveJudgePrompt, - JUDGE_SCORE_SCHEMA, QA_ANTI_REFUSAL_JUDGE_SCHEMA, CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA, - VOICE_DIRECTIVE_JUDGE_SCHEMA, VOICE_DIRECTIVE_DIMENSIONS, JUDGE_SCORE_DIMENSIONS, DEFAULT_JUDGE_MAX_TOKENS, + buildQaAntiRefusalJudgePrompt, buildCrossSkillConsistencyJudgePrompt, buildVoiceDirectiveJudgePrompt, + QA_ANTI_REFUSAL_JUDGE_SCHEMA, CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA, + VOICE_DIRECTIVE_JUDGE_SCHEMA, VOICE_DIRECTIVE_DIMENSIONS, judgePanelMedian, judgePanelMajority, } from '../test/helpers/llm-judge'; -import { buildWorkflowJudgePrompt, WORKFLOW_JUDGE_RESPONSE_SCHEMA } from '../test/helpers/workflow-judge-input'; -import { COOKIE_WORKFLOW_JUDGE } from '../test/helpers/cookie-workflow-judge-input'; import { CLAUDE_JUDGE_EVAL_MODEL } from '../lib/eval-model'; import type { CalibrationConfig, Verdict } from './judge-calibration'; @@ -33,23 +30,6 @@ function requireFields(value: unknown, numbers: readonly string[], booleans: rea const meets = (median: Record, floors: Record): Verdict => Object.entries(floors).every(([key, floor]) => median[key]! >= floor) ? 'pass' : 'fail'; -const QA_FLOORS = { clarity: 3, completeness: 3, actionability: 4 }; -const scoreJudge = (floors: Record) => ({ - validate: (value: unknown) => requireFields(value, JUDGE_SCORE_DIMENSIONS), - verdict: (samples: unknown[]) => meets(judgePanelMedian(samples as Sample[], JUDGE_SCORE_DIMENSIONS), floors), -}); - -const workflow = (id: string, priority: number, cases: string[], extra: { agentCapability?: 'frontier'; model?: string; floors?: Record } = {}): CalibrationConfig => ({ - id, priority, cases, - model: extra.model ?? CLAUDE_JUDGE_EVAL_MODEL, - build: inputs => buildWorkflowJudgePrompt({ judgeContext: inputs.judgeContext!, judgeGoal: inputs.judgeGoal!, agentCapability: extra.agentCapability }, - { files: [], text: inputs.bundle! }), - oldOptions: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS }, - newOptions: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA }, - ...scoreJudge(extra.floors ?? QA_FLOORS), - estimatedOutputTokens: 1200, -}); - export const CALIBRATION_CONFIGS: CalibrationConfig[] = [ { id: 'arm', priority: 1, cases: ['arm-benchmark (skill-e2e-arm-benchmark.test.ts)'], model: ARM_JUDGE_MODEL, @@ -63,18 +43,6 @@ export const CALIBRATION_CONFIGS: CalibrationConfig[] = [ }, estimatedOutputTokens: 300, }, - { - id: 'qa-workflow', priority: 2, cases: ['qa/SKILL.md workflow'], model: CLAUDE_JUDGE_EVAL_MODEL, - build: inputs => buildQaWorkflowJudgePrompt(inputs.section!), - oldOptions: {}, newOptions: { jsonSchema: JUDGE_SCORE_SCHEMA }, - ...scoreJudge(QA_FLOORS), estimatedOutputTokens: 1200, - }, - { - id: 'qa-health-rubric', priority: 3, cases: ['qa/SKILL.md health rubric'], model: CLAUDE_JUDGE_EVAL_MODEL, - build: inputs => buildQaHealthRubricJudgePrompt(inputs.section!), - oldOptions: {}, newOptions: { jsonSchema: JUDGE_SCORE_SCHEMA }, - ...scoreJudge(QA_FLOORS), estimatedOutputTokens: 1200, - }, { id: 'qa-anti-refusal', priority: 4, cases: ['qa/SKILL.md anti-refusal'], model: CLAUDE_JUDGE_EVAL_MODEL, build: inputs => buildQaAntiRefusalJudgePrompt(inputs.diffAwareSection!, inputs.rulesSection!), @@ -102,16 +70,4 @@ export const CALIBRATION_CONFIGS: CalibrationConfig[] = [ Object.fromEntries(VOICE_DIRECTIVE_DIMENSIONS.map(key => [key, 4]))), estimatedOutputTokens: 1200, }, - workflow('workflow-default', 7, ['document-release/SKILL.md workflow', 'plan-ceo-review/SKILL.md modes', 'plan-eng-review/SKILL.md sections', - 'plan-design-review/SKILL.md passes', 'design-review/SKILL.md fix loop', 'design-consultation/SKILL.md research', - 'land-and-deploy/SKILL.md workflow', 'canary/SKILL.md monitoring loop', 'benchmark/SKILL.md perf collection', - 'setup-deploy/SKILL.md platform setup', 'sync-gbrain/SKILL.md read-only readiness', 'retro/SKILL.md instructions', - 'qa-only/SKILL.md workflow', 'gstack-upgrade/SKILL.md upgrade flow']), - { - id: 'cookie-workflow', priority: 8, cases: ['setup-browser-cookies/SKILL.md workflow'], model: COOKIE_WORKFLOW_JUDGE.model, - build: inputs => buildWorkflowJudgePrompt(COOKIE_WORKFLOW_JUDGE, { files: [], text: inputs.bundle! }), - oldOptions: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS }, - newOptions: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA }, - ...scoreJudge(COOKIE_WORKFLOW_JUDGE.thresholds), estimatedOutputTokens: 1200, - }, ]; diff --git a/scripts/judge-controls.ts b/scripts/judge-controls.ts deleted file mode 100644 index 9c4db9647..000000000 --- a/scripts/judge-controls.ts +++ /dev/null @@ -1,123 +0,0 @@ -#!/usr/bin/env -S bun --no-env-file --config=/dev/null -/** - * C1 workflow-judge control bundles (pre-registered in docs/evals/judge-controls-2026-10.md). - * - * For each judge that produced low samples, builds the production bundle with the - * eval's own reader and prompt builder, derives a negative and a targeted-defect - * bundle, and runs one 3-sample panel per bundle with the eval's request options. - * - * Usage: - * bun run scripts/judge-controls.ts --list (builds every bundle, no API calls) - * bun run scripts/judge-controls.ts --run (36 judge calls; writes docs/evals/judge-controls/results.json) - */ -import * as fs from 'node:fs'; -import * as path from 'node:path'; -import { buildQaWorkflowJudgePrompt, callJudge, judgePanelMean, judgePanelMedian, DEFAULT_JUDGE_MAX_TOKENS, JUDGE_SCORE_DIMENSIONS, type CallJudgeOptions } from '../test/helpers/llm-judge'; -import { buildWorkflowJudgePrompt, readWorkflowJudgeInput, QA_DISCOVERY_REFERENCES, WORKFLOW_JUDGE_RESPONSE_SCHEMA } from '../test/helpers/workflow-judge-input'; - -const ROOT = path.resolve(import.meta.dir, '..'); -const FLOORS = { clarity: 3, completeness: 3, actionability: 4 } as const; -const SAMPLES = 3; - -type Judge = { - name: string; - read: () => string; - prompt: (text: string) => string; - options: CallJudgeOptions; - defect: (text: string) => string; -}; - -function cut(text: string, start: string, end: string): string { - const from = text.indexOf(start); - const to = text.indexOf(end, from + start.length); - if (from < 0 || to < 0) throw new Error(`defect anchor missing: ${start} … ${end}`); - return text.slice(0, from) + text.slice(to); -} - -function insertAfter(text: string, anchor: string, line: string): string { - const at = text.indexOf(anchor); - if (at < 0) throw new Error(`defect anchor missing: ${anchor}`); - return text.slice(0, at + anchor.length) + '\n\n' + line + text.slice(at + anchor.length); -} - -const workflow = (skillPath: string, startMarker: string, endMarker: string | null, references: readonly string[] | undefined) => - () => readWorkflowJudgeInput({ root: ROOT, skillPath, startMarker, endMarker, references }).text; - -const JUDGES: Judge[] = [ - { - name: 'qa/SKILL.md workflow', - read: workflow('qa/SKILL.md', '# /qa: Test', null, - ['qa/templates/functional-report-template.md', 'qa/templates/qa-report-template.md', 'qa/references/issue-taxonomy.md']), - prompt: buildQaWorkflowJudgePrompt, - options: {}, - defect: text => cut(text, '## Phase 10: Report', '## Phase 11:'), - }, - { - name: 'qa-only/SKILL.md workflow', - read: workflow('qa-only/SKILL.md', '# /qa-only:', null, QA_DISCOVERY_REFERENCES.filter(file => file !== 'qa/sections/exploratory.md')), - prompt: text => buildWorkflowJudgePrompt({ judgeContext: 'a report-only QA testing workflow', - judgeGoal: 'how to select browser/native functional/mixed targets, explore safely with repository tools, report exact contract evidence and coverage limits, conditionally load browser/DX instructions and never mutate product/tests/Git through any tool' }, { files: [], text }), - options: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA }, - defect: text => insertAfter(text, '# /qa-only: Report-Only QA Testing', 'After reporting each bug, fix it in the source and commit the fix.'), - }, - { - name: 'review/SKILL.md workflow', - read: workflow('review/SKILL.md', '## Step 0: Detect platform and base branch', null, - [...QA_DISCOVERY_REFERENCES, 'review/checklist.md', 'review/specialists/testing.md']), - prompt: text => buildWorkflowJudgePrompt({ judgeContext: 'a pre-landing review with bounded exploratory QA', - judgeGoal: 'how to review and explore changed behavior even for small diffs without a plan or server, preserve report-only discovery and the test_stub ASK gate, handle incomplete probes honestly, and rerun affected evidence after approved repairs', - agentCapability: 'frontier' }, { files: [], text }), - options: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS }, - defect: text => cut(text, '## Step 3: Get the diff', '## Step 3.4'), - }, - { - name: 'setup-deploy/SKILL.md platform setup', - read: workflow('setup-deploy/SKILL.md', '### Step 2: Detect platform', '## Important Rules', undefined), - prompt: text => buildWorkflowJudgePrompt({ judgeContext: 'a deployment configuration setup workflow that detects deploy platforms and writes config to CLAUDE.md', - judgeGoal: 'how to detect deploy platforms (Fly.io, Render, Vercel, Netlify, Heroku, GitHub Actions, custom), gather platform-specific configuration (URLs, status commands, health checks, custom hooks), and persist everything to CLAUDE.md for future automated use' }, { files: [], text }), - options: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA }, - defect: text => { - const step = text.indexOf('### Step 2: Detect platform'); - const open = text.indexOf('```bash', step); - const close = text.indexOf('```', open + 7); - if (step < 0 || open < 0 || close < 0) throw new Error('defect anchor missing: Step 2 bash block'); - return text.slice(0, open) + text.slice(close + 3); - }, - }, -]; - -/** Keep only file boundaries and headings. */ -const negative = (text: string) => text.split('\n') - .filter(line => /^--- (BEGIN|END) FILE /.test(line) || /^#{1,6} /.test(line)).join('\n'); - -function bundles(judge: Judge): Record<'positive' | 'negative' | 'defect', string> { - const positive = judge.read(); - return { positive, negative: negative(positive), defect: judge.defect(positive) }; -} - -const mode = process.argv[2]; -if (mode !== '--list' && mode !== '--run') { - console.error('usage: bun run scripts/judge-controls.ts --list | --run'); - process.exit(2); -} -if (mode === '--list') { - for (const judge of JUDGES) { - const built = bundles(judge); - console.log(judge.name, Object.fromEntries(Object.entries(built).map(([k, v]) => [k, v.length]))); - } - process.exit(0); -} - -type Sample = { clarity: number; completeness: number; actionability: number; reasoning: string }; -const results = await Promise.all(JUDGES.flatMap(judge => Object.entries(bundles(judge)).map(async ([bundle, text]) => { - const prompt = judge.prompt(text); - const samples = await Promise.all(Array.from({ length: SAMPLES }, () => callJudge(prompt, undefined, judge.options))); - const mean = judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS); - const median = judgePanelMedian(samples, JUDGE_SCORE_DIMENSIONS); - const verdict = Object.entries(FLOORS).every(([key, floor]) => median[key as keyof typeof FLOORS] >= floor) ? 'pass' : 'fail'; - return { judge: judge.name, bundle, verdict, median, mean, samples }; -}))); -const out = path.join(ROOT, 'docs/evals/judge-controls/results.json'); -fs.mkdirSync(path.dirname(out), { recursive: true }); -fs.writeFileSync(out, JSON.stringify({ recorded_at: new Date().toISOString(), results }, null, 2) + '\n'); -for (const r of results) console.log(`${r.judge} | ${r.bundle} | ${r.verdict} | median ${JSON.stringify(r.median)} mean ${JSON.stringify(r.mean)}`); diff --git a/scripts/lib/paid-report.ts b/scripts/lib/paid-report.ts index e0d18da4e..31355b5a9 100644 --- a/scripts/lib/paid-report.ts +++ b/scripts/lib/paid-report.ts @@ -14,7 +14,6 @@ import { import { readSessionLedger, type SessionLedgerRow } from '../../test/helpers/session-ledger'; import { publishedFence, sanitizeFixedFenceLine } from './published-text'; import { E2E_KINDS } from '../../test/helpers/touchfiles-data'; -import { manualReviewProblem } from '../../test/helpers/cookie-workflow-manual-review'; import { writeNegativeReceipt, writePanelReceipt } from '../e2e-shard-reuse'; import { E2E_TOUCHFILES, E2E_TIERS, LLM_JUDGE_TOUCHFILES } from '../../test/helpers/touchfiles'; import { type EvalHost, hostSkipReason } from '../../test/helpers/e2e-hosts'; @@ -617,7 +616,6 @@ export function runPaidReport(reportDir: string, options: { writeDurations?: boo flaky: number; total: number; executed: number; reused: number; passed: number; failed: number; manual_accepted: number; attempts: number }> = []; const manualProblems: string[] = []; - const manualClaims = new Map(); const recordsByShard = new Map(); let costUsd = 0; let flakyCollectorFailures = 0; @@ -638,26 +636,15 @@ export function runPaidReport(reportDir: string, options: { writeDurations?: boo if (shard) recordsByShard.set(shard.file, [...(recordsByShard.get(shard.file) ?? []), ...parsed.tests]); // Trial records are verdict input for panelVerdict(), never collector gates. if (shard?.trial) continue; - const seen = new Map(); for (const [index, entry] of parsed.tests.entries()) { if (!entry || typeof entry !== 'object' || typeof entry.name !== 'string' || !entry.name || typeof entry.passed !== 'boolean') { manualProblems.push(`${rel}: attempt ${index + 1}: malformed collector entry (name/passed required)`); continue; } - const key = `${entry.suite ?? ''}\0${entry.name}`; - const occurrence = (seen.get(key) ?? 0) + 1; - seen.set(key, occurrence); - if (Object.hasOwn(entry, 'manual_review') && occurrence !== 1) { - manualProblems.push(`${rel}: attempt ${index + 1}: manual review is only valid on the first case attempt`); - } if (Object.hasOwn(entry, 'manual_review')) { - const previous = manualClaims.get(key); - if (previous && previous !== rel) manualProblems.push(`${rel}: duplicate manual-review claim for ${entry.name} (also in ${previous})`); - else manualClaims.set(key, rel); + manualProblems.push(`${rel}: attempt ${index + 1}: manual-review claims were retired with the workflow judges`); } - const problem = manualReviewProblem(entry, rootDir); - if (problem) manualProblems.push(`${rel}: attempt ${index + 1}: ${problem}`); } collectors.push(parsed); const counts = collectorOutcomeCounts([parsed]); diff --git a/scripts/typecheck-test-baseline.json b/scripts/typecheck-test-baseline.json index 16f3bdea4..6835d28e4 100644 --- a/scripts/typecheck-test-baseline.json +++ b/scripts/typecheck-test-baseline.json @@ -120,7 +120,6 @@ "test/claude-code-runner.test.ts\tTS7006\tParameter 'callback' implicitly has an 'any' type.": 1, "test/claude-code-runner.test.ts\tTS7006\tParameter 'delay' implicitly has an 'any' type.": 1, "test/claude-code-runner.test.ts\tTS7019\tRest parameter 'args' implicitly has an 'any[]' type.": 1, - "test/cookie-workflow-manual-review.test.ts\tTS2769\tNo overload matches this call. The last overload gave the following error. Argument of type 'ManualJudgeReview | undefined' is not assignable to parameter of type 'ManualJudgeReview | null'. Type 'undefined' is not assignable to type 'ManualJudgeReview | null'.": 1, "test/cso-eval.test.ts\tTS2345\tArgument of type '(command: string, args: readonly string[] | undefined, options: any) => string' is not assignable to parameter of type '{ (file: string): NonSharedBuffer; (file: string, options: ExecFileSyncOptionsWithStringEncoding): string; (file: string, options: ExecFileSyncOptionsWithBufferEncoding): NonSharedBuffer; (file: string, options?: ExecFileSyncOptions | undefined): string | NonSharedBuffer; (file: string, args: readonly string[]): Non...'. Target signature provides too few arguments. Expected 3 or more, but got 1.": 1, "test/cso-eval.test.ts\tTS2790\tThe operand of a 'delete' operator must be optional.": 1, "test/cso-eval.test.ts\tTS7005\tVariable 'receipts' implicitly has an 'any[]' type.": 1, diff --git a/test/bun-entry-isolation.test.ts b/test/bun-entry-isolation.test.ts index 5b53d30e9..da0bbe85b 100644 --- a/test/bun-entry-isolation.test.ts +++ b/test/bun-entry-isolation.test.ts @@ -111,7 +111,6 @@ describe('GSTA-56: gstack never takes configuration from the audited repository' 'scripts/eval-compare.ts', 'scripts/eval-summary.ts', 'scripts/garry-output-comparison.ts', - 'scripts/judge-controls.ts', 'scripts/measure-journal-read.ts', ]; expect(scripts.filter(f => firstLine(f) !== ISOLATED_SHEBANG)).toEqual([]); diff --git a/test/ci-paid-coordination.test.ts b/test/ci-paid-coordination.test.ts index 0bb517205..394e0764d 100644 --- a/test/ci-paid-coordination.test.ts +++ b/test/ci-paid-coordination.test.ts @@ -5,7 +5,7 @@ import * as path from 'node:path'; import { spawnSync } from 'node:child_process'; import { buildRunManifest, collectPaidTestFiles, shardCaseId, shardTrial, type PaidRunManifest, type SliceResult } from '../scripts/test-paid-shards'; import { STRICT_RETRY_CASE_BUDGETS } from './helpers/eval-budgets'; -import { approvedCookieWorkflowSource, manualReviewFixture } from './helpers/manual-judge-review-fixture'; +import { manualReviewFixture } from './helpers/manual-judge-review-fixture'; const ROOT = path.resolve(import.meta.dir, '..'); type Step = { uses?: string; run?: string; if?: string; with?: Record }; @@ -206,11 +206,6 @@ describe('dependency-free CI planner and report execution', () => { .toBe(fs.readFileSync(path.join(ROOT, 'test/helpers/llm-judge.ts'), 'utf8')); fs.cpSync(path.join(ROOT, 'lib'), path.join(fixture, 'lib'), sourceOnly); expect(fs.existsSync(path.join(fixture, 'lib/diagram-render/node_modules'))).toBe(false); - fs.mkdirSync(path.join(fixture, '.github'), { recursive: true }); - for (const file of ['.github/cookie-workflow-manual-review.json', 'setup-browser-cookies/SKILL.md', 'BROWSER.md']) { - fs.mkdirSync(path.dirname(path.join(fixture, file)), { recursive: true }); - fs.copyFileSync(path.join(ROOT, file), path.join(fixture, file)); - } for (const file of collectPaidTestFiles()) { fs.copyFileSync(path.join(ROOT, file), path.join(fixture, file)); } @@ -312,12 +307,7 @@ describe('dependency-free CI planner and report execution', () => { }); } - test('report verifies every manual claim against current source, preserves attempts, and never masks a failed shard', () => { - const skillPath = path.join(fixture, 'setup-browser-cookies/SKILL.md'); - const currentSkill = fs.readFileSync(skillPath, 'utf8'); - fs.writeFileSync(skillPath, approvedCookieWorkflowSource(currentSkill)); - const approvedBrowserPath = path.join(fixture, 'BROWSER.md'); - fs.writeFileSync(approvedBrowserPath, approvedCookieWorkflowSource(fs.readFileSync(approvedBrowserPath, 'utf8'))); + test('report rejects every manual-review claim, preserves attempts, and never masks a failed shard', () => { const reportDir = path.join(fixture, 'manual-report'); const manifestPath = path.join(reportDir, 'manifest.json'); const planned = run(['--emit-plan', manifestPath, '--slices', '1'], 'gate'); @@ -334,89 +324,53 @@ describe('dependency-free CI planner and report execution', () => { const slicePath = path.join(reportDir, 'slice-1.json'); const collectorPath = path.join(reportDir, 'judge-results.json'); const summaryPath = path.join(reportDir, 'collector-outcomes.json'); - const receipt = manualReviewFixture(fixture); + const receipt = manualReviewFixture(); + const executed = { name: 'executed', suite: 'other', passed: true, execution: 'executed' }; + const reused = { name: 'automated', suite: 'other', passed: true, execution: 'reused' }; const write = (tests: unknown[]) => fs.writeFileSync(collectorPath, JSON.stringify({ total_tests: tests.length, tier: 'llm-judge', shard: 1, total_cost_usd: 0, - tests, flaky_retries: [{ name: receipt.name, attempts: tests.length }], + tests, flaky_retries: [{ name: executed.name, attempts: tests.length }], })); fs.writeFileSync(slicePath, JSON.stringify(slice)); - write([{ ...receipt, passed: true }, receipt]); - const historical = run(['--report', reportDir], 'gate'); - expect(historical.status).toBe(1); - expect(historical.stderr).toContain('attempt 1: Malformed manual-review claim'); + write([receipt, reused]); + const retired = run(['--report', reportDir], 'gate'); + expect(retired.status).toBe(1); + expect(retired.stderr).toContain('attempt 1: manual-review claims were retired with the workflow judges'); expect(fs.existsSync(summaryPath)).toBe(false); - write([{ ...receipt, manual_review: { ...receipt.manual_review, refusal: { - ...receipt.manual_review!.refusal, response_id: '', - } } }, receipt]); - const malformed = run(['--report', reportDir], 'gate'); - expect(malformed.status).toBe(1); - expect(malformed.stderr).toContain('attempt 1: Malformed manual-review claim'); - - write([{ ...receipt, manual_review: undefined, passed: false }, receipt]); - const forgedFirstAttempt = run(['--report', reportDir], 'gate'); - expect(forgedFirstAttempt.status).toBe(1); - expect(forgedFirstAttempt.stderr).toContain('attempt 2: manual review is only valid on the first case attempt'); - expect(fs.existsSync(summaryPath)).toBe(false); - write([receipt, { ...receipt, attempt: 2 }]); - const retriedManual = run(['--report', reportDir], 'gate'); - expect(retriedManual.status).toBe(1); - expect(retriedManual.stderr).toContain('attempt 2: Malformed manual-review claim'); + write([executed, { ...receipt, passed: true }]); + const forgedPass = run(['--report', reportDir], 'gate'); + expect(forgedPass.status).toBe(1); + expect(forgedPass.stderr).toContain('attempt 2: manual-review claims were retired with the workflow judges'); expect(fs.existsSync(summaryPath)).toBe(false); - write([receipt]); - const secondCollector = path.join(reportDir, 'other-results.json'); - fs.writeFileSync(secondCollector, JSON.stringify({ total_tests: 1, tests: [receipt] })); - const duplicateCollector = run(['--report', reportDir], 'gate'); - expect(duplicateCollector.status).toBe(1); - expect(duplicateCollector.stderr).toContain('duplicate manual-review claim'); - expect(fs.existsSync(summaryPath)).toBe(false); - fs.rmSync(secondCollector); - - write([receipt, { name: 'automated', suite: 'other', passed: true, execution: 'reused' }]); + write([executed, reused]); const clean = run(['--report', reportDir], 'gate'); expect(clean.status, clean.stderr).toBe(0); - expect(clean.stdout).toContain('1 passed, 0 failed, 1 manual accepted (unscored; no score-cache credit) (2 attempt records'); + expect(clean.stdout).toContain('2 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (2 attempt records'); const summary = JSON.parse(fs.readFileSync(summaryPath, 'utf8')); - expect(summary.totals).toEqual({ executed: 1, reused: 1, passed: 1, failed: 0, - manual_accepted: 1, attempts: 2, total: 2, flaky: 1 }); - expect(summary.files[0]).toMatchObject({ file: 'judge-results.json', total: 2, manual_accepted: 1, passed: 1 }); - expect(JSON.parse(fs.readFileSync(collectorPath, 'utf8')).tests[0]).toEqual(receipt); + expect(summary.totals).toEqual({ executed: 1, reused: 1, passed: 2, failed: 0, + manual_accepted: 0, attempts: 2, total: 2, flaky: 1 }); + expect(summary.files[0]).toMatchObject({ file: 'judge-results.json', total: 2, manual_accepted: 0, passed: 2 }); - fs.writeFileSync(skillPath, currentSkill); - const obsoleteApproval = run(['--report', reportDir], 'gate'); - expect(obsoleteApproval.status).toBe(1); - expect(obsoleteApproval.stderr).toContain('does not match current source and approval'); - fs.writeFileSync(skillPath, approvedCookieWorkflowSource(currentSkill)); - - const browserPath = path.join(fixture, 'BROWSER.md'); - const browserSource = fs.readFileSync(browserPath, 'utf8'); - fs.writeFileSync(browserPath, browserSource.replace('#### Choosing a source and checking sign-in', - '#### Choosing a source and checking sign-in\nchanged approved source')); - const sourceDrift = run(['--report', reportDir], 'gate'); - expect(sourceDrift.status).toBe(1); - expect(sourceDrift.stderr).toContain('does not match current source and approval'); - expect(fs.existsSync(summaryPath)).toBe(false); - fs.writeFileSync(browserPath, browserSource); - - write([receipt, { name: 'unapproved', passed: false, execution: 'executed' }]); + write([executed, { name: 'unapproved', passed: false, execution: 'executed' }]); const failedCollector = run(['--report', reportDir], 'gate'); expect(failedCollector.status).toBe(1); expect(failedCollector.stderr).toContain('1 unapproved final collector failure(s)'); - expect(JSON.parse(fs.readFileSync(summaryPath, 'utf8')).totals).toMatchObject({ failed: 1, manual_accepted: 1 }); + expect(JSON.parse(fs.readFileSync(summaryPath, 'utf8')).totals).toMatchObject({ failed: 1, passed: 1 }); - write([receipt, { passed: true }]); + write([executed, { passed: true }]); const missingName = run(['--report', reportDir], 'gate'); expect(missingName.status).toBe(1); expect(missingName.stderr).toContain('attempt 2: malformed collector entry (name/passed required)'); expect(fs.existsSync(summaryPath)).toBe(false); - write([receipt, { name: 'malformed', passed: 'true' }]); + write([executed, { name: 'malformed', passed: 'true' }]); const malformedPassed = run(['--report', reportDir], 'gate'); expect(malformedPassed.status).toBe(1); expect(malformedPassed.stderr).toContain('attempt 2: malformed collector entry (name/passed required)'); expect(fs.existsSync(summaryPath)).toBe(false); - write([receipt, { name: 'automated', suite: 'other', passed: true, execution: 'reused' }]); + write([executed, reused]); slice.outcomes[0].status = 'failed'; slice.outcomes[0].exitCode = 1; @@ -424,14 +378,6 @@ describe('dependency-free CI planner and report execution', () => { const failedShard = run(['--report', reportDir], 'gate'); expect(failedShard.status).toBe(1); expect(failedShard.stderr).toContain(`${slice.outcomes[0].files[0]}: failed`); - expect(JSON.parse(fs.readFileSync(summaryPath, 'utf8')).totals.manual_accepted).toBe(1); - - const stale = structuredClone(receipt); - stale.manual_review!.approval.prompt_sha256 = '0'.repeat(64); - write([stale]); - const mismatched = run(['--report', reportDir], 'gate'); - expect(mismatched.status).toBe(1); - expect(mismatched.stderr).toContain('attempt 1: Malformed manual-review claim'); - expect(fs.existsSync(summaryPath)).toBe(false); + expect(JSON.parse(fs.readFileSync(summaryPath, 'utf8')).totals.passed).toBe(2); }); }); diff --git a/test/cookie-workflow-judge-input.test.ts b/test/cookie-workflow-judge-input.test.ts deleted file mode 100644 index c79a67918..000000000 --- a/test/cookie-workflow-judge-input.test.ts +++ /dev/null @@ -1,382 +0,0 @@ -import { afterEach, describe, expect, spyOn, test } from 'bun:test'; -import { createHash } from 'node:crypto'; -import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import { join, resolve } from 'node:path'; -import { buildCookieWorkflowJudgeInput, COOKIE_WORKFLOW_JUDGE } from './helpers/cookie-workflow-judge-input'; -import { buildWorkflowJudgePrompt, readWorkflowJudgeInput, WORKFLOW_JUDGE_RESPONSE_SCHEMA } from './helpers/workflow-judge-input'; -import { prepareWorkflowJudgeCache, validWorkflowJudgeScore, type WorkflowCacheOptions } from './helpers/workflow-judge-cache'; -import type { EvalTestEntry } from './helpers/eval-store'; -import { selectTests } from './helpers/test-selection'; -import { E2E_TOUCHFILES, LLM_JUDGE_TOUCHFILES } from './helpers/touchfiles-data'; -import { selectPrProfile } from '../scripts/test-pr-profile'; -import { JUDGE_MS } from './helpers/eval-budgets'; -import { Messages } from '@anthropic-ai/sdk/resources/messages'; -import { callJudge, type CallJudgeOptions, JudgeRefusalError, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMean, judgePanelMedian, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, JUDGE_PANEL_SAMPLES } from './helpers/llm-judge'; -import { COOKIE_MANUAL_REVIEW_FILE, getCookieWorkflowManualReview, isManualReviewEntry } from './helpers/cookie-workflow-manual-review'; - -const ROOT = resolve(import.meta.dir, '..'); -const NAME = 'setup-browser-cookies/SKILL.md workflow'; -const scratch: string[] = []; -const skill = 'excluded preamble\n# Setup Browser Cookies\nRead the gstack root\'s `BROWSER.md`, **Choosing a source and checking sign-in**.\nExact body.\n'; -const browser = 'excluded browser introduction\n#### Choosing a source and checking sign-in\nExact reference.\n\n### Tabs + frames\nexcluded tab commands\n'; - -function fixture(entry: string | null = skill, reference: string | null = browser): string { - const root = mkdtempSync(join(tmpdir(), 'gstack-cookie-judge-')); - scratch.push(root); - mkdirSync(join(root, 'setup-browser-cookies')); - if (entry !== null) writeFileSync(join(root, 'setup-browser-cookies/SKILL.md'), entry); - if (reference !== null) writeFileSync(join(root, 'BROWSER.md'), reference); - return root; -} - -function approveFixture(root: string) { - const input = buildCookieWorkflowJudgeInput(root); - const approval = { ...JSON.parse(readFileSync(join(ROOT, COOKIE_MANUAL_REVIEW_FILE), 'utf8')), - prompt_sha256: input.sha256, prompt_bytes: Buffer.byteLength(input.prompt), model: COOKIE_WORKFLOW_JUDGE.model, - reason: 'Synthetic approval fixture, not live review evidence' }; - mkdirSync(join(root, '.github'), { recursive: true }); - writeFileSync(join(root, COOKIE_MANUAL_REVIEW_FILE), JSON.stringify(approval)); - return approval; -} - -const refusal = () => new JudgeRefusalError({ id: 'msg_synthetic', _request_id: 'req_synthetic', - model: COOKIE_WORKFLOW_JUDGE.model, usage: { input_tokens: 1, output_tokens: 0 }, content: [] }); - -afterEach(() => { - for (const root of scratch.splice(0)) rmSync(root, { recursive: true, force: true }); -}); - -type FixtureScore = { clarity: number; completeness: number; actionability: number; reasoning: string }; -const passingScore: FixtureScore = { ...COOKIE_WORKFLOW_JUDGE.thresholds, reasoning: 'Synthetic fixture score' }; - -function actualCookieCallback(root: string, overrides: { - judge?: () => Promise; - callJudge?: (prompt: string, model: string | undefined, options: CallJudgeOptions) => Promise; - clock?: () => number; - budget?: number; - setTimer?: typeof setTimeout; - clearTimer?: typeof clearTimeout; -} = {}) { - const source = readFileSync(join(ROOT, 'test/skill-llm-eval.test.ts'), 'utf8'); - const managedStart = source.indexOf('async function runWorkflowJudge'); - const managedEnd = source.indexOf('// Block 1:', managedStart); - const start = source.indexOf("describeIfSelected('Cookie setup workflow quality'"); - const end = source.indexOf('// Module-level afterAll', start); - expect(managedStart).toBeGreaterThan(-1); - expect(managedEnd).toBeGreaterThan(managedStart); - expect(start).toBeGreaterThan(-1); - expect(end).toBeGreaterThan(start); - const registration = new Bun.Transpiler({ loader: 'ts' }).transformSync(source.slice(managedStart, managedEnd) + source.slice(start, end)); - const requests: Array<{ prompt: string; model: string | undefined; signal: AbortSignal }> = []; - const records: EvalTestEntry[] = []; - const attempts = new Map(); - let callback: () => Promise = async () => { throw new Error('Judge callback was not registered'); }; - new Function('describeIfSelected', 'testIfSelected', 'ROOT', 'buildCookieWorkflowJudgeInput', 'resolveEvalModel', 'callJudge', 'COOKIE_WORKFLOW_JUDGE', 'JUDGE_MS', 'WORKFLOW_JUDGE_TEST_MS', 'WORKFLOW_JUDGE_RECORD_MS', 'evalCollector', 'expect', 'console', 'readWorkflowJudgeInput', 'buildWorkflowJudgePrompt', 'prepareWorkflowJudgeCache', 'workflowJudgeAttempts', 'performance', 'setTimeout', 'clearTimeout', 'JudgeRefusalError', 'getCookieWorkflowManualReview', 'DEFAULT_JUDGE_MAX_TOKENS', 'WORKFLOW_JUDGE_RESPONSE_SCHEMA', 'validWorkflowJudgeScore', 'judgePanel', 'judgePanelMean', 'judgePanelMedian', 'judgePanelReasoning', 'JUDGE_SCORE_DIMENSIONS', registration)( - (_suite: string, names: string[], run: () => void) => { expect(names).toEqual([NAME]); run(); }, - (name: string, run: () => Promise, budget: number) => { expect(name).toBe(NAME); expect(budget).toBe(JUDGE_MS + 10_000); callback = run; }, - root, buildCookieWorkflowJudgeInput, (_kind: string, explicit?: string) => explicit ?? 'fixture-model', - async (prompt: string, model: string | undefined, options: CallJudgeOptions & { signal: AbortSignal }) => { - requests.push({ prompt, model, signal: options.signal }); - if (overrides.callJudge) return overrides.callJudge(prompt, model, options); - return overrides.judge ? overrides.judge() : passingScore; - }, - COOKIE_WORKFLOW_JUDGE, overrides.budget ?? JUDGE_MS, JUDGE_MS + 10_000, 5_000, - { addTest: (record: EvalTestEntry) => records.push(record) }, expect, { log() {} }, - readWorkflowJudgeInput, buildWorkflowJudgePrompt, - (options: WorkflowCacheOptions) => prepareWorkflowJudgeCache({ ...options, env: { - EVALS_CACHE_DIR: join(root, 'cache'), EVALS_CACHE_REPOSITORY: 'fixture/cookie', EVALS_CACHE_PR: '1', - EVALS_CACHE_RUNTIME_ID: 'a'.repeat(64), EVALS_TIER: 'gate', EVALS_RUN_ID: 'cookie-fixture', - } }), - attempts, overrides.clock ? { now: overrides.clock } : performance, - overrides.setTimer ?? setTimeout, overrides.clearTimer ?? clearTimeout, - JudgeRefusalError, getCookieWorkflowManualReview, DEFAULT_JUDGE_MAX_TOKENS, - WORKFLOW_JUDGE_RESPONSE_SCHEMA, validWorkflowJudgeScore, judgePanel, judgePanelMean, judgePanelMedian, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, - ); - return { run: () => callback(), requests, records, attempts }; -} - -describe('cookie workflow judge input', () => { - test('the registered callback accepts only an approved empty provider refusal, with no score or cache credit', async () => { - const root = fixture(); - approveFixture(root); - const h = actualCookieCallback(root, { judge: async () => { throw refusal(); } }); - await h.run(); - expect(h.requests).toHaveLength(JUDGE_PANEL_SAMPLES); - expect(h.records).toHaveLength(1); - expect(h.records[0]).toMatchObject({ passed: false, execution: 'executed', exit_reason: 'provider_refusal' }); - expect(isManualReviewEntry(h.records[0])).toBe(true); - expect(h.records[0]).not.toHaveProperty('judge_scores'); - expect(h.records[0]).not.toHaveProperty('judge_reasoning'); - expect(h.records[0]).not.toHaveProperty('reused_from'); - expect(existsSync(join(root, 'cache'))).toBe(false); - }); - - test('an approval cannot turn low scores, malformed output, or a refusal-named ordinary error into acceptance', async () => { - const root = fixture(); approveFixture(root); - for (const result of [ - { ...passingScore, clarity: 1 }, - new SyntaxError('Malformed provider JSON'), - Object.assign(new Error('Synthetic refusal text'), { name: 'JudgeRefusalError' }), - ]) { - const h = actualCookieCallback(root, { judge: async () => { if (result instanceof Error) throw result; return result; } }); - await expect(h.run()).rejects.toThrow(); - expect(h.records).toHaveLength(1); - expect(h.records[0].passed).toBe(false); - expect(h.records[0]).not.toHaveProperty('manual_review'); - expect(isManualReviewEntry(h.records[0])).toBe(false); - } - const scored = actualCookieCallback(root); - await scored.run(); - expect(scored.records[0]).toMatchObject({ passed: true, judge_scores: COOKIE_WORKFLOW_JUDGE.thresholds }); - expect(scored.records[0]).not.toHaveProperty('manual_review'); - }); - - test.each(['prompt', 'model', 'budget', 'thresholds', 'malformed', 'missing', 'evidence', 'response-evidence'])( - 'the actual callback rejects an unapproved refusal: %s', async mismatch => { - const root = fixture(); const approval = approveFixture(root); - const error = refusal(); - if (mismatch === 'prompt') writeFileSync(join(root, 'BROWSER.md'), browser.replace('Exact reference.', 'Changed reference.')); - if (mismatch === 'model') approval.model = 'different-model'; - if (mismatch === 'budget') approval.max_tokens++; - if (mismatch === 'thresholds') approval.thresholds.clarity++; - if (mismatch === 'evidence') error.refusal.request_id = null; - if (mismatch === 'response-evidence') error.refusal.response_id = null; - writeFileSync(join(root, COOKIE_MANUAL_REVIEW_FILE), mismatch === 'malformed' ? '{}' : JSON.stringify(approval)); - if (mismatch === 'missing') rmSync(join(root, COOKIE_MANUAL_REVIEW_FILE)); - const h = actualCookieCallback(root, { judge: async () => { throw error; } }); - await expect(h.run()).rejects.toThrow(); - expect(h.records).toHaveLength(1); - expect(h.records[0].passed).toBe(false); - expect(h.records[0]).not.toHaveProperty('manual_review'); - expect(existsSync(join(root, 'cache'))).toBe(false); - }); - - test('a late refusal stays a timeout despite exact approval', async () => { - const root = fixture(); approveFixture(root); - let now = 0; - const h = actualCookieCallback(root, { budget: 20, clock: () => now, - judge: async () => { now = 20; throw refusal(); } }); - await expect(h.run()).rejects.toThrow('deadline'); - expect(h.records).toHaveLength(1); - expect(h.records[0]).toMatchObject({ passed: false, exit_reason: 'timeout' }); - expect(h.records[0]).not.toHaveProperty('manual_review'); - }); - - test('a retry refusal cannot erase an earlier scored failure with manual acceptance', async () => { - const root = fixture(); approveFixture(root); - let calls = 0; - const h = actualCookieCallback(root, { judge: async () => { - if (++calls <= JUDGE_PANEL_SAMPLES) return { ...passingScore, clarity: 1 }; - throw refusal(); - } }); - await expect(h.run()).rejects.toThrow(); - await expect(h.run()).rejects.toThrow('provider refused'); - expect(h.records.map(record => [record.attempt, record.passed, record.exit_reason])) - .toEqual([[1, false, 'validation_failed'], [2, false, 'provider_refusal']]); - for (const record of h.records) expect(record).not.toHaveProperty('manual_review'); - }); - - test('preserves exact source bytes, line ranges, and the immutable rubric', () => { - const input = buildCookieWorkflowJudgeInput(fixture()); - expect(input.files).toEqual([ - { path: 'setup-browser-cookies/SKILL.md', kind: 'entrypoint', startLine: 2, endLine: 4, content: skill.slice(skill.indexOf('# Setup')) }, - { path: 'BROWSER.md', kind: 'section', startLine: 2, endLine: 4, content: browser.slice(browser.indexOf('#### Choosing'), browser.indexOf('### Tabs')) }, - ]); - for (const file of input.files) { - expect(input.text).toContain(`--- BEGIN FILE ${JSON.stringify(file.path)} (lines ${file.startLine}-${file.endLine}; ${file.kind}) ---\n${file.content}\n--- END FILE ${JSON.stringify(file.path)} ---`); - expect(input.text.split(file.content)).toHaveLength(2); - } - expect(input.prompt).toBe(buildWorkflowJudgePrompt(COOKIE_WORKFLOW_JUDGE, input)); - expect(input.sha256).toBe(createHash('sha256').update(input.prompt).digest('hex')); - expect(input.prompt).not.toContain('excluded'); - expect(COOKIE_WORKFLOW_JUDGE.thresholds).toEqual({ clarity: 4, completeness: 3, actionability: 4 }); - }); - - test('preserves CRLF instead of rewriting excerpts', () => { - const input = buildCookieWorkflowJudgeInput(fixture(skill.replaceAll('\n', '\r\n'), browser.replaceAll('\n', '\r\n'))); - expect(input.files[0].content).toBe(skill.slice(skill.indexOf('# Setup')).replaceAll('\n', '\r\n')); - expect(input.files[1].content).toBe(browser.slice(browser.indexOf('#### Choosing'), browser.indexOf('### Tabs')).replaceAll('\n', '\r\n')); - expect(input.files.map(file => [file.startLine, file.endLine])).toEqual([[2, 4], [2, 4]]); - }); - - test('fingerprints consumed bytes, not excluded source text', () => { - const original = buildCookieWorkflowJudgeInput(fixture()); - expect(buildCookieWorkflowJudgeInput(fixture(skill.replace('Exact body.', 'Changed body.'))).sha256).not.toBe(original.sha256); - expect(buildCookieWorkflowJudgeInput(fixture(skill, browser.replace('Exact reference.', 'Changed reference.'))).sha256).not.toBe(original.sha256); - expect(buildCookieWorkflowJudgeInput(fixture(skill.replace('excluded preamble', 'different preamble'), browser.replace('excluded tab commands', 'different tab commands'))).sha256).toBe(original.sha256); - }); - - test('reads the actual generated workflow and only its referenced cookie section', () => { - const input = buildCookieWorkflowJudgeInput(ROOT); - const entry = readFileSync(join(ROOT, 'setup-browser-cookies/SKILL.md'), 'utf8'); - const reference = readFileSync(join(ROOT, 'BROWSER.md'), 'utf8'); - expect(input.files[0].content).toBe(entry.slice(entry.indexOf('# Setup Browser Cookies'))); - expect(input.files[1].content).toBe(reference.slice(reference.indexOf('#### Choosing a source and checking sign-in'), reference.indexOf('### Tabs + frames'))); - expect(input.files[0].content.length).toBeGreaterThan(500); - expect(input.files[1].content.length).toBeGreaterThan(500); - expect(input.prompt).toContain('GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY'); - expect(input.prompt).toContain('isolated world'); - expect(input.prompt).not.toContain('### Tabs + frames'); - }); - - test('fails closed when either source file is absent', () => { - expect(() => buildCookieWorkflowJudgeInput(fixture(null))).toThrow(); - expect(() => buildCookieWorkflowJudgeInput(fixture(skill, null))).toThrow(); - }); - - test('rejects missing, duplicate, or non-heading markers', () => { - for (const broken of [skill.replace('# Setup Browser Cookies', '# Missing'), skill + '# Setup Browser Cookies\nDuplicate\n', skill.replace('# Setup Browser Cookies', 'Quoted # Setup Browser Cookies')]) { - expect(() => buildCookieWorkflowJudgeInput(fixture(broken))).toThrow('expected exactly one marker'); - } - for (const marker of ['#### Choosing a source and checking sign-in', '### Tabs + frames']) { - expect(() => buildCookieWorkflowJudgeInput(fixture(skill, browser.replace(marker, 'missing')))).toThrow('expected exactly one marker'); - expect(() => buildCookieWorkflowJudgeInput(fixture(skill, browser + marker + '\n'))).toThrow('expected exactly one marker'); - } - }); - - test('rejects empty, reversed, or unreferenced excerpts', () => { - expect(() => buildCookieWorkflowJudgeInput(fixture('# Setup Browser Cookies\n'))).toThrow('empty or reversed'); - expect(() => buildCookieWorkflowJudgeInput(fixture(skill, '#### Choosing a source and checking sign-in\n\n### Tabs + frames\n'))).toThrow('empty or reversed'); - expect(() => buildCookieWorkflowJudgeInput(fixture(skill, '### Tabs + frames\n#### Choosing a source and checking sign-in\nContent\n'))).toThrow('empty or reversed'); - expect(() => buildCookieWorkflowJudgeInput(fixture(skill.replace('`BROWSER.md`', '`other.md`')))).toThrow('missing cookie reference link'); - expect(() => buildCookieWorkflowJudgeInput(fixture(skill.replace('**Choosing a source and checking sign-in**', '**Other section**')))).toThrow('missing cookie reference link'); - }); - - test('generated host workflows resolve shared references from the installation root and check Aside first', () => { - for (const file of ['setup-browser-cookies/SKILL.md', '.agents/skills/gstack-setup-browser-cookies/SKILL.md']) { - const source = readFileSync(join(ROOT, file), 'utf8'); - const body = source.slice(source.indexOf('# Setup Browser Cookies')); - expect(body).toContain('Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`'); - expect(body).toContain('the installed root containing `bin/gstack-skill-start`, never a generated host stub'); - expect(body).toContain("Read that root's `browse/SKILL.md`"); - expect(body).toContain('**Aside probe prints `READY`:** no import is needed'); - expect(body.indexOf('**BROWSER SETUP**')).toBeLessThan(body.indexOf('**Browser fallback**')); - expect(body).toContain("Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**"); - expect(body).not.toContain('../BROWSER.md'); - } - expect(readFileSync(join(ROOT, 'browse/SKILL.md'), 'utf8')).toContain('## BROWSER SETUP (Aside'); - expect(readFileSync(join(ROOT, 'BROWSER.md'), 'utf8')).toContain('#### Choosing a source and checking sign-in'); - }); - - test('each owned dependency selects this judge in the fast PR profile', () => { - for (const file of ['setup-browser-cookies/SKILL.md.tmpl', 'setup-browser-cookies/SKILL.md', 'BROWSER.md', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/manual-judge-review-fixture.ts', '.github/cookie-workflow-manual-review.json']) { - const selectedJudges = selectTests([file], LLM_JUDGE_TOUCHFILES).selected; - // Helpers the judge file imports select every judge that file registers (derived closure). - const exact = !file.startsWith('test/helpers/') || file === 'test/helpers/manual-judge-review-fixture.ts'; - if (exact) expect(selectedJudges).toEqual([NAME]); else expect(selectedJudges).toContain(NAME); - const selectedE2E = selectTests([file], E2E_TOUCHFILES).selected; - const profile = selectPrProfile({ changedFiles: [file], selectedJudges, selectedE2E }); - if (exact) expect(profile.judges).toEqual([NAME]); else expect(profile.judges).toContain(NAME); - expect(profile.deferredPromptFiles).toEqual([]); - expect(profile.missingCoverage).toEqual([]); - expect(profile.needsFullValidation).toBe(false); - } - expect(selectTests(['README.md'], LLM_JUDGE_TOUCHFILES).selected).not.toContain(NAME); - }); - - test('the registered cookie judge sends the response schema, so a free-text reply like train 42\'s is not what gets parsed', async () => { - const { replies } = JSON.parse(readFileSync(join(ROOT, 'test/fixtures/cookie-judge-train42-replies.json'), 'utf8')) as { - replies: Array<{ id: string; model: string; stop_reason: string; text: string }> }; - expect(replies.map(reply => reply.id)).toEqual(['msg_011CgVnRtVj5vyLvb56m6KJH', 'msg_011CgVne5GStZ6f9pRoCvgWE']); - const original = process.env.ANTHROPIC_API_KEY; - process.env.ANTHROPIC_API_KEY = 'fake'; - const create = spyOn(Messages.prototype, 'create'); - try { - for (const reply of replies) { - expect(() => JSON.parse(reply.text)).toThrow(); - const marker = '"reasoning": "'; - const judgment: FixtureScore = { - clarity: Number(/"clarity": (\d)/.exec(reply.text)![1]), - completeness: Number(/"completeness": (\d)/.exec(reply.text)![1]), - actionability: Number(/"actionability": (\d)/.exec(reply.text)![1]), - reasoning: reply.text.slice(reply.text.indexOf(marker) + marker.length, reply.text.lastIndexOf('"}')), - }; - expect(judgment.reasoning).toContain('"one --domain per run"'); - create.mockReset(); - create.mockResolvedValue({ id: reply.id, model: reply.model, stop_reason: 'end_turn', - content: [{ type: 'text', text: JSON.stringify(judgment) }], usage: { input_tokens: 1, output_tokens: 1 } } as never); - const root = fixture(); - const h = actualCookieCallback(root, { callJudge: (prompt, model, options) => callJudge(prompt, model, options) }); - await h.run(); - expect(create).toHaveBeenCalledTimes(JUDGE_PANEL_SAMPLES); - for (const [request, options] of create.mock.calls as unknown as Array<[Record, { signal: AbortSignal }]>) { - expect(request).toEqual({ model: COOKIE_WORKFLOW_JUDGE.model, max_tokens: DEFAULT_JUDGE_MAX_TOKENS, - output_config: { format: { type: 'json_schema', schema: WORKFLOW_JUDGE_RESPONSE_SCHEMA } }, - messages: [{ role: 'user', content: buildCookieWorkflowJudgeInput(root).prompt }] }); - expect(options.signal).toBeInstanceOf(AbortSignal); - } - expect(h.records[0]).toMatchObject({ name: NAME, passed: true, execution: 'executed', - judge_scores: { clarity: judgment.clarity, completeness: judgment.completeness, actionability: judgment.actionability } }); - } - } finally { - create.mockRestore(); - if (original === undefined) delete process.env.ANTHROPIC_API_KEY; else process.env.ANTHROPIC_API_KEY = original; - } - }); - - test('the managed callback sends and records exact custom input without borrowing a normal cache identity', async () => { - const root = fixture(); - const input = buildCookieWorkflowJudgeInput(root); - let scores = passingScore; - const h = actualCookieCallback(root, { judge: async () => scores }); - await h.run(); - expect(h.requests).toHaveLength(JUDGE_PANEL_SAMPLES); - expect(h.requests[0].prompt).toBe(input.prompt); - expect(h.requests[0].model).toBe(COOKIE_WORKFLOW_JUDGE.model); - expect(h.requests[0].signal).toBeInstanceOf(AbortSignal); - expect(h.records[0]).toMatchObject({ name: NAME, prompt: input.prompt, model: COOKIE_WORKFLOW_JUDGE.model, execution: 'executed', passed: true }); - expect(existsSync(join(root, 'cache'))).toBe(false); - const fresh = actualCookieCallback(root); - await fresh.run(); - expect(fresh.requests).toHaveLength(JUDGE_PANEL_SAMPLES); - for (const dimension of ['clarity', 'completeness', 'actionability'] as const) { - scores = { ...COOKIE_WORKFLOW_JUDGE.thresholds, [dimension]: COOKIE_WORKFLOW_JUDGE.thresholds[dimension] - 1, reasoning: 'Synthetic failing fixture score' }; - await expect(h.run()).rejects.toThrow(); - expect(h.records.at(-1)).toMatchObject({ passed: false, exit_reason: 'validation_failed', prompt: input.prompt }); - } - expect(existsSync(join(root, 'cache'))).toBe(false); - }); - - test('custom input timeout aborts once and a late response cannot overwrite its successful retry', async () => { - const root = fixture(); - let completeLate!: (value: FixtureScore) => void; - let calls = 0, now = 0, timerId = 0; - const timers = new Map void>(); - const h = actualCookieCallback(root, { - budget: 20, clock: () => now, - setTimer: ((callback: () => void) => { timers.set(++timerId, callback); return timerId; }) as any, - clearTimer: ((id: number) => { timers.delete(id); }) as any, - judge: async () => ++calls === 1 ? new Promise(resolve => { completeLate = resolve; }) : passingScore, - }); - const expired = h.run().catch((error: Error) => error); - now = 20; - for (const callback of [...timers.values()]) callback(); - expect((await expired as Error).message).toContain('deadline'); - expect(h.records).toHaveLength(1); - expect(h.records[0]).toMatchObject({ passed: false, exit_reason: 'timeout', prompt: buildCookieWorkflowJudgeInput(root).prompt }); - expect(h.requests[0].signal.aborted).toBe(true); - await h.run(); - expect(h.records.map(record => record.passed)).toEqual([false, true]); - expect(h.attempts.get(NAME)?.attempt).toBe(2); - completeLate(passingScore); - await new Promise(resolve => setImmediate(resolve)); - expect(h.records).toHaveLength(2); - expect(existsSync(join(root, 'cache'))).toBe(false); - }); - - test('a superseding custom-input attempt cancels its predecessor without stale recording', async () => { - let completeLate!: (value: FixtureScore) => void; - let calls = 0; - const h = actualCookieCallback(fixture(), { - judge: async () => ++calls === 1 ? new Promise(resolve => { completeLate = resolve; }) : passingScore, - }); - const old = h.run().catch((error: Error) => error); - await h.run(); - expect((await old as Error).name).toBe('WorkflowJudgeSuperseded'); - expect(h.requests[0].signal.aborted).toBe(true); - completeLate(passingScore); - await new Promise(resolve => setImmediate(resolve)); - expect(h.records.map(record => [record.passed, record.exit_reason])).toEqual([[false, 'cancelled'], [true, undefined]]); - }); -}); diff --git a/test/cookie-workflow-manual-review.test.ts b/test/cookie-workflow-manual-review.test.ts deleted file mode 100644 index 309f14788..000000000 --- a/test/cookie-workflow-manual-review.test.ts +++ /dev/null @@ -1,106 +0,0 @@ -import { afterEach, expect, test } from 'bun:test'; -import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import { join, resolve } from 'node:path'; -import { buildCookieWorkflowJudgeInput, COOKIE_WORKFLOW_JUDGE } from './helpers/cookie-workflow-judge-input'; -import { COOKIE_MANUAL_REVIEW_FILE, getCookieWorkflowManualReview, isManualReviewEntry, manualReviewProblem } from './helpers/cookie-workflow-manual-review'; -import { approvedCookieWorkflowSource, manualReviewFixture } from './helpers/manual-judge-review-fixture'; -import { validWorkflowJudgeScore } from './helpers/workflow-judge-cache'; - -const ROOT = resolve(import.meta.dir, '..'); -const roots: string[] = []; -afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); }); - -function fixture() { - const root = mkdtempSync(join(tmpdir(), 'cookie-policy-')); roots.push(root); - for (const file of [COOKIE_MANUAL_REVIEW_FILE, 'setup-browser-cookies/SKILL.md', 'BROWSER.md']) { - const target = join(root, file); mkdirSync(resolve(target, '..'), { recursive: true }); - const source = readFileSync(join(ROOT, file), 'utf8'); - writeFileSync(target, file === COOKIE_MANUAL_REVIEW_FILE ? source : approvedCookieWorkflowSource(source)); - } - const entry = manualReviewFixture(root); - const approval = entry.manual_review!.approval; - const request = { testName: entry.name, prompt: entry.prompt!, model: entry.model!, - maxTokens: approval.max_tokens, thresholds: { ...approval.thresholds }, attempt: 1 }; - return { root, entry, approval, request, refusal: entry.manual_review!.refusal }; -} - -test('the committed approval names precisely the historical reviewed request, not the current generated workflow', () => { - const f = fixture(); - expect(buildCookieWorkflowJudgeInput(f.root).sha256).toBe(f.approval.prompt_sha256); - expect(buildCookieWorkflowJudgeInput(ROOT).sha256).not.toBe(f.approval.prompt_sha256); - expect(manualReviewProblem(f.entry, ROOT)).toBe('Manual review does not match current source and approval'); - expect(f.approval.thresholds).toEqual(COOKIE_WORKFLOW_JUDGE.thresholds); - expect(isManualReviewEntry(f.entry)).toBe(true); - expect(manualReviewProblem(f.entry, f.root)).toBeNull(); - expect(getCookieWorkflowManualReview(f.root, f.request, f.refusal)).toEqual(f.entry.manual_review); - expect(validWorkflowJudgeScore(f.entry as any, f.request.thresholds)).toBe(false); - expect(validWorkflowJudgeScore(f.entry.manual_review as any, f.request.thresholds)).toBe(false); -}); - -test.each(['case', 'prompt', 'model', 'budget', 'thresholds', 'source', 'approval', 'retry'])( - 'admission rejects mismatched %s without changing the approval', mismatch => { - const f = fixture(); const original = readFileSync(join(f.root, COOKIE_MANUAL_REVIEW_FILE), 'utf8'); - if (mismatch === 'case') f.request.testName = 'ship/SKILL.md workflow'; - if (mismatch === 'prompt') f.request.prompt += '\n'; - if (mismatch === 'model') f.request.model += '-other'; - if (mismatch === 'budget') f.request.maxTokens++; - if (mismatch === 'thresholds') f.request.thresholds.clarity++; - if (mismatch === 'retry') f.request.attempt++; - if (mismatch === 'source') writeFileSync(join(f.root, 'BROWSER.md'), readFileSync(join(f.root, 'BROWSER.md'), 'utf8').replace('Storage stays intact', 'Changed storage stays intact')); - if (mismatch === 'approval') writeFileSync(join(f.root, COOKIE_MANUAL_REVIEW_FILE), JSON.stringify({ ...f.approval, prompt_sha256: 'a'.repeat(64) })); - expect(getCookieWorkflowManualReview(f.root, f.request, f.refusal)).toBeNull(); - if (mismatch !== 'approval') expect(readFileSync(join(f.root, COOKIE_MANUAL_REVIEW_FILE), 'utf8')).toBe(original); - }); - -test.each(['passed', 'scores', 'reused', 'timeout', 'case', 'suite', 'prompt', 'model', 'refusal', 'missing-id', 'missing-response-id', 'output', 'text', 'tokens', 'schema', 'retry'])( - 'malformed manual receipt %s cannot count as acceptance', invalid => { - const f = fixture(); const entry: any = f.entry; - if (invalid === 'passed') entry.passed = true; - if (invalid === 'retry') entry.attempt++; - if (invalid === 'scores') entry.judge_scores = { clarity: 1 }; - if (invalid === 'reused') entry.execution = 'reused'; - if (invalid === 'timeout') entry.exit_reason = 'timeout'; - if (invalid === 'case') entry.name = 'ship/SKILL.md workflow'; - if (invalid === 'suite') entry.suite = 'Unrelated suite'; - if (invalid === 'prompt') entry.prompt += '\n'; - if (invalid === 'model') entry.model = 'another-model'; - if (invalid === 'refusal') entry.manual_review.refusal.stop_reason = 'end_turn'; - if (invalid === 'missing-id') entry.manual_review.refusal.request_id = null; - if (invalid === 'missing-response-id') entry.manual_review.refusal.response_id = null; - if (invalid === 'output') entry.manual_review.refusal.output_tokens = 1; - if (invalid === 'text') entry.manual_review.refusal.text_blocks = 1; - if (invalid === 'tokens') entry.manual_review.refusal.input_tokens = -1; - if (invalid === 'schema') entry.manual_review.approval.schema_version = 2; - expect(isManualReviewEntry(entry)).toBe(false); - expect(manualReviewProblem(entry, f.root)).not.toBeNull(); - }); - -test('reconciliation rejects changed source, approval provenance, missing approval and corrupt JSON', () => { - const f = fixture(); - const modified = structuredClone(f.entry); - modified.manual_review!.approval.approved_by = 'unapproved-reviewer'; - expect(manualReviewProblem(modified, f.root)).not.toBeNull(); - const file = join(f.root, COOKIE_MANUAL_REVIEW_FILE); - writeFileSync(file, '{'); - expect(manualReviewProblem(f.entry, f.root)).not.toBeNull(); - rmSync(file); - expect(manualReviewProblem(f.entry, f.root)).not.toBeNull(); - writeFileSync(file, JSON.stringify(f.approval)); - writeFileSync(join(f.root, 'BROWSER.md'), readFileSync(join(f.root, 'BROWSER.md'), 'utf8').replace('Storage stays intact', 'Changed storage stays intact')); - expect(manualReviewProblem(f.entry, f.root)).not.toBeNull(); - expect(isManualReviewEntry(f.entry)).toBe(true); -}); - -test('a current judge-model override cannot reuse the different approved model', () => { - const f = fixture(); - const original = process.env.GSTACK_EVAL_MODEL_JUDGE; - try { - process.env.GSTACK_EVAL_MODEL_JUDGE = 'synthetic-unapproved-model'; - expect(manualReviewProblem(f.entry, f.root)).toBe('Manual review model is not the current judge model'); - expect(isManualReviewEntry(f.entry)).toBe(true); - } finally { - if (original === undefined) delete process.env.GSTACK_EVAL_MODEL_JUDGE; - else process.env.GSTACK_EVAL_MODEL_JUDGE = original; - } -}); diff --git a/test/design-craft-resolvers.test.ts b/test/design-craft-resolvers.test.ts index d05d961df..28cde1657 100644 --- a/test/design-craft-resolvers.test.ts +++ b/test/design-craft-resolvers.test.ts @@ -10,7 +10,7 @@ import * as fs from 'fs'; import * as path from 'path'; import { DESIGN_CRAFT_RULE, DESIGN_PERSUADE_CHECKS } from '../scripts/resolvers/design-craft'; import { selectTests } from './helpers/test-selection'; -import { E2E_TOUCHFILES, LLM_JUDGE_TOUCHFILES } from './helpers/touchfiles-data'; +import { E2E_TOUCHFILES } from './helpers/touchfiles-data'; import type { TemplateContext } from '../scripts/resolvers/types'; const ROOT = path.resolve(import.meta.dir, '..'); @@ -81,7 +81,6 @@ describe('design-shotgun to design-html handoff', () => { test('a change to the craft resolvers selects the design-review and plan-design-review cases that render them', () => { const e2e = selectTests(['scripts/resolvers/design-craft.ts'], E2E_TOUCHFILES).selected; for (const name of ['design-review-fix', 'design-review-detector-shim', 'design-review-detector-shim-dom', 'design-review-plugin-handoff', 'plan-design-review-plan-mode']) expect(e2e).toContain(name); - expect(selectTests(['scripts/resolvers/design-craft.ts'], LLM_JUDGE_TOUCHFILES).selected).toContain('plan-design-review/SKILL.md passes'); }); // Value: protects=design-html runs the lint and keeps waivers in the page, with no finalized.json exception record left behind; fails_when=the prose names the deleted checker, --accepted or craft_exceptions again; why_new=option (a) replaced craft_exceptions with in-page waivers; seam=none; no_claim=that an agent writes good waiver reasons (paid case) diff --git a/test/eval-flake-rank.test.ts b/test/eval-flake-rank.test.ts index dacb83f07..ffc1c03e4 100644 --- a/test/eval-flake-rank.test.ts +++ b/test/eval-flake-rank.test.ts @@ -48,10 +48,10 @@ describe('eval-flake-rank aggregate', () => { const display = spawnSync(process.execPath, [path.resolve(import.meta.dir, '../scripts/eval-flake-rank.ts'), '--dir', dir], { encoding: 'utf8', timeout: 10_000 }); expect(display.status, display.stderr).toBe(0); - // pass-rates view: the prior automated pass is the one scored pre-policy - // trial; the manual acceptance is counted in its own column, never scored. + // pass-rates view: the retired cookie judge is no longer a registered case, + // so its historical records are listed as unattributed, never guessed into a series. expect(display.stdout).toContain('pre-policy manual case'); - expect(display.stdout).toMatch(new RegExp(`1/1 \\[[^\\]]+\\]\\s+1 ${manual.name.replace(/[.*+?^${}()|[\]\\/]/g, '\\$&')}`)); + expect(display.stdout).toContain(`unattributed records (1, never guessed): ${manual.name}`); fs.writeFileSync(path.join(dir, 'invalid-retry.json'), run([ { ...ordinary, attempt: 1 }, { ...manual, attempt: 2 }, ])); diff --git a/test/fixtures/browse-judge/panel-means.json b/test/fixtures/browse-judge/panel-means.json deleted file mode 100644 index ad4864e77..000000000 --- a/test/fixtures/browse-judge/panel-means.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "_comment": "Browse reference judge panel means against baseline {clarity:4, completeness:4, actionability:4}. known_good passes the floors (the old ratchet failed the first two); known_bad fails them.", - "known_good": { - "clarity-dip-to-3.67": { "clarity": 3.67, "completeness": 4, "actionability": 4 }, - "clarity-at-floor": { "clarity": 3, "completeness": 4.33, "actionability": 4 }, - "at-baseline": { "clarity": 4, "completeness": 4, "actionability": 4 } - }, - "known_bad": { - "clarity-below-floor": { "clarity": 2.67, "completeness": 4, "actionability": 4 }, - "completeness-below-floor": { "clarity": 4, "completeness": 3.67, "actionability": 4 }, - "actionability-below-floor": { "clarity": 4, "completeness": 4, "actionability": 3.33 } - } -} diff --git a/test/fixtures/cookie-judge-train42-replies.json b/test/fixtures/cookie-judge-train42-replies.json deleted file mode 100644 index ea3043176..000000000 --- a/test/fixtures/cookie-judge-train42-replies.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "source": "train 42 gate run 40809051510 (paid-logs-slice-5-a1, rerun-logs-1-a1): llm-judge-response-parse-error events", - "replies": [ - { - "id": "msg_011CgVnRtVj5vyLvb56m6KJH", - "model": "claude-sonnet-5-5", - "stop_reason": "end_turn", - "text": "{\"clarity\": 4, \"completeness\": 4, \"actionability\": 4, \"reasoning\": \"The workflow is well ordered: check Aside, collect decisions up front, prepare the browser, import, and report. Consent for storage reset and --all is explicit, and the guidance separates copied cookies from verified sign-in. Error tables, platform limits, and the privacy rules are clear. Weaknesses: (1) the picker-versus-direct choice is underspecified, because the picker is used when browser, profile or domains are open and direct import when they are named, but the target-URL, `--all`, and verify-auth flows under each are not fully mapped. (2) The `--all` handling is thin. SKILL.md does not give direct-import syntax for it and does not say whether `$B goto` is still needed. (3) Verification setup has an ordering wrinkle: the env vars must be set before the daemon starts, while the Aside and `$B status` checks come first, and the instructions only partly say when to stop and restart the daemon. (4) A few edge cases are not defined, such as what to do when the reset or verification fails, how to count partial results, and how to proceed after the failure-reason handoff. (5) The multi-domain instruction is \"one --domain per run\", but it is unclear whether to repeat runs or how to aggregate the receipts. (6) The \"Windows App-Bound Encryption\" recovery depends on a display, with no alternative if none is available. These gaps are minor, and the workflow is mostly executable.\"}" - }, - { - "id": "msg_011CgVne5GStZ6f9pRoCvgWE", - "model": "claude-sonnet-5-5", - "stop_reason": "end_turn", - "text": "{\"clarity\": 4, \"completeness\": 4, \"actionability\": 4, \"reasoning\": \"The workflow is well ordered: probe Aside, collect all decisions up front, set up the browser, import, then report three honest lines. Consent boundaries for verification, storage clearing and --all are explicit, the sign-in-versus-cookie-copy distinction is stated repeatedly, and the error and receipt tables give recovery paths. Gaps: (1) Step 2 asks for decisions before `$B` exists, yet the picker-versus-direct choice is tied to browser, profile and domains rather than to the user's wording, and the step 3.1 env vars come before the daemon starts. (2) The conditional step 3.3 (Mode: cdp) and the `$B stop` guidance could be cleaner. (3) The direct-import syntax for `--all` is not shown, and the \"one --domain per run\" rule is stated without saying what to do for several domains (loop, with the tab open on each). (4) The verification-missing return path says to return to step 3 but doesn't say how to handle the already-running daemon beyond `$B stop`. (5) Handling of partial and failed receipts, and the handoff for non-Windows cases, is thin. (6) How the agent learns the profile directory when the user only knows a display name is left to the picker. (7) The root-selection logic with \"never a generated host stub\" is a bit dense. These are minor; the workflow is mostly executable.\"}" - } - ] -} diff --git a/test/fixtures/eval-baselines.json b/test/fixtures/eval-baselines.json deleted file mode 100644 index 2978e45db..000000000 --- a/test/fixtures/eval-baselines.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "browse_skill": { "clarity": 4, "completeness": 4, "actionability": 4 }, - "qa_workflow": { "clarity": 4, "completeness": 4, "actionability": 4 }, - "qa_health_rubric": { "clarity": 4, "completeness": 3, "actionability": 4 } -} diff --git a/test/fixtures/judge-calibration/README.md b/test/fixtures/judge-calibration/README.md index 7fe8c221f..57183f77b 100644 --- a/test/fixtures/judge-calibration/README.md +++ b/test/fixtures/judge-calibration/README.md @@ -22,6 +22,13 @@ and priced usage, and `comparison-1.result.json` holds the flip rates, agreement false passes, error rates, p95 latency, landing decision and the UC3 prose-step trigger. `manifest.json` prices every planned call before dispatch. +The `qa-workflow`, `qa-health-rubric`, `workflow-default` and `cookie-workflow` +configurations were removed with the skill-text clarity judges (GST-1371, +2026-11-04). Their corpus directories no longer serve any configuration and are +deleted in a follow-up (the deletion alone is larger than the /ship review +ceiling). Their rows below, and their entries in `manifest.json`, are the +historical record of the comparison. + ## Comparison (1) results (2026-10-03, `claude-fable-5-1`, unless the row says otherwise) Flips are panel-verdict differences on items where both panels completed: old-vs-old diff --git a/test/fixtures/manual-judge-review-entry.json b/test/fixtures/manual-judge-review-entry.json new file mode 100644 index 000000000..e6e57a0aa --- /dev/null +++ b/test/fixtures/manual-judge-review-entry.json @@ -0,0 +1,42 @@ +{ + "name": "setup-browser-cookies/SKILL.md workflow", + "suite": "Cookie setup workflow quality", + "tier": "llm-judge", + "passed": false, + "execution": "executed", + "exit_reason": "provider_refusal", + "attempt": 1, + "duration_ms": 1, + "cost_usd": 0, + "model": "claude-fable-5-1", + "prompt": "You are evaluating the quality of a fallback-browser cookie import workflow for an AI coding agent.\n\nThe agent reads these source files to learn how to select an authorized source browser, profile, and domain without guessing an account; configure optional authentication verification before mutation; obtain explicit consent for precisely scoped storage reset; distinguish copied cookies from positive sign-in evidence; and recover within the documented platform and privacy boundaries. Shared preamble definitions and\nexternal tools/files are documented separately; do not penalize their absence from this bundle.\nOn-demand sections retain their original file boundaries and Read instructions; the section\nindex refers to those files, not duplicate work. The bundle order is not execution order.\nJudge the actual instructions, including contradictory ordering or missing decisions.\n\nRate on three dimensions (1-5 scale):\n- **clarity** (1-5): Can an agent follow the instructions without ambiguity?\n- **completeness** (1-5): Are all steps, decision points, and outputs well-defined?\n- **actionability** (1-5): Can an agent execute this workflow and produce the expected deliverables?\n\nRespond with ONLY valid JSON:\n{\"clarity\": N, \"completeness\": N, \"actionability\": N, \"reasoning\": \"brief explanation\"}\n\nHere is the source-file bundle to evaluate:\n\nSKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step.\n\n--- BEGIN FILE \"setup-browser-cookies/SKILL.md\" (lines 153-235; entrypoint) ---\n# Setup Browser Cookies\n\n## 1. Choose the browser\n\nUse this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its probe first. On `READY`, stop importing: use Aside's sessions or ask the user to sign in there. Otherwise follow the probe's fallback handling, then continue below.\n\n## SETUP (run this check BEFORE any browse command)\n\n```bash\n_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)\nB=\"\"\n[ -n \"$_ROOT\" ] && [ -x \"$_ROOT/.claude/skills/gstack/browse/dist/browse\" ] && B=\"$_ROOT/.claude/skills/gstack/browse/dist/browse\"\n[ -z \"$B\" ] && B=\"$HOME/.claude/skills/gstack/browse/dist/browse\"\nif [ -x \"$B\" ]; then\n echo \"READY: $B\"\nelse\n echo \"NEEDS_SETUP\"\nfi\n```\n\nIf `NEEDS_SETUP`:\n1. Tell the user: \"gstack browse needs a one-time build (~10 seconds). OK to proceed?\" Then STOP and wait.\n2. Run: `cd && ./setup`\n3. If `bun` is not installed:\n ```bash\n if ! command -v bun >/dev/null 2>&1; then\n BUN_VERSION=\"1.3.10\"\n BUN_INSTALL_SHA=\"bab8acfb046aac8c72407bdcce903957665d655d7acaa3e11c7c4616beae68dd\"\n tmpfile=$(mktemp)\n curl -fsSL \"https://bun.sh/install\" -o \"$tmpfile\"\n # shasum is macOS/perl; coreutils-only Linux ships sha256sum instead —\n # resolve whichever exists so the verify never fails on a missing tool.\n if command -v sha256sum >/dev/null 2>&1; then\n actual_sha=$(sha256sum \"$tmpfile\" | awk '{print $1}')\n else\n actual_sha=$(shasum -a 256 \"$tmpfile\" | awk '{print $1}')\n fi\n if [ \"$actual_sha\" != \"$BUN_INSTALL_SHA\" ]; then\n echo \"ERROR: bun install script checksum mismatch\" >&2\n echo \" expected: $BUN_INSTALL_SHA\" >&2\n echo \" got: $actual_sha\" >&2\n rm \"$tmpfile\"; exit 1\n fi\n BUN_VERSION=\"$BUN_VERSION\" bash \"$tmpfile\"\n rm \"$tmpfile\"\n fi\n ```\n\n```bash\n$B status\n```\nIf status says `Mode: cdp`, stop: the real browser already has sessions.\n\n## 2. Confirm options before import\n\nOpen the known target and keep its tab unchanged. Both options default off and require explicit request:\n\n- **`--verify-auth` / picker checkbox:** reloads the target. Have the user privately configure daemon `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` **before startup**. Never invent values or assume CLI env reconfigures an existing daemon. Missing config rejects before mutation. Require a successful same-origin response and exactly one visible element whose whitespace-normalized text exactly matches the expected identity.\n- **`--clear-storage`:** for suspected stale storage, obtain explicit approval. Chromium only: clears captured-origin localStorage (shared across same-origin context tabs) and target-tab sessionStorage. Other origins, other tabs' sessionStorage, IndexedDB, and service workers stay intact. It uses an isolated world/native deadline; other engines reject reset, not imports/auth checks. Never auto-approve or claim rollback after partial failure.\n\n## 3. Select source and scope\n\n```bash\n$B cookie-import-browser\n```\n\nAsk the user to choose browser, account/profile, and domains, then say when done. Never guess accounts or treat default Comet as consent. Unreadable profiles are unknown, not empty. Rerun for an expired five-minute one-use link.\n\nDirect import: pass the chosen browser and `--domain` after navigating to a matching target. `--profile` takes a directory, not a display name; omit only for an unambiguous relevant profile, otherwise use the picker. `--all` requires consent for all non-expired profile cookies; it cannot accompany `--domain` or `--clear-storage`.\n\nRead that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for examples, profile labels, supported sources and platform setup.\n\n## 4. Report honestly\n\nReport receipt/picker counts, partial/zero/error and reset outcomes, not raw `$B cookies`. Imports affect the context, not one tab. **Not checked** means no requested check; **not verified** means it failed; **verified** requires positive target evidence. Zero imports, counts, or HTTP 200 never prove login.\n\nNever request/publish cookie values, passwords, identity/profile text, session details, or raw errors in public logs.\n\n## Platform boundaries\n\nDia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts.\n\nWindows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available.\n\n--- END FILE \"setup-browser-cookies/SKILL.md\" ---\n\n--- BEGIN FILE \"BROWSER.md\" (lines 569-587; section) ---\n#### Choosing a source and checking sign-in\n\nSelect the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes.\n\nFor direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`:\n\n```bash\n$B goto https://example.com\n$B cookie-import-browser chrome --domain example.com --profile \"Profile 2\"\n```\n\n`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in.\n\n`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes.\n\nStorage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it.\n\n**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available.\n\n\n--- END FILE \"BROWSER.md\" ---", + "error": "Synthetic provider refusal fixture, not live model evidence", + "manual_review": { + "approval": { + "schema_version": 1, + "test_name": "setup-browser-cookies/SKILL.md workflow", + "prompt_sha256": "7f7f76f49912818d51f25c86d686aab6ad2c95ccdea60ac912c1e4c8b6936e5f", + "prompt_bytes": 10159, + "model": "claude-fable-5-1", + "max_tokens": 8192, + "thresholds": { + "clarity": 4, + "completeness": 3, + "actionability": 4 + }, + "approved_by": "garrytan", + "approved_at": "2026-09-24", + "approval_url": "https://github.com/garrytan/gstack/pull/2964#issuecomment-5822514476", + "reason": "Maintainer-approved manual review of this exact cookie workflow after empty provider refusals. No automated quality score or pass credit; other errors and changed inputs remain blocking." + }, + "refusal": { + "stop_reason": "refusal", + "response_id": "msg_synthetic_fixture", + "request_id": "req_synthetic_fixture", + "model": "claude-fable-5-1", + "input_tokens": 1, + "output_tokens": 0, + "text_blocks": 0 + } + } +} diff --git a/test/fixtures/touchfile-moved-code/w1-state-root.json b/test/fixtures/touchfile-moved-code/w1-state-root.json index a460cbcec..c262af1ca 100644 --- a/test/fixtures/touchfile-moved-code/w1-state-root.json +++ b/test/fixtures/touchfile-moved-code/w1-state-root.json @@ -22,9 +22,7 @@ "qa-only-no-fix", "qa-quick" ], - "llmJudge": [ - "browse/SKILL.md reference" - ] + "llmJudge": [] }, "lib/cso/state.ts": { "e2e": [ diff --git a/test/fixtures/touchfile-moved-code/w3-browse-routes.json b/test/fixtures/touchfile-moved-code/w3-browse-routes.json index 59d6887cf..abbdb7e19 100644 --- a/test/fixtures/touchfile-moved-code/w3-browse-routes.json +++ b/test/fixtures/touchfile-moved-code/w3-browse-routes.json @@ -22,9 +22,7 @@ "qa-only-no-fix", "qa-quick" ], - "llmJudge": [ - "browse/SKILL.md reference" - ] + "llmJudge": [] } }, "modules": { diff --git a/test/fixtures/touchfile-moved-code/w5-review-resolvers.json b/test/fixtures/touchfile-moved-code/w5-review-resolvers.json index 6cbf605a1..4b8b23bd0 100644 --- a/test/fixtures/touchfile-moved-code/w5-review-resolvers.json +++ b/test/fixtures/touchfile-moved-code/w5-review-resolvers.json @@ -59,12 +59,7 @@ "ship-exploratory-unavailable", "ship-skipped-queued-finding" ], - "llmJudge": [ - "plan-design-review/SKILL.md passes", - "plan-eng-review/SKILL.md sections", - "review/SKILL.md workflow", - "ship/SKILL.md workflow" - ] + "llmJudge": [] }, "scripts/resolvers/design.ts": { "e2e": [ @@ -78,10 +73,7 @@ "design-review-plugin-handoff", "plan-design-with-ui-scope" ], - "llmJudge": [ - "design-consultation/SKILL.md research", - "plan-design-review/SKILL.md passes" - ] + "llmJudge": [] } }, "modules": { diff --git a/test/helpers/cookie-workflow-judge-input.ts b/test/helpers/cookie-workflow-judge-input.ts deleted file mode 100644 index 814d66892..000000000 --- a/test/helpers/cookie-workflow-judge-input.ts +++ /dev/null @@ -1,51 +0,0 @@ -import { createHash } from 'node:crypto'; -import { readFileSync } from 'node:fs'; -import { join } from 'node:path'; -import { buildWorkflowJudgePrompt, type WorkflowJudgeFile, type WorkflowJudgeInput } from './workflow-judge-input'; - -export const COOKIE_WORKFLOW_JUDGE = { - model: 'claude-sonnet-5-5', - judgeContext: 'a fallback-browser cookie import workflow', - judgeGoal: 'how to select an authorized source browser, profile, and domain without guessing an account; configure optional authentication verification before mutation; obtain explicit consent for precisely scoped storage reset; distinguish copied cookies from positive sign-in evidence; and recover within the documented platform and privacy boundaries', - thresholds: { clarity: 4, completeness: 3, actionability: 4 }, -} as const; - -export function buildCookieWorkflowJudgeInput(root: string): WorkflowJudgeInput & { prompt: string; sha256: string } { - const files: WorkflowJudgeFile[] = [ - { path: 'setup-browser-cookies/SKILL.md', kind: 'entrypoint', start: '# Setup Browser Cookies', end: null }, - { path: 'BROWSER.md', kind: 'section', start: '#### Choosing a source and checking sign-in', end: '### Tabs + frames' }, - ].map(spec => { - const source = readFileSync(join(root, spec.path), 'utf8'); - const locate = (marker: string): number => { - const matches = [...source.matchAll(new RegExp(`^${marker.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}\\r?$`, 'gm'))]; - if (matches.length !== 1) throw new Error(`${spec.path}: expected exactly one marker ${JSON.stringify(marker)}, found ${matches.length}`); - return matches[0].index!; - }; - const start = locate(spec.start); - const end = spec.end === null ? source.length : locate(spec.end); - if (end <= start || !source.slice(start + spec.start.length, end).trim()) { - throw new Error(`${spec.path}: empty or reversed cookie workflow excerpt`); - } - return { - path: spec.path, - kind: spec.kind as WorkflowJudgeFile['kind'], - content: source.slice(start, end), - startLine: source.slice(0, start).split('\n').length, - endLine: source.slice(0, end - 1).split('\n').length, - }; - }); - if (!files[0].content.includes('`BROWSER.md`') || !files[0].content.includes('**Choosing a source and checking sign-in**')) { - throw new Error('setup-browser-cookies/SKILL.md: missing cookie reference link'); - } - const text = [ - 'SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step.', - ...files.map(file => [ - `--- BEGIN FILE ${JSON.stringify(file.path)} (lines ${file.startLine}-${file.endLine}; ${file.kind}) ---`, - file.content, - `--- END FILE ${JSON.stringify(file.path)} ---`, - ].join('\n')), - ].join('\n\n'); - const input = { files, text }; - const prompt = buildWorkflowJudgePrompt(COOKIE_WORKFLOW_JUDGE, input); - return { ...input, prompt, sha256: createHash('sha256').update(prompt).digest('hex') }; -} diff --git a/test/helpers/cookie-workflow-manual-review.ts b/test/helpers/cookie-workflow-manual-review.ts index 723b9e63e..22b6f3207 100644 --- a/test/helpers/cookie-workflow-manual-review.ts +++ b/test/helpers/cookie-workflow-manual-review.ts @@ -1,11 +1,7 @@ +/** Recognizes historical cookie-workflow manual-review records; the judge and its approval are retired. */ import { createHash } from 'node:crypto'; -import { readFileSync } from 'node:fs'; -import { join } from 'node:path'; -import { buildCookieWorkflowJudgeInput, COOKIE_WORKFLOW_JUDGE } from './cookie-workflow-judge-input'; import type { JudgeRefusalEvidence } from './llm-judge'; -import { DEFAULT_JUDGE_MAX_TOKENS, resolveEvalModel } from '../../lib/eval-model'; -export const COOKIE_MANUAL_REVIEW_FILE = '.github/cookie-workflow-manual-review.json'; const CASE = 'setup-browser-cookies/SKILL.md workflow'; type Thresholds = { clarity: number; completeness: number; actionability: number }; @@ -31,7 +27,6 @@ export interface ManualJudgeReview { const object = (value: unknown): value is Record => value !== null && typeof value === 'object' && !Array.isArray(value); const nonempty = (value: unknown): value is string => typeof value === 'string' && value.trim().length > 0; const dimensions = ['clarity', 'completeness', 'actionability'] as const; -const sameThresholds = (a: Thresholds, b: Thresholds) => dimensions.every(key => a[key] === b[key]); function validApproval(value: unknown): value is CookieManualApproval { return object(value) && value.schema_version === 1 && value.test_name === CASE @@ -63,39 +58,3 @@ export function isManualReviewEntry(entry: unknown): boolean { && Buffer.byteLength(entry.prompt) === approval.prompt_bytes && createHash('sha256').update(entry.prompt).digest('hex') === approval.prompt_sha256; } - -export function getCookieWorkflowManualReview(root: string, request: { - testName: string; prompt: string; model: string; maxTokens: number; thresholds: Thresholds; attempt: number; -}, refusal: JudgeRefusalEvidence): ManualJudgeReview | null { - if (request.testName !== CASE || request.attempt !== 1 || !validRefusal(refusal, request.model)) return null; - let approval: unknown; - try { approval = JSON.parse(readFileSync(join(root, COOKIE_MANUAL_REVIEW_FILE), 'utf8')); } - catch (error) { - if ((error as NodeJS.ErrnoException).code === 'ENOENT') return null; - throw error; - } - if (!validApproval(approval)) throw new Error('Invalid cookie workflow manual-review approval'); - const current = buildCookieWorkflowJudgeInput(root); - if (request.prompt !== current.prompt || current.sha256 !== approval.prompt_sha256 - || Buffer.byteLength(request.prompt) !== approval.prompt_bytes || request.model !== approval.model - || request.maxTokens !== approval.max_tokens || request.maxTokens !== DEFAULT_JUDGE_MAX_TOKENS - || !sameThresholds(request.thresholds, approval.thresholds) - || !sameThresholds(request.thresholds, COOKIE_WORKFLOW_JUDGE.thresholds)) return null; - return { approval, refusal }; -} - -export function manualReviewProblem(entry: unknown, root: string): string | null { - if (!object(entry) || !Object.hasOwn(entry, 'manual_review')) return null; - if (!isManualReviewEntry(entry)) return 'Malformed manual-review claim'; - if (entry.model !== resolveEvalModel('judge')) return 'Manual review model is not the current judge model'; - try { - const claimed = entry.manual_review as ManualJudgeReview; - const verified = getCookieWorkflowManualReview(root, { testName: entry.name, prompt: entry.prompt, - model: entry.model, maxTokens: claimed.approval.max_tokens, thresholds: claimed.approval.thresholds, - attempt: entry.attempt }, claimed.refusal); - if (!verified || Object.entries(verified.approval).some(([key, value]) => key === 'thresholds' - ? !sameThresholds(value as Thresholds, claimed.approval.thresholds) - : value !== claimed.approval[key as keyof CookieManualApproval])) return 'Manual review does not match current source and approval'; - return null; - } catch { return 'Manual review approval or current input is unavailable or invalid'; } -} diff --git a/test/helpers/eval-budgets.ts b/test/helpers/eval-budgets.ts index 667d0e09b..847a2225d 100644 --- a/test/helpers/eval-budgets.ts +++ b/test/helpers/eval-budgets.ts @@ -99,9 +99,8 @@ export const FILE_RETRY_BUDGETS = [ { file: 'test/skill-e2e-qa-callers.test.ts', attemptMs: 5 * (CAPTURE_MS + 15_000), caseMs: CAPTURE_MS + 15_000 }, { file: 'test/skill-e2e-shared-libs-paths.test.ts', attemptMs: 3 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS }, { file: 'test/skill-e2e-ship-docsync.test.ts', attemptMs: 4 * CAPTURE_LONG_MS + 8 * CAPTURE_MS, caseMs: CAPTURE_LONG_MS }, - // Seventeen workflow judges include their 10s recording grace; the other - // seven judges retain 120s. Supervise all 24. - { file: 'test/skill-llm-eval.test.ts', attemptMs: 17 * (JUDGE_MS + 10_000) + 7 * JUDGE_MS, caseMs: JUDGE_MS + 10_000 }, + // Three judges retain 120s each. Supervise all three. + { file: 'test/skill-llm-eval.test.ts', attemptMs: 3 * JUDGE_MS, caseMs: JUDGE_MS }, { file: 'test/skill-e2e-auq-matrix.test.ts', attemptMs: 6 * CAPTURE_MS, caseMs: CAPTURE_MS }, { file: 'test/skill-e2e-plan-format.test.ts', attemptMs: 4 * (CAPTURE_MS + 10_000), caseMs: CAPTURE_MS + 10_000 }, { file: 'test/skill-e2e-auto-decide-preserved.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS }, diff --git a/test/helpers/free-fixtures-data.ts b/test/helpers/free-fixtures-data.ts index 51f4bc8df..5d704de4b 100644 --- a/test/helpers/free-fixtures-data.ts +++ b/test/helpers/free-fixtures-data.ts @@ -30,7 +30,6 @@ export const FREE_FIXTURES: Readonly> = { 'test/fixtures/autoplan-setup-packet-o-call.json': ['test/plan-pending-question-pty.test.ts'], 'test/fixtures/autoplan-setup-packet-o-screen.txt': ['test/plan-pending-question-pty.test.ts'], 'test/fixtures/autoplan-with-result-au.json': ['test/autoplan-phase-observer.test.ts'], - 'test/fixtures/browse-judge/**': ['test/workflow-judge-cache.test.ts'], 'test/fixtures/ceo-completion-handoff-m-call.json': ['test/plan-count-completion.test.ts'], 'test/fixtures/claude-usage-2.1.292.json': ['test/claude-usage.test.ts'], 'test/fixtures/context-restore-transcript.jsonl': ['test/context-restore-transcripts.test.ts'], @@ -63,7 +62,6 @@ export const FREE_FIXTURES: Readonly> = { 'test/fixtures/context-bill/**': ['test/context-bill.test.ts', 'test/context-budget-ratchet.test.ts'], 'test/fixtures/context-budget.json': ['test/helpers/capture-context-budget.ts', 'test/test-value-bar.test.ts'], 'test/fixtures/context-restore-fragment-match-captures.json': ['test/context-restore-fragment-verdict.test.ts'], - 'test/fixtures/cookie-judge-train42-replies.json': ['test/cookie-workflow-judge-input.test.ts'], 'test/fixtures/coverage-audit-cd-chain.json': ['test/coverage-audit-evidence.test.ts'], 'test/fixtures/coverage-audit-parallel-column.json': ['test/coverage-audit-evidence.test.ts'], 'test/fixtures/coverage-audit-plan-eng-wc-bytes.json': ['test/coverage-audit-evidence.test.ts'], @@ -165,6 +163,7 @@ export const FREE_FIXTURES: Readonly> = { 'test/fixtures/judge-panel-replay-2026-10-05.json': ['test/judge-panel-median.test.ts'], 'test/fixtures/junit/**': ['test/flaky-run.test.ts'], 'test/fixtures/launch-env-echo/**': ['test/launch-env-echo-ratchet.test.ts'], + 'test/fixtures/manual-judge-review-entry.json': ['test/helpers/manual-judge-review-fixture.ts'], 'test/fixtures/merge-train-3345/**': ['test/changelog-merge-damage.test.ts', 'test/compare-pr-version.test.ts'], 'test/fixtures/merge-train-3453/**': ['test/changelog-merge-damage.test.ts'], 'test/fixtures/module-size-ratchet.json': ['test/helpers/module-size.ts', 'test/module-size-ratchet.test.ts'], diff --git a/test/helpers/llm-judge.ts b/test/helpers/llm-judge.ts index ff32d03f3..b804a4585 100644 --- a/test/helpers/llm-judge.ts +++ b/test/helpers/llm-judge.ts @@ -23,8 +23,6 @@ export interface JudgeScore { reasoning: string; } -export const JUDGE_SCORE_DIMENSIONS = ['clarity', 'completeness', 'actionability'] as const; - export interface JudgeRefusalEvidence { stop_reason: 'refusal'; response_id: string | null; @@ -359,14 +357,6 @@ export function judgePanelReasoning(samples: ReadonlyArray): string { } const score = { type: 'integer', enum: [1, 2, 3, 4, 5] } as const; -// Structured output guarantees parseable JSON; free-form judges failed on -// unescaped quotes inside their reasoning (run 36798539821, setup block). -export const JUDGE_SCORE_SCHEMA = { - type: 'object', - properties: { clarity: score, completeness: score, actionability: score, reasoning: { type: 'string' } }, - required: ['clarity', 'completeness', 'actionability', 'reasoning'], - additionalProperties: false, -}; export const OUTCOME_JUDGE_SCHEMA = { type: 'object', properties: { @@ -418,39 +408,6 @@ export const VOICE_DIRECTIVE_JUDGE_SCHEMA = { additionalProperties: false, }; -/** - * Score documentation quality on clarity/completeness/actionability (1-5). - */ -export async function judge(section: string, content: string): Promise { - return callJudge(`You are evaluating documentation quality for an AI coding agent's CLI tool reference. - -The agent reads this documentation to learn how to use a headless browser CLI. It needs to: -1. Understand what each command does -2. Know what arguments to pass -3. Know valid values for enum-like parameters -4. Construct correct command invocations without guessing - -Rate the following ${section} on three dimensions (1-5 scale): - -- **clarity** (1-5): Can an agent understand what each command/flag does from the description alone? -- **completeness** (1-5): Are arguments, valid values, and important behaviors documented? Would an agent need to guess anything? -- **actionability** (1-5): Can an agent construct correct command invocations from this reference alone? - -Scoring guide: -- 5: Excellent — no ambiguity, all info present -- 4: Good — minor gaps an experienced agent could infer -- 3: Adequate — some guessing required -- 2: Poor — significant info missing -- 1: Unusable — agent would fail without external help - -Respond with ONLY valid JSON in this exact format: -{"clarity": N, "completeness": N, "actionability": N, "reasoning": "brief explanation"} - -Here is the ${section} to evaluate: - -${content}`, undefined, { jsonSchema: JUDGE_SCORE_SCHEMA }); -} - /** * Evaluate a QA report against planted-bug ground truth. * Returns detection metrics for the planted bugs. @@ -671,52 +628,6 @@ Respond with ONLY valid JSON: // Prompt builders live here so the W2 calibration harness builds its inputs // through the same functions the eval sends. -/** QA workflow quality judge (qa/SKILL.md workflow). */ -export function buildQaWorkflowJudgePrompt(section: string): string { - return `You are evaluating the quality of a QA testing workflow document for an AI coding agent. - -The agent reads this source-file bundle to select browser, native functional or mixed -surfaces, explore with bounded probes, reproduce and diagnose defects, add a regression -before repair, recheck behavior and report evidence/coverage. Sections are separate -files loaded only at their stated conditions; bundle order is not execution order. -Evaluate the complete workflow, including authority, isolation, native contracts, -conditional browser/DX loading and blocked paths, for clarity and executable decisions. - -Rate on three dimensions (1-5 scale): -- **clarity** (1-5): Can an agent follow the step-by-step phases without ambiguity? -- **completeness** (1-5): Are all phases, decision points, and outputs well-defined? -- **actionability** (1-5): Can an agent execute the workflow and produce the expected deliverables? - -Respond with ONLY valid JSON: -{"clarity": N, "completeness": N, "actionability": N, "reasoning": "brief explanation"} - -Here is the QA workflow to evaluate: - -${section}`; -} - -/** QA health score rubric judge (qa/SKILL.md health rubric). */ -export function buildQaHealthRubricJudgePrompt(section: string): string { - return `You are evaluating a health score rubric that an AI agent must follow to compute a numeric QA score. - -The agent uses this rubric after QA testing a website. It needs to: -1. Understand each scoring category and what counts as a deduction -2. Apply the weights correctly to compute a final score out of 100 -3. Produce a consistent, reproducible score - -Rate on three dimensions (1-5 scale): -- **clarity** (1-5): Are the categories, deduction criteria, and weights unambiguous? -- **completeness** (1-5): Are all edge cases and scoring boundaries defined? -- **actionability** (1-5): Can an agent compute a correct score from this rubric alone? - -Respond with ONLY valid JSON: -{"clarity": N, "completeness": N, "actionability": N, "reasoning": "brief explanation"} - -Here is the rubric to evaluate: - -${section}`; -} - /** QA anti-refusal judge (qa/SKILL.md anti-refusal). */ export function buildQaAntiRefusalJudgePrompt(diffAwareSection: string, rulesSection: string): string { return `You are evaluating whether a QA testing skill document would cause an AI agent to USE THE BROWSER or REFUSE to use the browser in a specific scenario. diff --git a/test/helpers/manual-judge-review-fixture.ts b/test/helpers/manual-judge-review-fixture.ts index 7a7874cd5..059e6b9f7 100644 --- a/test/helpers/manual-judge-review-fixture.ts +++ b/test/helpers/manual-judge-review-fixture.ts @@ -1,107 +1,6 @@ -import { readFileSync } from 'node:fs'; -import { createHash } from 'node:crypto'; -import { resolve } from 'node:path'; import type { EvalTestEntry } from './eval-store'; -import { buildCookieWorkflowJudgeInput } from './cookie-workflow-judge-input'; -import { COOKIE_MANUAL_REVIEW_FILE } from './cookie-workflow-manual-review'; +import entry from '../fixtures/manual-judge-review-entry.json'; -// Later documentation added to the judged BROWSER.md section after the approval -// was recorded (v1.91.4.0 Windows Opera wave). Reversing it reconstructs the -// exact historical prompt bytes, including the section's line range. -const LATER_BROWSER_BLOCK = /\*\*Windows: Opera and Opera GX\.\*\*[\s\S]*?appear in CLI output\.\n\n/; -const LATER_BROWSER_EDITS: Array<[string, string]> = [ - ['The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet and Arc.', 'The picker recognizes Chrome, Chromium, Brave, Edge, and macOS-only Comet, Arc, and Dia.'], - [' Opera and Opera GX are Windows-only and read from `%APPDATA%\\Opera Software\\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in.', ''], -]; - -// The judged SKILL.md excerpt follows the generated preamble, so later preamble -// edits move its line range without changing its bytes. The approved request -// named lines from 153; the parallel-waves one-line skill start (#2763) removed -// three preamble lines. A raw SKILL.md gets those lines back, and a built prompt, -// which has no preamble, gets the approved start line. -const APPROVED_ENTRYPOINT_START = 153; -const LATER_PREAMBLE_EDITS: Array<[string, string]> = [ - ['~/.claude/skills/gstack/bin/gstack-skill-start --skill "setup-browser-cookies" --model "claude"\n', '_SS="$HOME/.claude/skills/gstack/bin/gstack-skill-start"\n[ -x "$_SS" ] || _SS=".claude/skills/gstack/bin/gstack-skill-start"\n"$_SS" --skill "setup-browser-cookies" --model "claude" --parent-pid "$PPID" \\\n || echo "SKILL_START: unavailable — stale install; run ./setup or /gstack-upgrade (preamble degraded, continue the user\'s task)"\n'], -]; - -// The v1.91.19.0 generated-bash lint (INV-3) gave the shared Bun-install block a -// ${TMPDIR:-/tmp} mktemp template; same line, different bytes. -// The v1.91.30.0 Bun floor (E1) pinned the install hint to the tested Bun. -const LATER_WAVE_EDITS: Array<[string, string]> = [ - ['tmpfile=$(mktemp "${TMPDIR:-/tmp}/bun-install.XXXXXX")', 'tmpfile=$(mktemp)'], - ['BUN_VERSION="1.4.2"', 'BUN_VERSION="1.3.10"'], - // Oct 7 wave: the artifacts-sync preamble block surfaces attention lines (one line longer). - ["Skill-start already ran artifacts sync. GBrain hint text (if any) says\nwhen to prefer `gbrain` over Grep. `ARTIFACTS_SYNC:` reports sync health\n(`off`, `mode=... | queue=N`, `remote-mode`, or a `gstack-brain-restore`\nhint). On an `attention:` line, tell the user in one sentence what\nit says and the command it names, then continue.\n\nThe one-time privacy stop-gate arrives as a `GSTACK_INSTRUCTION` block\nfrom skill-start when consent is pending; fire it via AskUserQuestion\nexactly as instructed.", - "The skill-start output above already ran artifacts sync. Act on its lines:\nGBrain hint text (if present) tells you when to prefer `gbrain` over Grep;\n`ARTIFACTS_SYNC:` reports sync health (`off`, `mode=... | queue=N`,\n`remote-mode`, or a restore hint naming `gstack-brain-restore`).\n\nThe one-time privacy stop-gate (artifacts-sync consent) arrives as a\n`GSTACK_INSTRUCTION` block from skill-start when consent is actually pending\n\u2014 fire it via AskUserQuestion exactly as the block instructs."], - // Oct 8 wave: the tier-1 voice directive gained the reply-language rule (two lines longer). - ["Reply in the language of the user's latest message unless asked otherwise. Code, commands, paths, identifiers and quoted output stay verbatim.\n\n", ''], - // Dia removal (GSTA-143) dropped its platform sentence; same line count. - ["\nKeychain approval is the user's choice.", "\nDia is macOS-only. Keychain approval is the user's choice."], - // #3129 dropped the Claude body's "When to invoke" section above the preamble (seven lines shorter). - ['\n\n## Preamble (run first)\n', '\n\n\n## When to invoke this skill\n\nOpens an interactive picker UI where you select which cookie domains to import.\nUse before QA testing authenticated pages. Use when asked to "import cookies",\n"login to the site", or "authenticate the browser".\n\n## Preamble (run first)\n'], -]; - -// The honest-install wave (lazy Chromium) made the fallback probe call -// gstack-browser-ensure, explained it in one sentence inside the entrypoint -// (two lines longer), and pinned the Bun install hint by positional argument. -const LATER_ENTRYPOINT_BLOCK = /`gstack-browser-ensure` installs Chromium on first use[^\n]*\n\n/; -const LATER_ENTRYPOINT_EDITS: Array<[string, string]> = [ - ['if [ -x "$B" ] && "${B%/browse/*}/bin/gstack-browser-ensure"; then', 'if [ -x "$B" ]; then'], - ['bash "$tmpfile" "bun-v$BUN_VERSION"', 'BUN_VERSION="$BUN_VERSION" bash "$tmpfile"'], -]; - -// The GSTA-49 cookie-workflow repair rewrote the entrypoint's steps 1-5 (split -// the two probe READY outcomes, dropped the duplicate SETUP block for the browse -// fallback, moved source selection before the target, asked about verification -// before the daemon starts, added a report shape) and one BROWSER.md comment. -// Reversing it first restores the pre-repair text, which the edits below then -// take back further. -const LATER_COOKIE_ORDER_EDITS: Array<[string, string]> = [ - ["## 1. Check for Aside\n\nUse this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe.\n\n- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill.\n- **Any other result:** continue with gstack's own headless browser (`$B`).\n\n## 2. Get every decision from the user first\n\nAsk these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent.\n\n1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty.\n2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL.\n3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately.\n4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back.\n5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage.\n\nUse the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains.\n\n## 3. Prepare gstack's browser\n\n1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values.\n2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`.\n3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill.\n4. Open the target URL from answer 2 and leave that tab alone until the import finishes:\n\n```bash\n$B goto \"\"\n```\n\n## 4. Import\n\nPicker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires):\n\n```bash\n$B cookie-import-browser\n```\n\nDirect import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2.\n\nIf the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup.\n\n## 5. Report honestly\n\nReport from the import receipt, not from raw `$B cookies`, as three lines:\n\n- `Imported: (, as the receipt says) from for `\n- `Storage reset: `\n- `Sign-in: `\n\n**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab.\n\nNever request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs.\n\n## Platform boundaries", "## 1. Choose the browser\n\nUse this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its probe first. On `READY`, stop importing: use Aside's sessions or ask the user to sign in there. Otherwise follow the probe's fallback handling, then continue below.\n\n## SETUP (run this check BEFORE any browse command)\n\n```bash\n_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)\nB=\"\"\n[ -n \"$_ROOT\" ] && [ -x \"$_ROOT/.claude/skills/gstack/browse/dist/browse\" ] && B=\"$_ROOT/.claude/skills/gstack/browse/dist/browse\"\n[ -z \"$B\" ] && B=\"$HOME/.claude/skills/gstack/browse/dist/browse\"\nif [ -x \"$B\" ] && \"${B%/browse/*}/bin/gstack-browser-ensure\"; then\n echo \"READY: $B\"\nelse\n echo \"NEEDS_SETUP\"\nfi\n```\n\n`gstack-browser-ensure` installs Chromium on first use when the install skipped it (lazy browser); it prints `BROWSER_OK` or `BROWSER_UNAVAILABLE ` first.\n\nIf `NEEDS_SETUP`:\n1. Tell the user: \"gstack browse needs a one-time build (~10 seconds). OK to proceed?\" Then STOP and wait.\n2. Run: `cd && ./setup`\n3. If `bun` is not installed:\n ```bash\n if ! command -v bun >/dev/null 2>&1; then\n BUN_VERSION=\"1.4.2\"\n BUN_INSTALL_SHA=\"bab8acfb046aac8c72407bdcce903957665d655d7acaa3e11c7c4616beae68dd\"\n tmpfile=$(mktemp \"${TMPDIR:-/tmp}/bun-install.XXXXXX\")\n curl -fsSL \"https://bun.sh/install\" -o \"$tmpfile\"\n # shasum is macOS/perl; coreutils-only Linux ships sha256sum instead —\n # resolve whichever exists so the verify never fails on a missing tool.\n if command -v sha256sum >/dev/null 2>&1; then\n actual_sha=$(sha256sum < \"$tmpfile\" | awk '{print $(1)}')\n else\n actual_sha=$(shasum -a 256 < \"$tmpfile\" | awk '{print $(1)}')\n fi\n if [ \"$actual_sha\" != \"$BUN_INSTALL_SHA\" ]; then\n echo \"ERROR: bun install script checksum mismatch\" >&2\n echo \" expected: $BUN_INSTALL_SHA\" >&2\n echo \" got: $actual_sha\" >&2\n rm \"$tmpfile\"; exit 1\n fi\n bash \"$tmpfile\" \"bun-v$BUN_VERSION\"\n rm \"$tmpfile\"\n fi\n ```\n\n```bash\n$B status\n```\nIf status says `Mode: cdp`, stop: the real browser already has sessions.\n\n## 2. Confirm options before import\n\nOpen the known target and keep its tab unchanged. Both options default off and require explicit request:\n\n- **`--verify-auth` / picker checkbox:** reloads the target. Have the user privately configure daemon `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` **before startup**. Never invent values or assume CLI env reconfigures an existing daemon. Missing config rejects before mutation. Require a successful same-origin response and exactly one visible element whose whitespace-normalized text exactly matches the expected identity.\n- **`--clear-storage`:** for suspected stale storage, obtain explicit approval. Chromium only: clears captured-origin localStorage (shared across same-origin context tabs) and target-tab sessionStorage. Other origins, other tabs' sessionStorage, IndexedDB, and service workers stay intact. It uses an isolated world/native deadline; other engines reject reset, not imports/auth checks. Never auto-approve or claim rollback after partial failure.\n\n## 3. Select source and scope\n\n```bash\n$B cookie-import-browser\n```\n\nAsk the user to choose browser, account/profile, and domains, then say when done. Never guess accounts or treat default Comet as consent. Unreadable profiles are unknown, not empty. Rerun for an expired five-minute one-use link.\n\nDirect import: pass the chosen browser and `--domain` after navigating to a matching target. `--profile` takes a directory, not a display name; omit only for an unambiguous relevant profile, otherwise use the picker. `--all` requires consent for all non-expired profile cookies; it cannot accompany `--domain` or `--clear-storage`.\n\nRead that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for examples, profile labels, supported sources and platform setup.\n\n## 4. Report honestly\n\nReport receipt/picker counts, partial/zero/error and reset outcomes, not raw `$B cookies`. Imports affect the context, not one tab. **Not checked** means no requested check; **not verified** means it failed; **verified** requires positive target evidence. Zero imports, counts, or HTTP 200 never prove login.\n\nNever request/publish cookie values, passwords, identity/profile text, session details, or raw errors in public logs.\n\n## Platform boundaries"], - ["# view the page; only --verify-auth proves sign-in", "# confirm the intended account"], -]; - -// The #3338 trust warning (GSTA-84) added one paragraph after step 3's goto -// (two lines longer). Removing it first lets the GSTA-49 reversal match. -const LATER_TRUST_PARAGRAPH = "Tell the user once, before the import: gstack's headless browser trusts the root store built into gstack's bundled Chromium, which still accepts certificates from the compromised eMudhra intermediates. Until gstack moves to a Playwright release that drops them, don't use the imported sessions on a network the user does not trust. That same root's `BROWSER.md`, **TLS trust: the bundled Chromium's root store**, has the details and the check.\n\n"; - -export function approvedCookieWorkflowSource(source: string): string { - let removedLines = 0; - let removedEntrypointLines = 0; - let reordered = source; - if (reordered.includes(LATER_TRUST_PARAGRAPH)) { - removedEntrypointLines += LATER_TRUST_PARAGRAPH.split('\n').length - 1; - reordered = reordered.replace(LATER_TRUST_PARAGRAPH, ''); - } - for (const [later, earlier] of LATER_COOKIE_ORDER_EDITS) { - if (!reordered.includes(later)) continue; - removedEntrypointLines += later.split('\n').length - earlier.split('\n').length; - reordered = reordered.replace(later, earlier); - } - let historical = reordered - .replace('sha256sum < "$tmpfile" | awk \'{print $(1)}\'', 'sha256sum "$tmpfile" | awk \'{print $1}\'') - .replace('shasum -a 256 < "$tmpfile" | awk \'{print $(1)}\'', 'shasum -a 256 "$tmpfile" | awk \'{print $1}\'') - .replace(LATER_BROWSER_BLOCK, block => { removedLines = block.split('\n').length - 1; return ''; }) - .replace(LATER_ENTRYPOINT_BLOCK, block => { removedEntrypointLines += block.split('\n').length - 1; return ''; }); - for (const [later, earlier] of [...LATER_BROWSER_EDITS, ...LATER_PREAMBLE_EDITS, ...LATER_WAVE_EDITS, ...LATER_ENTRYPOINT_EDITS]) historical = historical.replace(later, earlier); - return historical - .replace(/(--- BEGIN FILE "BROWSER\.md" \(lines \d+-)(\d+)(; section\) ---)/, (_, head, end, tail) => `${head}${Number(end) - removedLines}${tail}`) - .replace(/(--- BEGIN FILE "setup-browser-cookies\/SKILL\.md" \(lines )(\d+)-(\d+)(; entrypoint\) ---)/, (_, head, start, end, tail) => `${head}${APPROVED_ENTRYPOINT_START}-${APPROVED_ENTRYPOINT_START + Number(end) - Number(start) - removedEntrypointLines}${tail}`); -} - -export function manualReviewFixture(root = resolve(import.meta.dir, '../..')): EvalTestEntry { - const approval = JSON.parse(readFileSync(resolve(root, COOKIE_MANUAL_REVIEW_FILE), 'utf8')); - const prompt = approvedCookieWorkflowSource(buildCookieWorkflowJudgeInput(root).prompt); - if (createHash('sha256').update(prompt).digest('hex') !== approval.prompt_sha256 - || Buffer.byteLength(prompt) !== approval.prompt_bytes) { - throw new Error('Historical cookie approval fixture no longer reconstructs the exact approved prompt'); - } - return { - name: 'setup-browser-cookies/SKILL.md workflow', suite: 'Cookie setup workflow quality', tier: 'llm-judge', - passed: false, execution: 'executed', exit_reason: 'provider_refusal', attempt: 1, duration_ms: 1, cost_usd: 0, - model: approval.model, prompt, - error: 'Synthetic provider refusal fixture, not live model evidence', - manual_review: { approval, refusal: { stop_reason: 'refusal', response_id: 'msg_synthetic_fixture', - request_id: 'req_synthetic_fixture', model: approval.model, input_tokens: 1, output_tokens: 0, text_blocks: 0 } }, - }; +export function manualReviewFixture(): EvalTestEntry { + return structuredClone(entry) as EvalTestEntry; } diff --git a/test/helpers/touchfiles-data.ts b/test/helpers/touchfiles-data.ts index 46431efac..07e2649af 100644 --- a/test/helpers/touchfiles-data.ts +++ b/test/helpers/touchfiles-data.ts @@ -1664,60 +1664,11 @@ export const E2E_TIERS: Record = { * LLM-judge test touchfiles — keyed by test description string. */ export const LLM_JUDGE_TOUCHFILES: Record = { - 'review/SKILL.md workflow': ['review/**', 'qa/**', 'scripts/resolvers/qa.ts', 'scripts/resolvers/review-army.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/sections.ts', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/skill-startup.ts'], - 'setup-browser-cookies/SKILL.md workflow': ['setup-browser-cookies/SKILL.md.tmpl', 'setup-browser-cookies/SKILL.md', 'BROWSER.md', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/manual-judge-review-fixture.ts', '.github/cookie-workflow-manual-review.json', 'test/helpers/workflow-judge-input.ts', 'test/skill-llm-eval.test.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/workflow-judge-cache.ts', 'test/helpers/skill-startup.ts'], - 'browse/SKILL.md reference': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'browse/sections/**', 'browse/SKILL.md', 'browse/SKILL.md.tmpl', 'browse/src/**', 'test/skill-llm-eval.test.ts', 'SKILL.md', 'SKILL.md.tmpl', 'gstack/llms.txt', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/workflow-judge-cache.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/skill-startup.ts'], - 'setup block': ['browse/SKILL.md', 'browse/SKILL.md.tmpl', 'scripts/resolvers/aside.ts', 'scripts/resolvers/browse.ts', 'test/skill-llm-eval.test.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/workflow-judge-cache.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/skill-startup.ts'], - 'qa/SKILL.md workflow': ['qa/**', 'qa/sections/**', 'qa/SKILL.md', 'qa/SKILL.md.tmpl', 'scripts/resolvers/qa.ts', 'scripts/resolvers/utility.ts', 'scripts/resolvers/sections.ts', 'test/helpers/workflow-judge-input.ts', 'test/skill-llm-eval.test.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/workflow-judge-cache.ts', 'test/helpers/skill-startup.ts'], - 'qa/SKILL.md health rubric': ['qa/sections/**', 'qa/SKILL.md', 'qa/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/workflow-judge-cache.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/skill-startup.ts'], - 'qa/SKILL.md anti-refusal': ['qa/sections/**', 'qa/SKILL.md', 'qa/SKILL.md.tmpl', 'qa-only/SKILL.md', 'qa-only/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/workflow-judge-cache.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/skill-startup.ts'], - 'cross-skill greptile consistency': ['review/SKILL.md', 'review/SKILL.md.tmpl', 'ship/SKILL.md', 'ship/SKILL.md.tmpl', 'review/greptile-triage.md', 'retro/SKILL.md', 'retro/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/workflow-judge-cache.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/skill-startup.ts'], - - // Ship & Release - 'ship/SKILL.md workflow': ['ship/SKILL.md', 'ship/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', - 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', - 'scripts/resolvers/testing.ts', 'ship/sections/**', 'qa/**', 'scripts/resolvers/qa.ts', 'scripts/resolvers/review-army.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'document-release/SKILL.md workflow': ['document-release/**', 'document-release/SKILL.md', 'document-release/SKILL.md.tmpl', 'scripts/resolvers/sections.ts', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - - // Plan Reviews - 'plan-ceo-review/SKILL.md modes': [ 'ETHOS.md','plan-ceo-review/sections/**', 'plan-ceo-review/SKILL.md', 'plan-ceo-review/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', - 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'plan-eng-review/SKILL.md sections': [ - 'scripts/resolvers/gbrain.ts', - 'scripts/resolvers/learnings.ts', - - "test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/SKILL.md', 'plan-eng-review/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', - "scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts", - 'scripts/resolvers/testing.ts', 'plan-eng-review/sections/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - - // /spec authored-spec quality (paid LLM-judge — periodic-tier). - 'plan-design-review/SKILL.md passes': [ - - "test/fixtures/plan-scope-recovery-av.json", - "test/fixtures/design-scope-checkpoint-at.json",'plan-design-review/SKILL.md', 'plan-design-review/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', - - - "scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts", - 'plan-design-review/sections/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/design.ts', 'scripts/resolvers/design-craft.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - - // Design skills - 'design-review/SKILL.md fix loop': ['design-review/SKILL.md', 'design-review/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'design-consultation/SKILL.md research': ['design-consultation/SKILL.md', 'design-consultation/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'scripts/resolvers/design.ts', 'scripts/resolvers/outside-voice.ts', 'design-consultation/sections/**', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - - // Deploy skills - 'land-and-deploy/SKILL.md workflow': ['land-and-deploy/SKILL.md', 'land-and-deploy/SKILL.md.tmpl', 'land-and-deploy/sections/**', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'canary/SKILL.md monitoring loop': ['canary/SKILL.md', 'canary/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'benchmark/SKILL.md perf collection': ['benchmark/SKILL.md', 'benchmark/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'setup-deploy/SKILL.md platform setup': ['setup-deploy/SKILL.md', 'setup-deploy/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - - // Other skills - 'retro/SKILL.md instructions': ['retro/sections/**', 'retro/SKILL.md', 'retro/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'qa-only/SKILL.md workflow': ['qa-only/**', 'qa-only/SKILL.md', 'qa-only/SKILL.md.tmpl', 'qa/**', 'scripts/resolvers/qa.ts', 'scripts/resolvers/utility.ts', 'scripts/resolvers/sections.ts', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'gstack-upgrade/SKILL.md upgrade flow': ['gstack-upgrade/SKILL.md', 'gstack-upgrade/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/helpers/workflow-excerpt.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], - 'sync-gbrain/SKILL.md read-only readiness': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'sync-gbrain/SKILL.md', 'sync-gbrain/SKILL.md.tmpl', 'bin/gstack-gbrain-read-capability.ts', 'test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/workflow-judge-cache.ts', 'scripts/eval-input-cache.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/skill-startup.ts'], + 'qa/SKILL.md anti-refusal': ['qa/sections/**', 'qa/SKILL.md', 'qa/SKILL.md.tmpl', 'qa-only/SKILL.md', 'qa-only/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], + 'cross-skill greptile consistency': ['review/SKILL.md', 'review/SKILL.md.tmpl', 'ship/SKILL.md', 'ship/SKILL.md.tmpl', 'review/greptile-triage.md', 'retro/SKILL.md', 'retro/SKILL.md.tmpl', 'test/skill-llm-eval.test.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], // Voice directive - 'voice directive tone': ['scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'review/SKILL.md', 'review/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts', 'test/skill-llm-eval.test.ts', 'test/fixtures/eval-baselines.json', 'test/helpers/cookie-workflow-judge-input.ts', 'test/helpers/cookie-workflow-manual-review.ts', 'test/helpers/llm-judge.ts', 'test/helpers/workflow-excerpt.ts', 'test/helpers/workflow-judge-cache.ts', 'test/helpers/workflow-judge-input.ts', 'test/helpers/skill-startup.ts'], + 'voice directive tone': ['scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'review/SKILL.md', 'review/SKILL.md.tmpl', 'scripts/gen-skill-docs.ts', 'test/skill-llm-eval.test.ts', 'test/helpers/llm-judge.ts', 'test/helpers/skill-startup.ts'], }; /** @@ -2044,29 +1995,8 @@ export const E2E_KINDS: Record = { 'router-failure-contradicted-cause': 'rule', 'router-failure-triage-resolves': 'rule', 'router-failure-triage-unclear': 'rule', - 'review/SKILL.md workflow': 'judge', - 'setup-browser-cookies/SKILL.md workflow': 'judge', - 'browse/SKILL.md reference': 'judge', - 'setup block': 'judge', - 'qa/SKILL.md workflow': 'judge', - 'qa/SKILL.md health rubric': 'judge', 'qa/SKILL.md anti-refusal': 'judge', 'cross-skill greptile consistency': 'judge', - 'ship/SKILL.md workflow': 'judge', - 'document-release/SKILL.md workflow': 'judge', - 'plan-ceo-review/SKILL.md modes': 'judge', - 'plan-eng-review/SKILL.md sections': 'judge', - 'plan-design-review/SKILL.md passes': 'judge', - 'design-review/SKILL.md fix loop': 'judge', - 'design-consultation/SKILL.md research': 'judge', - 'land-and-deploy/SKILL.md workflow': 'judge', - 'canary/SKILL.md monitoring loop': 'judge', - 'benchmark/SKILL.md perf collection': 'judge', - 'setup-deploy/SKILL.md platform setup': 'judge', - 'retro/SKILL.md instructions': 'judge', - 'qa-only/SKILL.md workflow': 'judge', - 'gstack-upgrade/SKILL.md upgrade flow': 'judge', - 'sync-gbrain/SKILL.md read-only readiness': 'judge', 'voice directive tone': 'judge', }; diff --git a/test/helpers/workflow-judge-cache.ts b/test/helpers/workflow-judge-cache.ts deleted file mode 100644 index f5f603163..000000000 --- a/test/helpers/workflow-judge-cache.ts +++ /dev/null @@ -1,153 +0,0 @@ -/** Audited cache adapter for runWorkflowJudge only. Native/PTY evals stay fresh. */ -import * as fs from 'node:fs'; -import * as path from 'node:path'; -import { spawnSync } from 'node:child_process'; -import { DEFAULT_JUDGE_MAX_TOKENS, resolveEvalModel } from '../../lib/eval-model'; -import { JUDGE_MS } from './eval-budgets'; -import { JUDGE_PANEL_SAMPLES, JUDGE_SCORE_DIMENSIONS, judgePanelMedian, type JudgeScore } from './llm-judge'; -import { readWorkflowJudgeInput, buildWorkflowJudgePrompt, WORKFLOW_JUDGE_RESPONSE_SCHEMA, WORKFLOW_JUDGE_REASONING_WORD_LIMIT } from './workflow-judge-input'; -import { buildEvalInputIdentity, lookupEvalInputCache, sourceDependencyClosure, storeEvalInputCache, - type EvalCacheValue, type EvalInputIdentity, type EvalPassingProof } from '../../scripts/eval-input-cache'; - -type Thresholds = { clarity: number; completeness: number; actionability: number }; -export interface WorkflowCacheOptions { - root: string; testName: string; skillPath: string; startMarker: string; endMarker: string | RegExp | null; - judgeContext: string; judgeGoal: string; model?: string; thresholds: Thresholds; prompt: string; attempt: number; - references?: readonly string[]; - agentCapability?: 'frontier'; - /** Sends WORKFLOW_JUDGE_RESPONSE_SCHEMA as the structured-output format. */ - schemaTransport?: boolean; - /** Validates the compact response contract: exact keys, non-empty reasoning under the word limit. */ - compactReasoning?: boolean; - maxTokens?: number; - stream?: boolean; - effort?: 'medium'; - env?: NodeJS.ProcessEnv; -} -export interface WorkflowJudgeReuse { - key: string; source: EvalPassingProof['source']; -} - -/** The judge's audited closure: its runner, rubric and documents, installed SDK bytes included. */ -export function workflowJudgeDependencies(root: string, documents: string[]): string[] { - return sourceDependencyClosure(root, ['test/skill-llm-eval.test.ts', 'test/helpers/workflow-judge-cache.ts', - 'test/helpers/llm-judge.ts', 'lib/eval-model.ts', 'test/helpers/eval-budgets.ts', - 'scripts/test-paid-shards.ts', 'scripts/test-strict-output.ts', 'scripts/eval-select.ts', - 'scripts/test-pr-profile.ts', '.github/workflows/evals.yml', - 'package.json', 'bun.lock', '.github/docker/Dockerfile.ci', ...documents]); -} - -export function validWorkflowJudgeScore(value: EvalCacheValue, thresholds: Thresholds, compactReasoning = false): value is JudgeScore & EvalCacheValue { - if (!value || typeof value !== 'object' || Array.isArray(value) - || Object.keys(value).sort().join(',') !== 'actionability,clarity,completeness,reasoning' - || typeof value.reasoning !== 'string' - || (compactReasoning && (!value.reasoning.trim() - || value.reasoning.trim().split(/\s+/).length >= WORKFLOW_JUDGE_REASONING_WORD_LIMIT))) return false; - return JUDGE_SCORE_DIMENSIONS.every(key => - typeof value[key] === 'number' && Number.isInteger(value[key]) && value[key] >= thresholds[key] && value[key] <= 5); -} - -const SAMPLE_RANGE: Thresholds = { clarity: 1, completeness: 1, actionability: 1 }; - -/** A complete judge panel: exactly JUDGE_PANEL_SAMPLES of valid samples whose per-dimension median (2 of 3) meets every threshold. */ -export function validWorkflowJudgePanel(value: EvalCacheValue, thresholds: Thresholds, compactReasoning = false): value is { samples: Array } { - if (!value || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).join(',') !== 'samples' - || !Array.isArray(value.samples) || value.samples.length !== JUDGE_PANEL_SAMPLES - || !value.samples.every(sample => validWorkflowJudgeScore(sample, SAMPLE_RANGE, compactReasoning))) return false; - const median = judgePanelMedian(value.samples as JudgeScore[], JUDGE_SCORE_DIMENSIONS); - return JUDGE_SCORE_DIMENSIONS.every(key => median[key] >= thresholds[key]); -} - -export function prepareWorkflowJudgeCache(opts: WorkflowCacheOptions): { - lookup(): { samples: JudgeScore[]; reuse: WorkflowJudgeReuse } | null; - /** The attempt guard is rechecked after synchronous input/provenance reads. */ - publish(samples: JudgeScore[], isActive?: () => boolean): (() => void) | undefined; -} { - const env = opts.env ?? process.env; - const noCache = { lookup: () => null, publish: (_samples: JudgeScore[]) => undefined }; - const pr = Number(env.EVALS_CACHE_PR); - // Runtime ID is the immutable CI image manifest, not a mutable image tag. - // Nonstandard Node/Bun preload code or custom model endpoints need a separate - // audited adapter: they can change the request outside the consumed source. - if (!env.EVALS_CACHE_DIR || !env.EVALS_CACHE_REPOSITORY || !Number.isSafeInteger(pr) || pr <= 0 - || !/^(?:sha256:)?[a-f0-9]{64}$/.test(env.EVALS_CACHE_RUNTIME_ID ?? '') - || env.EVALS_TIER !== 'gate' || env.EVALS_FRESH === '1' - || ['release', 'periodic'].includes(env.EVALS_CACHE_PURPOSE ?? '') - || opts.attempt !== 1 || env.NODE_OPTIONS || env.BUN_OPTIONS - || (env.ANTHROPIC_BASE_URL && env.ANTHROPIC_BASE_URL !== 'https://api.anthropic.com')) return noCache; - - const currentIdentity = (): EvalInputIdentity | null => { - try { - const input = readWorkflowJudgeInput(opts); - const prompt = buildWorkflowJudgePrompt(opts, input); - // Never substitute this adapter's interpretation for the actual API input. - if (prompt !== opts.prompt) return null; - const { version: _releaseLabel, ...rootPackage } = JSON.parse(fs.readFileSync(path.join(opts.root, 'package.json'), 'utf8')); - const identity = buildEvalInputIdentity({ root: opts.root, - scope: { repository: env.EVALS_CACHE_REPOSITORY!, pullRequest: pr }, - coverage: { dependencies: 'complete', prompts: 'complete', environment: 'complete' }, unknownDependencies: [], - files: workflowJudgeDependencies(opts.root, input.files.map(file => file.path)), - prompts: { [opts.testName]: prompt }, - parameters: { rootPackage, thresholds: opts.thresholds, max_tokens: opts.maxTokens ?? DEFAULT_JUDGE_MAX_TOKENS, temperature: null, budget_ms: JUDGE_MS, - request: opts.stream ? 'messages.stream/user' : 'messages.create/user', retries: 0, - panel: { samples: JUDGE_PANEL_SAMPLES, numeric: 'mean', boolean: 'majority' }, - ...(opts.stream ? { stream: true } : {}), - ...(opts.effort ? { effort: opts.effort } : {}), - ...(opts.schemaTransport ? { output_config: { format: { type: 'json_schema', schema: WORKFLOW_JUDGE_RESPONSE_SCHEMA } } } : {}), - ...(opts.compactReasoning ? { response_validation: { reasoning_words_below: WORKFLOW_JUDGE_REASONING_WORD_LIMIT } } : {}) }, - runtime: { image: env.EVALS_CACHE_RUNTIME_ID!, bun: Bun.version, node: process.versions.node, - platform: process.platform, arch: process.arch, judge: resolveEvalModel('judge', opts.model, env), - anthropic_base_url: env.ANTHROPIC_BASE_URL ?? 'https://api.anthropic.com', - anthropic_log: env.ANTHROPIC_LOG ?? null, - proxies: Object.fromEntries(['HTTP_PROXY', 'HTTPS_PROXY', 'ALL_PROXY', 'NO_PROXY', 'http_proxy', 'https_proxy', 'all_proxy', 'no_proxy'] - .map(name => [name, env[name] ?? null])) }, - }); - return identity.status === 'eligible' ? identity.identity : null; - } catch { return null; } - }; - const before = currentIdentity(); - if (!before) return noCache; - const common = { cacheDir: env.EVALS_CACHE_DIR, purpose: 'gate' as const }; - return { - lookup() { - const result = lookupEvalInputCache({ ...common, identity: before, - validateResult: value => validWorkflowJudgePanel(value, opts.thresholds, opts.compactReasoning) }); - return result.status === 'reused' - ? { samples: (result.result as unknown as { samples: JudgeScore[] }).samples, reuse: { key: result.key, source: result.source } } : null; - }, - publish(samples, isActive = () => true) { - // Caller reaches here ONLY after its actual assertions passed. A later - // failed case in the file does not erase this independently completed case. - const panel = { samples: samples.map(({ clarity, completeness, actionability, reasoning }) => ({ clarity, completeness, actionability, reasoning })) }; - if (!isActive() || !validWorkflowJudgePanel(panel as unknown as EvalCacheValue, opts.thresholds, opts.compactReasoning)) return; - const after = currentIdentity(); - const runId = env.GITHUB_RUN_ID ? `${env.GITHUB_RUN_ID}/${env.GITHUB_RUN_ATTEMPT ?? '1'}` : env.EVALS_RUN_ID; - if (!after || !runId || !isActive()) return; - const revision = spawnSync('git', ['rev-parse', 'HEAD'], { cwd: opts.root, encoding: 'utf8', timeout: 3000 }); - if (revision.status !== 0 || !isActive()) return; - const stored = storeEvalInputCache({ ...common, before, after, proof: { - execution: 'new', finalized: true, completeAttemptHistory: true, exitCode: 0, timedOut: false, - cancelled: false, skipped: 0, failed: 0, passed: 1, - cases: [{ id: opts.testName, outcome: 'passed', attempt: 1 }], - source: { runId, revision: revision.stdout.trim(), completedAt: Date.now() }, - result: panel, - } }); - // A slow synchronous write can consume the recording allowance. The - // caller withdraws this new receipt if its final deadline check fails. - if (stored.status === 'stored') return () => fs.rmSync(path.join(common.cacheDir, `${stored.key}.json`), { force: true }); - }, - }; -} - -/** - * Pass floors for the browse reference judge panel, applied to its gating - * per-dimension median (EVAL_POLICY v3). The stored baseline is recorded for - * comparison only: a three-sample aggregate is too noisy for a no-dip ratchet, - * and gating on it silently raised the clarity floor to 4. - */ -export const BROWSE_JUDGE_FLOORS = { clarity: 3, completeness: 4, actionability: 4 } as const; - -export function browseJudgeFloorsMet(scores: Record): boolean { - return (Object.keys(BROWSE_JUDGE_FLOORS) as Array) - .every(dim => scores[dim] >= BROWSE_JUDGE_FLOORS[dim]); -} diff --git a/test/helpers/workflow-judge-input.ts b/test/helpers/workflow-judge-input.ts index f7d8070b1..d4e8bf344 100644 --- a/test/helpers/workflow-judge-input.ts +++ b/test/helpers/workflow-judge-input.ts @@ -1,4 +1,4 @@ -/** Preserve source-file boundaries when a workflow judge reads carved skills. */ +/** Read a carved skill as one bundle that preserves source-file boundaries. */ import * as fs from 'node:fs'; import * as path from 'node:path'; import { markerIndex } from './workflow-excerpt'; @@ -17,71 +17,6 @@ export interface WorkflowJudgeInput { text: string; } -export const QA_DISCOVERY_REFERENCES = [ - 'qa/sections/scope.md', - 'qa/sections/exploratory.md', - 'qa/sections/system-functional.md', - 'qa/sections/browser-setup.md', - 'qa/sections/qa-patterns.md', - 'qa/templates/functional-report-template.md', -]; - -export const WORKFLOW_JUDGE_REASONING_WORD_LIMIT = 150; -/** The instructed length sits below the enforced limit: judges asked for <150 landed at 130-156 words. */ -export const WORKFLOW_JUDGE_REASONING_WORD_TARGET = 120; - -export const WORKFLOW_JUDGE_RESPONSE_SCHEMA = { - type: 'object', - properties: { - clarity: { type: 'integer', enum: [1, 2, 3, 4, 5] }, - completeness: { type: 'integer', enum: [1, 2, 3, 4, 5] }, - actionability: { type: 'integer', enum: [1, 2, 3, 4, 5] }, - reasoning: { type: 'string', - description: `Under ${WORKFLOW_JUDGE_REASONING_WORD_TARGET} words with at most two decisive examples, evaluating the complete supplied workflow.` }, - }, - required: ['clarity', 'completeness', 'actionability', 'reasoning'], - additionalProperties: false, -}; - -export function buildWorkflowJudgePrompt(opts: { - judgeContext: string; - judgeGoal: string; - agentCapability?: 'frontier'; -}, input: WorkflowJudgeInput): string { - return `You are evaluating the quality of ${opts.judgeContext} for an AI coding agent. - -The agent reads these source files to learn ${opts.judgeGoal}. Shared preamble definitions and -external tools/files are documented separately; do not penalize their absence from this bundle. -On-demand sections retain their original file boundaries and Read instructions; the section -index refers to those files, not duplicate work. The bundle order is not execution order. -Judge the actual instructions, including contradictory ordering or missing decisions.${opts.agentCapability === 'frontier' ? ` - -Target reader: a frontier coding agent with GPT-5.6 Sol-level capability or stronger. -Assume it can follow explicit cross-references, track saved state and a bounded work list, -and distinguish conditional branches. Length, technical vocabulary and multiple explicit recovery paths alone are not clarity defects. -Do not invent missing policies, permissions or evidence to make a workflow executable. - -Clarity 4 means the target agent can determine the next permitted action on each applicable path; -5 additionally means those paths are easy to locate and understand. -Score clarity 3 or lower when execution still requires guessing because of -conflicting order, undefined decisions, unclear authority or missing input/output handling. -Evaluate the whole workflow, but keep the JSON reasoning under ${WORKFLOW_JUDGE_REASONING_WORD_TARGET} words with at most two decisive examples. -For a clarity defect, cite the specific file/step and explain the competing actions or missing decision. -Keep completeness and actionability independent: reader capability does not supply missing requirements.` : ''} - -Rate on three dimensions (1-5 scale): -- **clarity** (1-5): Can an agent follow the instructions without ambiguity? -- **completeness** (1-5): Are all steps, decision points, and outputs well-defined? -- **actionability** (1-5): Can an agent execute this workflow and produce the expected deliverables? - -Respond with ONLY valid JSON: -{"clarity": N, "completeness": N, "actionability": N, "reasoning": "brief explanation"} - -Here is the source-file bundle to evaluate: - -${input.text}`; -} - export function readWorkflowJudgeInput(opts: { root: string; skillPath: string; @@ -94,10 +29,10 @@ export function readWorkflowJudgeInput(opts: { kind: 'entrypoint' as const, content: readSkillStartup(path.join(opts.root, opts.skillPath)), }]; - const sectionDir = path.join(path.dirname(opts.skillPath), 'sections'); + const sectionDir = path.posix.join(path.posix.dirname(opts.skillPath), 'sections'); const sectionRoot = path.join(opts.root, sectionDir); // Runner-profile sections (manifest `runner_only`, plan B4) are read by - // bin/gstack-autoplan alone; the judge evaluates the interactive workflow. + // bin/gstack-autoplan alone; the bundle holds the interactive workflow. const manifestPath = path.join(sectionRoot, 'manifest.json'); const manifestSections: Array<{ file: string; runner_only?: boolean }> = fs.existsSync(manifestPath) ? JSON.parse(fs.readFileSync(manifestPath, 'utf8')).sections ?? [] : []; @@ -105,7 +40,7 @@ export function readWorkflowJudgeInput(opts: { const sections = fs.existsSync(sectionRoot) ? fs.readdirSync(sectionRoot).sort().filter(name => name.endsWith('.md') && !runnerOnly.has(name) && name !== SHARED_PREAMBLE_FILE) .map(name => ({ - path: path.join(sectionDir, name), + path: path.posix.join(sectionDir, name), kind: 'section' as const, content: fs.readFileSync(path.join(sectionRoot, name), 'utf8'), })) @@ -129,8 +64,7 @@ export function readWorkflowJudgeInput(opts: { const files: WorkflowJudgeFile[] = []; let offset = 0; for (const file of allSources) { - // Every section was already supplied in full by the old judge input. Keep - // that coverage, but include each file once even when the marker window + // Every section is supplied in full. Include each file once even when the marker window // also covers part of it. Only the entrypoint retains the requested slice. const from = file.kind !== 'entrypoint' ? 0 : Math.max(0, start - offset); const to = file.kind !== 'entrypoint' ? file.content.length : Math.min(file.content.length, end - offset); diff --git a/test/judge-calibration.test.ts b/test/judge-calibration.test.ts index 4eca85c0d..91c6ffc69 100644 --- a/test/judge-calibration.test.ts +++ b/test/judge-calibration.test.ts @@ -4,7 +4,6 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { callJudge, JudgeRefusalError, type CallJudgeOptions, type JudgeResponseMeta } from './helpers/llm-judge'; -import { buildCookieWorkflowJudgeInput, COOKIE_WORKFLOW_JUDGE } from './helpers/cookie-workflow-judge-input'; import { analyze, classifyError, loadCorpus, undispatched, PANEL_SAMPLES, PHASES, planCalls, priceManifest, runCalls, runSample, type CalibrationConfig, type CorpusItem, type JudgeCall, type SampleRecord, @@ -202,8 +201,8 @@ describe('judge calibration harness (free, stubbed provider)', () => { describe('committed calibration configurations and corpora', () => { test('comparison (1) changes only the JSON schema; prompts come from the eval builders', async () => { const { CALIBRATION_CONFIGS } = await import('../scripts/judge-calibration-configs'); - expect(CALIBRATION_CONFIGS.map(config => config.id)).toEqual(['arm', 'qa-workflow', 'qa-health-rubric', 'qa-anti-refusal', 'cross-skill', 'voice', 'workflow-default', 'cookie-workflow']); - expect(CALIBRATION_CONFIGS.map(config => config.priority)).toEqual([1, 2, 3, 4, 5, 6, 7, 8]); + expect(CALIBRATION_CONFIGS.map(config => config.id)).toEqual(['arm', 'qa-anti-refusal', 'cross-skill', 'voice']); + expect(CALIBRATION_CONFIGS.map(config => config.priority)).toEqual([1, 4, 5, 6]); for (const config of CALIBRATION_CONFIGS) { const { jsonSchema, ...rest } = config.newOptions; expect(jsonSchema).toBeTruthy(); @@ -233,34 +232,9 @@ describe('committed calibration configurations and corpora', () => { } }); - // /ship Step 7 coverage audit (GSTA-91). - // Value: protects=the cookie-workflow calibration sends the eval's exact prompt for the same bundle; - // fails_when=its build drops COOKIE_WORKFLOW_JUDGE context/goal or adds a capability the eval does not send; - // why_new=the corpus test only checks the bundle is contained in the prompt; seam=none; - // no_claim=provider behavior (no request is sent; both prompts come from the live SKILL.md and BROWSER.md) - test('the cookie-workflow calibration builds the prompt the cookie eval sends', async () => { - const { CALIBRATION_CONFIGS } = await import('../scripts/judge-calibration-configs'); - const cookie = CALIBRATION_CONFIGS.find(config => config.id === 'cookie-workflow')!; - const production = buildCookieWorkflowJudgeInput(path.join(import.meta.dir, '..')); - expect(cookie.build({ bundle: production.text })).toBe(production.prompt); - }); - test('panel verdicts restate each eval pass rule', async () => { const { CALIBRATION_CONFIGS } = await import('../scripts/judge-calibration-configs'); const byId = Object.fromEntries(CALIBRATION_CONFIGS.map(config => [config.id, config])); - const qa = (clarity: number, completeness: number, actionability: number) => ({ clarity, completeness, actionability, reasoning: '' }); - for (const id of ['qa-workflow', 'qa-health-rubric', 'workflow-default']) { - expect(byId[id]!.verdict([qa(3, 3, 4), qa(3, 3, 4), qa(3, 3, 4)])).toBe('pass'); - expect(byId[id]!.verdict([qa(5, 5, 4), qa(5, 5, 4), qa(5, 5, 3)])).toBe('pass'); - expect(byId[id]!.verdict([qa(5, 5, 4), qa(5, 5, 3), qa(5, 5, 3)])).toBe('fail'); - expect(() => byId[id]!.validate({ ...qa(3, 3, 4), clarity: '3' })).toThrow(); - } - expect(byId['cookie-workflow']!.verdict([qa(4, 3, 4), qa(4, 3, 4), qa(4, 3, 4)])).toBe('pass'); - expect(byId['cookie-workflow']!.verdict([qa(5, 5, 4), qa(5, 5, 4), qa(3, 2, 3)])).toBe('pass'); - expect(byId['cookie-workflow']!.verdict([qa(4, 3, 4), qa(3, 3, 4), qa(3, 3, 4)])).toBe('fail'); - expect(byId['cookie-workflow']!.verdict([qa(5, 2, 5), qa(5, 2, 5), qa(5, 3, 5)])).toBe('fail'); - expect(byId['cookie-workflow']!.verdict([qa(5, 5, 4), qa(5, 5, 3), qa(5, 5, 3)])).toBe('fail'); - expect(() => byId['cookie-workflow']!.validate({ ...qa(4, 3, 4), clarity: '4' })).toThrow(); const arm = (over_engineering: number) => ({ over_engineering, construct: over_engineering ? 'helper' : 'none', reasoning: '' }); expect(byId.arm!.verdict([arm(0), arm(1), arm(3)])).toBe('pass'); expect(byId.arm!.verdict([arm(1), arm(2), arm(2)])).toBe('fail'); @@ -281,8 +255,6 @@ describe('landed schemas match the recorded calibration', () => { const judgeSource = fs.readFileSync(path.join(import.meta.dir, 'helpers', 'llm-judge.ts'), 'utf8'); const result = (id: string) => JSON.parse(fs.readFileSync(path.join(import.meta.dir, 'fixtures', 'judge-calibration', id, 'runs', 'comparison-1.result.json'), 'utf8')); const wiring: Array<[string, string, string]> = [ - ['qa-workflow', 'buildQaWorkflowJudgePrompt(section)', 'JUDGE_SCORE_SCHEMA'], - ['qa-health-rubric', 'buildQaHealthRubricJudgePrompt(section)', 'JUDGE_SCORE_SCHEMA'], ['qa-anti-refusal', 'buildQaAntiRefusalJudgePrompt(diffAwareSection, rulesSection)', 'QA_ANTI_REFUSAL_JUDGE_SCHEMA'], ['cross-skill', 'buildCrossSkillConsistencyJudgePrompt(collected)', 'CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA'], ['voice', 'buildVoiceDirectiveJudgePrompt(voiceSection)', 'VOICE_DIRECTIVE_JUDGE_SCHEMA'], @@ -296,49 +268,18 @@ describe('landed schemas match the recorded calibration', () => { expect(sent).toBe(recorded.landing.lands); }); - // /ship Step 7 coverage audit (GSTA-91). - // Value: protects=the held cookie-workflow result is what analyze() derives from its committed labels, samples, stop record and request options; - // fails_when=result.json is hand-edited, a corpus label moves after the run, a sample is altered, or the config's model/options drift; - // why_new=no test reads samples.jsonl or stop.json, and the wiring test trusts result.json's landing verbatim; seam=none; - // no_claim=that the recorded samples are genuine provider output (the committed JSONL stands in for the paid run), or that prompt - // text and bundle bytes are unchanged since the run (request_hash covers model and options only) - test('the recorded cookie-workflow result is reproducible from its committed labels, samples, stop record and request options', async () => { - const { CALIBRATION_CONFIGS } = await import('../scripts/judge-calibration-configs'); - const cookie = CALIBRATION_CONFIGS.find(config => config.id === 'cookie-workflow')!; - const runs = path.join(import.meta.dir, 'fixtures', 'judge-calibration', 'cookie-workflow', 'runs'); - const samples: SampleRecord[] = fs.readFileSync(path.join(runs, 'comparison-1.samples.jsonl'), 'utf8').trim().split('\n').map(line => JSON.parse(line)); - const { stopped } = JSON.parse(fs.readFileSync(path.join(runs, 'comparison-1.stop.json'), 'utf8')); - expect(JSON.parse(JSON.stringify(analyze(cookie, loadCorpus('cookie-workflow'), samples, stopped)))).toEqual(result('cookie-workflow')); - }); - test('armJudge sends ARM_JUDGE_SCHEMA because its calibration landed', () => { expect(result('arm').landing.lands).toBe(true); expect(judgeSource).toContain('buildArmJudgePrompt(task, diff), ARM_JUDGE_MODEL, { jsonSchema: ARM_JUDGE_SCHEMA })'); }); +}); - test('workflow transport follows the workflow-default and cookie-workflow calibrations; uncalibrated configurations stay on today\'s request', () => { - const recorded = result('workflow-default'); - expect(recorded.complete).toBe(true); - const cookie = result('cookie-workflow'); - expect(cookie.complete).toBe(true); - expect(cookie.landing.lands).toBe(false); - const release = JSON.parse(fs.readFileSync(path.join(import.meta.dir, 'fixtures', 'judge-calibration', 'cookie-workflow', 'release.json'), 'utf8')); - expect(release).toMatchObject({ config: 'cookie-workflow', released: true, by: 'GStack Minder', scope: 'cookie workflow judge transport only', - ruling: 'GStack Minder, 2026-10-29: release the GSTA-91 hold; a07 is matched by old-b; free-text parse errors cost train 42' }); - const registrations = [...evalSource.matchAll(/await runWorkflowJudge\(\{([\s\S]*?)\n \}\);/g)].map(match => match[1]!); - expect(registrations.length).toBeGreaterThan(14); - for (const body of registrations) { - const testName = /testName: '([^']+)'/.exec(body)![1]; - const defaultConfig = !/^\s+(?:agentCapability|model|readInput|thresholds|maxTokens|stream|effort):|\.\.\.COOKIE_WORKFLOW_JUDGE/m.test(body); - const expected = testName === 'ship/SKILL.md workflow' || (defaultConfig && recorded.landing.lands) - || testName === 'setup-browser-cookies/SKILL.md workflow'; - expect(body.includes('schemaTransport: true'), testName).toBe(expected); - expect(body.includes('compactReasoning: true'), testName).toBe(testName === 'ship/SKILL.md workflow'); +describe('retired skill-prose judges (GST-1371)', () => { + // Value: protects=the removal of the skill-prose clarity judges; fails_when=a merge or revert restores their control-bundle script or the cookie manual-review record that scripts/lib/paid-report.ts now rejects; why_new=no remaining test names either path; seam=none; no_claim=that the kept judges pass + test('the control-bundle script and the cookie manual-review record stay deleted', () => { + const root = path.resolve(import.meta.dir, '..'); + for (const file of ['scripts/judge-controls.ts', '.github/cookie-workflow-manual-review.json']) { + expect(fs.existsSync(path.join(root, file)), `${file} was retired with the skill-prose judges (GST-1371); fix: delete it`).toBe(false); } - expect(registrations.filter(body => body.includes('schemaTransport: true'))).toHaveLength(16); - const cookieBody = registrations.find(body => body.includes("testName: 'setup-browser-cookies/SKILL.md workflow'"))!; - expect(cookieBody).toContain('...COOKIE_WORKFLOW_JUDGE'); - expect(cookieBody).not.toMatch(/^\s+(?:agentCapability|model|thresholds|maxTokens|stream|effort):/m); - expect(Object.keys(COOKIE_WORKFLOW_JUDGE).sort()).toEqual(['judgeContext', 'judgeGoal', 'model', 'thresholds']); }); }); diff --git a/test/judge-panel-median.test.ts b/test/judge-panel-median.test.ts index 0c5d4d39d..b6204a861 100644 --- a/test/judge-panel-median.test.ts +++ b/test/judge-panel-median.test.ts @@ -1,7 +1,6 @@ import { describe, expect, test } from 'bun:test'; import replay from './fixtures/judge-panel-replay-2026-10-05.json'; import { judgePanelMajority, judgePanelMean, judgePanelMedian, JUDGE_PANEL_SAMPLES } from './helpers/llm-judge'; -import { BROWSE_JUDGE_FLOORS } from './helpers/workflow-judge-cache'; import { EVAL_POLICY } from './helpers/periodic-exclude-data'; const gate = (scores: number[], threshold = 4) => judgePanelMedian(scores.map(score => ({ score })), ['score']).score >= threshold; @@ -40,7 +39,7 @@ function verdict(name: string, samples: Sample[], aggregate: typeof judgePanelMe return keys.every(key => scores[key]! >= floors[key]!); }; if (name === 'Setup block') return meets(['clarity', 'actionability'], { clarity: 3, actionability: 3 }); - if (name === 'Browse SKILL.md') return meets(SCORE, BROWSE_JUDGE_FLOORS); + if (name === 'Browse SKILL.md') return meets(SCORE, { clarity: 3, completeness: 4, actionability: 4 }); if (name === 'QA anti-refusal') return judgePanelMajority(samples as Record[], 'would_browse') && meets(['confidence'], { confidence: 4 }); if (name === 'Cross-skill consistency') return judgePanelMajority(samples as Record[], 'consistent') && meets(['score'], { score: 4 }); if (name === 'Voice directive') { diff --git a/test/llm-judge-stream.test.ts b/test/llm-judge-stream.test.ts index 204368c01..13dcd9190 100644 --- a/test/llm-judge-stream.test.ts +++ b/test/llm-judge-stream.test.ts @@ -1,8 +1,14 @@ import { afterEach, beforeEach, expect, spyOn, test } from 'bun:test'; import { callJudge } from './helpers/llm-judge'; -import { WORKFLOW_JUDGE_RESPONSE_SCHEMA } from './helpers/workflow-judge-input'; const scores = { clarity: 4, completeness: 4, actionability: 4, reasoning: 'Complete workflow with explicit gates.' }; +const score = { type: 'integer', enum: [1, 2, 3, 4, 5] }; +const SCORE_SCHEMA = { + type: 'object', + properties: { clarity: score, completeness: score, actionability: score, reasoning: { type: 'string' } }, + required: ['clarity', 'completeness', 'actionability', 'reasoning'], + additionalProperties: false, +}; let originalKey: string | undefined; let transport: ReturnType; let diagnostics: ReturnType; @@ -48,12 +54,12 @@ test('the pinned SDK refuses a default nonstreaming 64k request before network a test('effort reaches the wire beside the response schema and is absent unless requested', async () => { transport.mockImplementation(async () => response()); await callJudge('score the whole bundle', 'claude-fable-5-1', { - max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, effort: 'medium', + max_tokens: 65_536, stream: true, jsonSchema: SCORE_SCHEMA, effort: 'medium', }); await callJudge('score the whole bundle', 'claude-fable-5-1', { max_tokens: 65_536, stream: true, effort: 'medium' }); await callJudge('score the whole bundle', 'claude-fable-5-1', { max_tokens: 65_536, stream: true }); const bodies = transport.mock.calls.map((call: any[]) => JSON.parse(call[1].body)); - expect(bodies[0].output_config).toEqual({ format: { type: 'json_schema', schema: WORKFLOW_JUDGE_RESPONSE_SCHEMA }, effort: 'medium' }); + expect(bodies[0].output_config).toEqual({ format: { type: 'json_schema', schema: SCORE_SCHEMA }, effort: 'medium' }); expect(bodies[1].output_config).toEqual({ effort: 'medium' }); expect(bodies[2]).not.toHaveProperty('output_config'); for (const body of bodies) expect(body).not.toHaveProperty('thinking'); @@ -62,12 +68,12 @@ test('effort reaches the wire beside the response schema and is absent unless re test('the real SDK streams 64k requests and parses only completed public text', async () => { transport.mockResolvedValue(response()); expect(await callJudge('score the whole bundle', 'claude-fable-5-1', { - max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, + max_tokens: 65_536, stream: true, jsonSchema: SCORE_SCHEMA, })).toEqual(scores); expect(transport).toHaveBeenCalledTimes(1); const request = transport.mock.calls[0][1]; expect(JSON.parse(request.body)).toEqual({ model: 'claude-fable-5-1', max_tokens: 65_536, - output_config: { format: { type: 'json_schema', schema: WORKFLOW_JUDGE_RESPONSE_SCHEMA } }, + output_config: { format: { type: 'json_schema', schema: SCORE_SCHEMA } }, messages: [{ role: 'user', content: 'score the whole bundle' }], stream: true }); expect(new Headers(request.headers).has('anthropic-beta')).toBe(false); expect(diagnostics).not.toHaveBeenCalled(); @@ -77,7 +83,7 @@ test('streamed truncation and refusal retain public evidence but never partial s for (const [stopReason, text] of [['max_tokens', '{"clarity":4'], ['max_tokens', null], ['refusal', null]] as const) { transport.mockResolvedValue(response(stopReason, text)); await expect(callJudge('score the whole bundle', 'claude-fable-5-1', { - max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, + max_tokens: 65_536, stream: true, jsonSchema: SCORE_SCHEMA, })).rejects.toThrow(stopReason === 'max_tokens' ? 'truncated at max_tokens=65536' : 'provider refused'); const diagnostic = diagnostics.mock.calls.at(-1)![0]; expect(diagnostic).not.toContain('PRIVATE_'); @@ -94,7 +100,7 @@ test('streaming retains the caller abort signal instead of extending its deadlin })); const controller = new AbortController(); const pending = callJudge('score the whole bundle', 'claude-fable-5-1', { - max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, signal: controller.signal, + max_tokens: 65_536, stream: true, jsonSchema: SCORE_SCHEMA, signal: controller.signal, }); const failure = new Error('Original workflow walltime elapsed'); const rejected = pending.catch(error => error); @@ -129,7 +135,7 @@ test('a judge aborted mid-stream retains what it had received, as counts without }); const controller = new AbortController(); const rejected = callJudge('score the whole bundle', 'claude-fable-5-1', { - max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, signal: controller.signal, + max_tokens: 65_536, stream: true, jsonSchema: SCORE_SCHEMA, signal: controller.signal, }).catch(error => error); await received; const failure = new Error('ship/SKILL.md workflow exceeded its workflow judge work deadline'); @@ -209,7 +215,7 @@ test('an event the diagnostic cannot read never changes the judge result', async const body = (await unreadable.text()).replace('"thinking":"PRIVATE_THINKING"', '"thinking":null'); transport.mockResolvedValue(new Response(body, { headers: { 'content-type': 'text/event-stream' } })); expect(await callJudge('score the whole bundle', 'claude-fable-5-1', { - max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, + max_tokens: 65_536, stream: true, jsonSchema: SCORE_SCHEMA, })).toEqual(scores); }); @@ -249,7 +255,7 @@ test('pings after the last SDK event show a live connection; a silent socket sho }, { highWaterMark: 0 }), { headers: { 'content-type': 'text/event-stream', 'request-id': 'req_quiet' } })); const controller = new AbortController(); const rejected = callJudge('score the whole bundle', 'claude-fable-5-1', { - max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, signal: controller.signal, + max_tokens: 65_536, stream: true, jsonSchema: SCORE_SCHEMA, signal: controller.signal, }).catch(error => error); await received; const failure = new Error('review/SKILL.md workflow exceeded its workflow judge work deadline'); diff --git a/test/paid-free-boundary.test.ts b/test/paid-free-boundary.test.ts index b69cb18cb..3cc41acd9 100644 --- a/test/paid-free-boundary.test.ts +++ b/test/paid-free-boundary.test.ts @@ -11,7 +11,7 @@ import { normalizeRelativePath as freeNormalize } from '../scripts/test-free-sha import { PAID_TEST_GLOBS } from './helpers/paid-test-set'; import { E2E_TIERS, LLM_JUDGE_TOUCHFILES } from './helpers/touchfiles-data'; import { selectTests } from './helpers/test-selection'; -import { workflowJudgeDependencies } from './helpers/workflow-judge-cache'; +import { sourceDependencyClosure } from '../scripts/eval-input-cache'; const ROOT = path.resolve(import.meta.dir, '..'); @@ -62,7 +62,10 @@ describe('paid/free dependency boundary', () => { const runner = runnerDependencies(ROOT, ['scripts/test-paid-shards.ts', 'scripts/eval-select.ts']); const paid = [...new Set(PAID_TEST_GLOBS.flatMap(pattern => [...new Bun.Glob(pattern).scanSync(ROOT)]))]; expect(paid.length).toBeGreaterThan(0); - const all = workflowJudgeDependencies(ROOT, paid); + const all = sourceDependencyClosure(ROOT, ['test/helpers/llm-judge.ts', 'lib/eval-model.ts', 'test/helpers/eval-budgets.ts', + 'scripts/test-paid-shards.ts', 'scripts/test-strict-output.ts', 'scripts/eval-select.ts', + 'scripts/test-pr-profile.ts', '.github/workflows/evals.yml', + 'package.json', 'bun.lock', '.github/docker/Dockerfile.ci', ...paid]); for (const dependencies of [runner, all]) { expect(dependencies).toContain('scripts/lib/shard-engine.ts'); expect(dependencies).toContain('test/helpers/test-selection.ts'); @@ -115,13 +118,13 @@ describe('paid/free dependency boundary', () => { expect(result.coverage?.unknownFiles, file).toEqual([]); expect(result.selection, file).toEqual({ e2e: [], judges: [] }); } - // ETHOS.md is read by the /office-hours and /plan-ceo-review prompts, so it selects their cases and judge, not the whole gate. + // ETHOS.md is read by the /office-hours and /plan-ceo-review prompts, so it selects their cases, not the whole gate. const ethos = computePaidCaseSelection({ profile: 'pr', env: {}, changedFiles: ['ETHOS.md', 'NOTICE.md'] }); expect(ethos.coverage?.mode).toBe('pr'); expect(ethos.coverage?.unknownFiles).toEqual([]); expect(ethos.selection.e2e).toContain('plan-ceo-review-plan-mode'); expect(ethos.selection.e2e).toContain('office-hours-auto-mode'); - expect(ethos.selection.judges).toEqual(['plan-ceo-review/SKILL.md modes']); + expect(ethos.selection.judges).toEqual([]); expect(ethos.selection.e2e!.length).toBeLessThan(10); }); @@ -220,10 +223,10 @@ describe('paid/free dependency boundary', () => { expect(result.judges).toEqual(category === 'e2e' ? [] : ['quality']); } const real = computePaidCaseSelection({ profile: 'pr', env: {}, - changedFiles: [file, 'plan-ceo-review/SKILL.md.tmpl'] }); + changedFiles: [file, 'plan-ceo-review/SKILL.md.tmpl', 'retro/SKILL.md'] }); expect(real.coverage?.mode).toBe('pr'); expect(real.selection.e2e).toContain('auq-format-gate'); - expect(real.selection.judges).toContain('plan-ceo-review/SKILL.md modes'); + expect(real.selection.judges).toContain('cross-skill greptile consistency'); expect(real.selection.e2e?.every(id => PR_PROFILE_CASE_IDS.includes(id as typeof PR_PROFILE_CASE_IDS[number]))).toBe(true); }); }); diff --git a/test/paid-pr-profile.test.ts b/test/paid-pr-profile.test.ts index 35b24787f..018fc79b3 100644 --- a/test/paid-pr-profile.test.ts +++ b/test/paid-pr-profile.test.ts @@ -22,7 +22,7 @@ const CEO_FILES = [ 'test/skill-e2e-retro.test.ts', 'test/skill-llm-eval.test.ts', 'test/skill-e2e-plan-ceo-plan-mode.test.ts', ]; const ceoManifest = () => buildRunManifest({ tier: 'gate', profile: 'pr', sliceCount: 1, - evalsAll: false, env: {}, changedFiles: ['plan-ceo-review/SKILL.md.tmpl'], discovered: CEO_FILES }); + evalsAll: false, env: {}, changedFiles: ['plan-ceo-review/SKILL.md.tmpl', 'retro/SKILL.md'], discovered: CEO_FILES }); describe('PR profile paid-runner integration', () => { test('F5 fixture changes select exactly its three registered native gate cases', () => { @@ -83,7 +83,7 @@ describe('PR profile paid-runner integration', () => { expect(manifest.profile).toBe('pr'); expect(manifest.selection?.e2e).toContain('auq-format-gate'); expect(manifest.selection?.e2e).toContain('plan-ceo-review-plan-mode'); - expect(manifest.selection?.judges).toContain('plan-ceo-review/SKILL.md modes'); + expect(manifest.selection?.judges).toContain('cross-skill greptile consistency'); expect(manifest.entries.find(entry => entry.file.includes('skill-e2e-retro'))?.status).toBe('skipped-by-diff'); expect(manifest.entries.find(entry => entry.file.includes('ask-user-question'))?.status).toBe('planned'); expect(manifest.prCoverage?.deferred.some(item => item.id === 'plan-ceo-review-plan-mode')).toBe(false); @@ -127,13 +127,13 @@ describe('PR profile paid-runner integration', () => { expect(result.coverage?.needsFullValidation).toBe(false); }); - test('F8 selects one judge and defers its two periodic readiness actors', () => { + test('F8 selects no judge and defers its two periodic readiness actors', () => { const selection = computePaidCaseSelection({ profile: 'pr', env: {}, changedFiles: [ 'sync-gbrain/SKILL.md.tmpl', 'bin/gstack-gbrain-read-capability.ts', 'test/helpers/sync-gbrain-readiness-fixture.ts', ] }); expect(selection.coverage?.mode).toBe('pr'); - expect(selection.selection.judges).toEqual(['sync-gbrain/SKILL.md read-only readiness']); + expect(selection.selection.judges).toEqual([]); expect(selection.selection.e2e).toEqual([]); expect(selection.coverage?.deferred.map(({ id }) => id).filter(id => id.startsWith('sync-gbrain-read-')).sort()).toEqual([ 'sync-gbrain-read-ready', 'sync-gbrain-read-unknown', diff --git a/test/paid-report-census.test.ts b/test/paid-report-census.test.ts index 103e1c8d2..d091a9550 100644 --- a/test/paid-report-census.test.ts +++ b/test/paid-report-census.test.ts @@ -201,17 +201,18 @@ describe('record builders attach ledger sessions and failure causes (plan 0.1/0. }); describe('red lines carry their values, evidence and a one-case command (plan 0.3)', () => { - const QA_ONLY = 'qa-only/SKILL.md workflow'; + const QA_ONLY = 'voice directive tone'; const addJudgeRed = (dir: string) => { const shard = path.join(dir, 'paid-slice-1-a1', 'shards', 'skill-llm-eval'); fs.mkdirSync(shard, { recursive: true }); const message = 'expect(received).toBeGreaterThanOrEqual(expected)\n\nExpected: >= 4\nReceived: 3.6666666666666665\n'; fs.writeFileSync(path.join(shard, 'junit.xml'), `\n` - + `` + + `` + `\n`); - // Trimmed from census 37198445662's paid-slice-5-a1 qa-only record. + // Trimmed from census 37198445662's paid-slice-5-a1 qa-only record, relabeled as a kept judge case + // after the workflow clarity judges were removed. fs.writeFileSync(path.join(shard, '1.91.19-x-llm-judge-2026-10-04-1129.json'), JSON.stringify({ tier: 'llm-judge', total_cost_usd: 0.06, tests: [{ - name: QA_ONLY, suite: 'Other skill evals', tier: 'llm-judge', passed: false, duration_ms: 34829, cost_usd: 0.06, execution: 'executed', + name: QA_ONLY, suite: 'Voice directive eval', tier: 'llm-judge', passed: false, duration_ms: 34829, cost_usd: 0.06, execution: 'executed', judge_scores: { clarity: 3.6666666666666665, completeness: 4.333333333333333, actionability: 3.6666666666666665 }, judge_reasoning: '[sample 1] The bundle is thorough. Clarity suffers from extreme density and cross-file indirection. Two actionability concerns: Browser Quick allots 30 seconds.\n[sample 2] Deliverables are concrete.\n[sample 3] Execution is feasible.', exit_reason: 'validation_failed', error: message.replace('>', '>') }] })); @@ -232,9 +233,9 @@ describe('red lines carry their values, evidence and a one-case command (plan 0. expect(line).toContain('FAIL assertion — expect(received).toBeGreaterThanOrEqual(expected)'); expect(line).toContain('· cause assertion clarity 3.67 < 4 (3 samples) "Clarity suffers from extreme density and cross-file indirection."; ' + 'actionability 3.67 < 4 (3 samples) "Two actionability concerns: Browser Quick allots 30 seconds."'); - expect(line).toContain('evidence: paid-slice-1-a1/shards/skill-llm-eval (fetch: bun run eval:pass-rates --run 37198445662 --case \'qa-only/SKILL.md workflow\')'); + expect(line).toContain('evidence: paid-slice-1-a1/shards/skill-llm-eval (fetch: bun run eval:pass-rates --run 37198445662 --case \'voice directive tone\')'); expect(line).toContain(`after a repair: EVALS=1 EVALS_TIER=gate EVALS_JUDGE_SELECTION_JSON='{"version":1,"selected":["${QA_ONLY}"],"reason":"after a repair"}' ` - + `bun test test/skill-llm-eval.test.ts -t '(?:^|\\s)(?:qa-only/SKILL\\.md workflow)$'`); + + `bun test test/skill-llm-eval.test.ts -t '(?:^|\\s)(?:voice directive tone)$'`); expect(line).not.toContain('rerun:'); const record = r.history.find(x => x.case === QA_ONLY); expect(record).toMatchObject({ failure_class: 'assertion', error: 'expect(received).toBeGreaterThanOrEqual(expected)', failure_cause: 'assertion', diff --git a/test/paid-retry-supervision.test.ts b/test/paid-retry-supervision.test.ts index b8a43dbb2..79391b15f 100644 --- a/test/paid-retry-supervision.test.ts +++ b/test/paid-retry-supervision.test.ts @@ -21,7 +21,7 @@ const expectedWalls = { 'test/skill-e2e-qa-callers.test.ts': 1_695_000, 'test/skill-e2e-shared-libs-paths.test.ts': 1_920_000, 'test/skill-e2e-ship-docsync.test.ts': 4_920_000, - 'test/skill-llm-eval.test.ts': 3_170_000, + 'test/skill-llm-eval.test.ts': 480_000, 'test/skill-e2e-auq-consistency.test.ts': 1_080_000, 'test/skill-e2e-auq-matrix.test.ts': 1_920_000, 'test/skill-e2e-plan-format.test.ts': 1_360_000, @@ -174,15 +174,12 @@ test('quality judge supervision includes the added judge without changing ordina expect(ALL_TIERS).toEqual({ JUDGE_MS: 120000, CAPTURE_MS: 300000, CAPTURE_LONG_MS: 600000, PTY_MS: 900000, PTY_LONG_MS: 1200000 }); const quality = 'test/skill-llm-eval.test.ts'; const qualityBudget = FILE_RETRY_BUDGETS.find(row => row.file === quality)!; - expect(resolvePaidShardBudget([quality])).toEqual({ timeoutMs: 3_170_000, source: 'registered', policyId: qualityBudget.id }); + expect(resolvePaidShardBudget([quality])).toEqual({ timeoutMs: 480_000, source: 'registered', policyId: qualityBudget.id }); expect(retriesForFiles([quality])).toBe(0); const qualitySource = read(quality); - const judgeTimeouts = [...qualitySource.matchAll(/}\s*,\s*(JUDGE_MS|WORKFLOW_JUDGE_TEST_MS)\s*\);/g)].map(match => match[1]); - expect(judgeTimeouts.filter(timeout => timeout === 'JUDGE_MS')).toHaveLength(7); - expect(judgeTimeouts.filter(timeout => timeout === 'WORKFLOW_JUDGE_TEST_MS')).toHaveLength(17); - expect(qualitySource).toContain('WORKFLOW_JUDGE_TEST_MS = JUDGE_MS + 10_000'); - expect(qualitySource).toContain('const workDeadline = started + JUDGE_MS'); - expect(qualityBudget.shardMs).toBe(7 * ALL_TIERS.JUDGE_MS + 17 * (ALL_TIERS.JUDGE_MS + 10_000) + 120_000); + const judgeTimeouts = [...qualitySource.matchAll(/}\s*,\s*(\w+)\s*\);/g)].map(match => match[1]); + expect(judgeTimeouts).toEqual(['JUDGE_MS', 'JUDGE_MS', 'JUDGE_MS']); + expect(qualityBudget.shardMs).toBe(3 * ALL_TIERS.JUDGE_MS + 120_000); expect(FINDING_RETRY_BUDGETS.map(row => [row.cases, row.testMs, row.shardMs])).toEqual([ ...Array(2).fill([1, 1500000, 1620000]), ]); @@ -294,7 +291,7 @@ test('the periodic executor supervises every actual case within its planned CI w evalsAll: true, env: { EVALS_ALL: '1' } }); const census = manifest.entries.filter(row => row.status === 'planned'); expect(new Set(census.map(row => shardFile(row.file)))).toEqual(new Set(selectPaidTestFiles(collectPaidTestFiles(), 'periodic').selected)); - expect(census.find(row => row.file === 'test/skill-llm-eval.test.ts')?.budget?.timeoutMs).toBe(3_170_000); + expect(census.find(row => row.file === 'test/skill-llm-eval.test.ts')?.budget?.timeoutMs).toBe(480_000); // W2c/ENG-2: each slice's job ceiling covers its longest single shard (and the serialized overlay // envelope) plus setup; the in-process slice deadline turns work that cannot start into not_run. const sliceFiles = Array.from({ length: manifest.sliceCount }, (_, i) => sliceExecutionOrder( diff --git a/test/ship-measure.test.ts b/test/ship-measure.test.ts index 4fecd9817..6dafb94c6 100644 --- a/test/ship-measure.test.ts +++ b/test/ship-measure.test.ts @@ -133,7 +133,7 @@ describe('per-kind batches decided by the measurement bar, with fake runners (EN }); test('standalone judge ids resolve to the one paid file that names them', () => { - expect(judgeFile('review/SKILL.md workflow')).toBe('test/skill-llm-eval.test.ts'); + expect(judgeFile('voice directive tone')).toBe('test/skill-llm-eval.test.ts'); expect(() => judgeFile('no-such-judge-id')).toThrow('no paid file names it'); }); }); diff --git a/test/skill-llm-eval.test.ts b/test/skill-llm-eval.test.ts index 403dccded..5e8f97417 100644 --- a/test/skill-llm-eval.test.ts +++ b/test/skill-llm-eval.test.ts @@ -1,8 +1,8 @@ /** - * LLM-as-a-Judge evals for generated SKILL.md quality. + * LLM-as-a-Judge evals for generated SKILL.md content. * - * Uses the Anthropic API directly (not Agent SDK) to evaluate whether - * generated command docs are clear, complete, and actionable for an AI agent. + * Uses the Anthropic API directly (not Agent SDK) for three targeted checks: + * QA anti-refusal, cross-skill greptile consistency and the voice directive. * * Requires: ANTHROPIC_API_KEY env var (or EVALS=1 with key already set) * Run: EVALS=1 bun run test:eval @@ -14,15 +14,7 @@ import { afterAll, expect } from 'bun:test'; import { JUDGE_MS } from './helpers/eval-budgets'; import * as fs from 'fs'; import * as path from 'path'; -import { callJudge, judge, JudgeRefusalError, JUDGE_SCORE_SCHEMA, QA_ANTI_REFUSAL_JUDGE_SCHEMA, VOICE_DIRECTIVE_JUDGE_SCHEMA, CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA, buildQaWorkflowJudgePrompt, buildQaHealthRubricJudgePrompt, buildQaAntiRefusalJudgePrompt, buildCrossSkillConsistencyJudgePrompt, buildVoiceDirectiveJudgePrompt, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMean, judgePanelMedian, judgePanelSummary, judgePanelMajority, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS } from './helpers/llm-judge'; -import { ASK_QUESTIONS_HEADING, ENG_REVIEW_EXCERPT } from './helpers/workflow-excerpt'; -import type { JudgeScore } from './helpers/llm-judge'; -import { readWorkflowJudgeInput, buildWorkflowJudgePrompt, QA_DISCOVERY_REFERENCES, WORKFLOW_JUDGE_RESPONSE_SCHEMA, type WorkflowJudgeInput } from './helpers/workflow-judge-input'; -import { BROWSE_JUDGE_FLOORS, browseJudgeFloorsMet, prepareWorkflowJudgeCache, validWorkflowJudgeScore } from './helpers/workflow-judge-cache'; -import { buildCookieWorkflowJudgeInput, COOKIE_WORKFLOW_JUDGE } from './helpers/cookie-workflow-judge-input'; -import { getCookieWorkflowManualReview, type ManualJudgeReview } from './helpers/cookie-workflow-manual-review'; -import { resolveEvalModel } from '../lib/eval-model'; -import type { EvalCacheValue } from '../scripts/eval-input-cache'; +import { callJudge, QA_ANTI_REFUSAL_JUDGE_SCHEMA, VOICE_DIRECTIVE_JUDGE_SCHEMA, CROSS_SKILL_CONSISTENCY_JUDGE_SCHEMA, buildQaAntiRefusalJudgePrompt, buildCrossSkillConsistencyJudgePrompt, buildVoiceDirectiveJudgePrompt, judgePanel, judgePanelMean, judgePanelMedian, judgePanelSummary, judgePanelMajority, judgePanelReasoning } from './helpers/llm-judge'; import { LLM_JUDGE_TOUCHFILES } from './helpers/touchfiles'; import { readSkillEffective } from './helpers/skill-startup'; // Runs when EVALS=1 is set (requires ANTHROPIC_API_KEY in env) — the EVALS @@ -43,37 +35,6 @@ import { // Eval result collector const evalCollector = createEvalCollector('llm-judge'); -/** - * Browse carve (token-reduction Phase 4): the '## Snapshot Flags' and - * '## Full Command List' reference blocks moved from browse/SKILL.md into the - * generated on-demand section browse/sections/command-list.md ('## Snapshot - * Flags' first, then '## Full Command List'). '## SETUP', '## Core QA - * Patterns', and '## CSS Inspector' stay in the skeleton. Non-empty guard: - * judging an empty slice would silently pass garbage to the judge. - */ -function readBrowseCommandSection(): string { - const p = path.join(ROOT, 'browse', 'sections', 'command-list.md'); - const content = fs.readFileSync(p, 'utf-8'); - if (!content.includes('## Snapshot Flags') || !content.includes('## Full Command List')) { - throw new Error( - `${p} is missing the expected headers — regenerate with: bun run gen:skill-docs`, - ); - } - return content; -} - -/** Slice a section out of the command-list section file, guarded non-empty. */ -function sliceBrowseSection(startHeader: string): string { - const content = readBrowseCommandSection(); - const start = content.indexOf(startHeader); - if (start < 0) throw new Error(`browse/sections/command-list.md: "${startHeader}" not found`); - const section = content.slice(start); - if (section.trim().length < 200) { - throw new Error(`browse/sections/command-list.md slice at "${startHeader}" is empty/stub — regenerate with: bun run gen:skill-docs`); - } - return section; -} - // --- Diff-based test selection (LLM_JUDGE_TOUCHFILES, not the E2E table) --- const selectedTests = resolveModuleSelection( process.env.EVALS ? process.env.EVALS_JUDGE_SELECTION_JSON : undefined, @@ -90,84 +51,6 @@ function testIfSelected(testName: string, fn: () => Promise, timeout: numb testConcurrentIfSelected(testName, fn, timeout, selectedTests); } -describeIfSelected('LLM-as-judge quality evals', [ - 'browse/SKILL.md reference', 'setup block', -], () => { - testIfSelected('browse/SKILL.md reference', async () => { - const t0 = Date.now(); - // Browse carve: snapshot flags + the full command list are the whole - // generated section file; one judge grades the union. Scores are also - // compared with test/fixtures/eval-baselines.json and the difference is - // recorded (UPDATE_BASELINES=1 rewrites it); only the floors gate. - const section = sliceBrowseSection('## Snapshot Flags'); - - const samples = await judgePanel(() => judge('browse skill reference (flags + commands)', section)); - const { median: scores, mean } = judgePanelSummary(samples, JUDGE_SCORE_DIMENSIONS); - console.log('Browse SKILL.md panel:', JSON.stringify({ gate: 'median', median: scores, mean, samples }, null, 2)); - - const baselinesPath = path.join(ROOT, 'test', 'fixtures', 'eval-baselines.json'); - const baselines = JSON.parse(fs.readFileSync(baselinesPath, 'utf-8')); - const regressions = (['clarity', 'completeness', 'actionability'] as const) - .filter(dim => scores[dim] < baselines.browse_skill[dim]) - .map(dim => `browse_skill.${dim}: ${scores[dim]} < baseline ${baselines.browse_skill[dim]}`); - if (process.env.UPDATE_BASELINES) { - baselines.browse_skill = { clarity: scores.clarity, completeness: scores.completeness, actionability: scores.actionability }; - fs.writeFileSync(baselinesPath, JSON.stringify(baselines, null, 2) + '\n'); - console.log('Updated eval baselines'); - } - - evalCollector?.addTest({ - name: 'browse/SKILL.md reference', - suite: 'LLM-as-judge quality evals', - tier: 'llm-judge', - passed: browseJudgeFloorsMet(scores), - duration_ms: Date.now() - t0, - cost_usd: 0.02 * samples.length, - judge_scores: { clarity: scores.clarity, completeness: scores.completeness, actionability: scores.actionability }, - judge_reasoning: regressions.length ? `${judgePanelReasoning(samples)} | ${regressions.join('; ')}` : judgePanelReasoning(samples), - }); - - expect(scores.clarity).toBeGreaterThanOrEqual(BROWSE_JUDGE_FLOORS.clarity); - expect(scores.completeness).toBeGreaterThanOrEqual(BROWSE_JUDGE_FLOORS.completeness); - expect(scores.actionability).toBeGreaterThanOrEqual(BROWSE_JUDGE_FLOORS.actionability); - }, JUDGE_MS); - - testIfSelected('setup block', async () => { - const t0 = Date.now(); - // P2 (v1.2.0): the browse setup block moved from the root router to browse/SKILL.md. - const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8'); - // The setup block is the Aside contract ('## BROWSER SETUP (Aside ...') with - // the browse binary as fallback; older renders headed it '## SETUP'. Slice - // from whichever heading is present to the next H2. - let setupStart = content.indexOf('## BROWSER SETUP'); - if (setupStart < 0) setupStart = content.indexOf('## SETUP'); - const setupEnd = content.indexOf('\n## ', setupStart + 3); - if (setupStart < 0 || setupEnd < 0) throw new Error('browse/SKILL.md: setup block not found — regenerate with: bun run gen:skill-docs'); - const section = content.slice(setupStart, setupEnd); - - const samples = await judgePanel(() => judge('setup/binary discovery instructions', section)); - const { median: scores, mean } = judgePanelSummary(samples, JUDGE_SCORE_DIMENSIONS); - console.log('Setup block panel:', JSON.stringify({ gate: 'median', median: scores, mean, samples }, null, 2)); - - evalCollector?.addTest({ - name: 'setup block', - suite: 'LLM-as-judge quality evals', - tier: 'llm-judge', - passed: scores.actionability >= 3 && scores.clarity >= 3, - duration_ms: Date.now() - t0, - cost_usd: 0.02 * samples.length, - judge_scores: { clarity: scores.clarity, completeness: scores.completeness, actionability: scores.actionability }, - judge_reasoning: judgePanelReasoning(samples), - }); - - // Setup block is intentionally minimal (binary discovery only). - // SKILL_DIR is inferred from context, so judge sometimes scores 3. - expect(scores.actionability).toBeGreaterThanOrEqual(3); - expect(scores.clarity).toBeGreaterThanOrEqual(3); - }, JUDGE_MS); - -}); - // --- Part 7: QA skill quality evals (C6) --- /** @@ -198,60 +81,7 @@ function sliceQaPatterns(startHeader: string, endHeader?: string): string { return section; } -describeIfSelected('QA skill quality evals', ['qa/SKILL.md workflow', 'qa/SKILL.md health rubric', 'qa/SKILL.md anti-refusal'], () => { - testIfSelected('qa/SKILL.md workflow', async () => { - const t0 = Date.now(); - const section = readWorkflowJudgeInput({ root: ROOT, skillPath: 'qa/SKILL.md', - startMarker: '# /qa: Test', endMarker: null, - // qa-patterns.md loads both browser assets; judges penalized their absence. - references: ['qa/templates/functional-report-template.md', 'qa/templates/qa-report-template.md', 'qa/references/issue-taxonomy.md'] }).text; - - const samples = await judgePanel(() => callJudge(buildQaWorkflowJudgePrompt(section))); - const { median: scores, mean } = judgePanelSummary(samples, JUDGE_SCORE_DIMENSIONS); - console.log('QA workflow panel:', JSON.stringify({ gate: 'median', median: scores, mean, samples }, null, 2)); - - evalCollector?.addTest({ - name: 'qa/SKILL.md workflow', - suite: 'QA skill quality evals', - tier: 'llm-judge', - passed: scores.clarity >= 3 && scores.completeness >= 3 && scores.actionability >= 4, - duration_ms: Date.now() - t0, - cost_usd: 0.02 * samples.length, - judge_scores: { clarity: scores.clarity, completeness: scores.completeness, actionability: scores.actionability }, - judge_reasoning: judgePanelReasoning(samples), - }); - - expect(scores.clarity).toBeGreaterThanOrEqual(3); - // Completeness scores 3 when judge notes the health rubric is in a separate - // section (the eval only passes the Workflow section, not the full document). - expect(scores.completeness).toBeGreaterThanOrEqual(3); - expect(scores.actionability).toBeGreaterThanOrEqual(4); - }, JUDGE_MS); - - testIfSelected('qa/SKILL.md health rubric', async () => { - const t0 = Date.now(); - const section = sliceQaPatterns('## Health Score Rubric'); - - const samples = await judgePanel(() => callJudge(buildQaHealthRubricJudgePrompt(section), undefined, { jsonSchema: JUDGE_SCORE_SCHEMA })); - const { median: scores, mean } = judgePanelSummary(samples, JUDGE_SCORE_DIMENSIONS); - console.log('QA health rubric panel:', JSON.stringify({ gate: 'median', median: scores, mean, samples }, null, 2)); - - evalCollector?.addTest({ - name: 'qa/SKILL.md health rubric', - suite: 'QA skill quality evals', - tier: 'llm-judge', - passed: scores.clarity >= 3 && scores.completeness >= 3 && scores.actionability >= 4, - duration_ms: Date.now() - t0, - cost_usd: 0.02 * samples.length, - judge_scores: { clarity: scores.clarity, completeness: scores.completeness, actionability: scores.actionability }, - judge_reasoning: judgePanelReasoning(samples), - }); - - expect(scores.clarity).toBeGreaterThanOrEqual(3); - expect(scores.completeness).toBeGreaterThanOrEqual(3); - expect(scores.actionability).toBeGreaterThanOrEqual(4); - }, JUDGE_MS); - +describeIfSelected('QA skill quality evals', ['qa/SKILL.md anti-refusal'], () => { testIfSelected('qa/SKILL.md anti-refusal', async () => { const t0 = Date.now(); // Extract both the diff-aware mode section and Important Rules section. @@ -329,409 +159,6 @@ describeIfSelected('Cross-skill consistency evals', ['cross-skill greptile consi }, JUDGE_MS); }); -// --- Workflow SKILL.md quality evals (10 new tests for 100% coverage) --- - -/** - * DRY helper for workflow SKILL.md judge tests. - * Extracts a section from a SKILL.md file and judges its quality as an agent workflow. - */ -// Keep model work at JUDGE_MS. Only terminal recording/cache I/O gets grace. -const WORKFLOW_JUDGE_RECORD_MS = 5_000; -const WORKFLOW_JUDGE_TEST_MS = JUDGE_MS + 10_000; -const workflowJudgeAttempts = new Map(); - -async function runWorkflowJudge(opts: { - testName: string; - suite: string; - skillPath: string; - startMarker: string; - endMarker: string | RegExp | null; - references?: readonly string[]; - judgeContext: string; - judgeGoal: string; - agentCapability?: 'frontier'; - schemaTransport?: boolean; - compactReasoning?: boolean; - maxTokens?: number; - stream?: boolean; - effort?: 'medium'; - model?: string; - thresholds?: { clarity: number; completeness: number; actionability: number }; - readInput?: () => WorkflowJudgeInput; -}) { - const started = performance.now(); - const previous = workflowJudgeAttempts.get(opts.testName); - previous?.cancel(); - const attempt = (previous?.attempt ?? 0) + 1; - const controller = new AbortController(); - const workDeadline = started + JUDGE_MS; - let stage: 'input' | 'judge' | 'validation' | 'recording' = 'input'; - let finalized = false; - let samples: JudgeScore[] | undefined; - let scores: Record | undefined; - let manualReview: ManualJudgeReview | undefined; - let customInputMetadata: { prompt: string; model: string } | undefined; - let reused: ReturnType['lookup']> = null; - let timer: ReturnType; - let rejectStopped: (error: Error) => void; - const stopped = new Promise((_, reject) => { rejectStopped = reject; }); - const deadlineError = () => Object.assign(new Error(`${opts.testName} exceeded its workflow judge ${stage === 'recording' ? 'recording' : 'work'} deadline`), { name: 'WorkflowJudgeDeadline' }); - const deadline = () => workDeadline + (stage === 'recording' ? WORKFLOW_JUDGE_RECORD_MS : 0); - const active = () => !finalized && workflowJudgeAttempts.get(opts.testName) === state - && performance.now() < deadline(); - const finish = (passed: boolean, error?: unknown) => { - if (finalized) return; - finalized = true; - clearTimeout(timer); - if (!passed) controller.abort(error); - evalCollector?.addTest({ - name: opts.testName, suite: opts.suite, tier: 'llm-judge', passed, attempt, - duration_ms: Math.max(0, performance.now() - started), - cost_usd: reused || !samples ? 0 : 0.02 * samples.length, - execution: reused ? 'reused' : 'executed', - ...customInputMetadata, - ...(manualReview ? { manual_review: manualReview } : {}), - ...(reused ? { reused_from: { input_key: reused.reuse.key, run_id: reused.reuse.source.runId, - revision: reused.reuse.source.revision, completed_at: new Date(reused.reuse.source.completedAt).toISOString() } } : {}), - ...(scores ? { judge_scores: { clarity: scores.clarity, completeness: scores.completeness, actionability: scores.actionability } } : {}), - ...(samples ? { judge_reasoning: judgePanelReasoning(samples) } : {}), - ...(passed ? {} : { exit_reason: error instanceof JudgeRefusalError ? 'provider_refusal' - : error instanceof Error && error.name === 'WorkflowJudgeDeadline' ? 'timeout' - : error instanceof Error && error.name === 'WorkflowJudgeSuperseded' ? 'cancelled' - : stage === 'validation' ? 'validation_failed' : 'harness_error', - error: `${error instanceof Error ? error.message : String(error)}${samples ? '' : error instanceof JudgeRefusalError - ? '\nNo automated score; provider refusal usage retained when manually accepted; cost unavailable.' - : '\nNo completed model response; cost and usage unavailable.'}` }), - }); - }; - const stop = (error: Error) => { - if (finalized) return; - try { finish(false, error); } finally { rejectStopped(error); } - }; - const state = { attempt, cancel: () => stop(performance.now() >= deadline() ? deadlineError() - : Object.assign(new Error(`${opts.testName} attempt superseded by retry`), { name: 'WorkflowJudgeSuperseded' })) }; - // Register before any input read: a failed fixture read is still attempt one. - workflowJudgeAttempts.set(opts.testName, state); - const checkActive = () => { - if (!active()) { - if (!finalized) stop(deadlineError()); - throw controller.signal.reason ?? deadlineError(); - } - }; - const arm = () => { clearTimeout(timer); timer = setTimeout(() => stop(deadlineError()), Math.max(0, deadline() - performance.now())); }; - arm(); - - // registered attempt -> bounded request -> unchanged assertions -> terminal record - // Timeout/retry finalizes once; late provider continuations cannot publish evidence. - const work = async () => { - checkActive(); - const thresholds = { clarity: 3, completeness: 3, actionability: 4, ...opts.thresholds }; - const input = opts.readInput ? opts.readInput() : readWorkflowJudgeInput({ root: ROOT, skillPath: opts.skillPath, - startMarker: opts.startMarker, endMarker: opts.endMarker, references: opts.references }); - checkActive(); - const prompt = buildWorkflowJudgePrompt(opts, input); - if (opts.readInput) customInputMetadata = { prompt, model: resolveEvalModel('judge', opts.model) }; - const cache = prepareWorkflowJudgeCache({ ...opts, root: ROOT, thresholds, prompt, attempt }); - checkActive(); - reused = cache.lookup(); - checkActive(); - stage = 'judge'; - const maxTokens = opts.maxTokens ?? DEFAULT_JUDGE_MAX_TOKENS; - let result: JudgeScore[]; - try { - result = reused?.samples ?? await judgePanel(() => callJudge(prompt, opts.model, { signal: controller.signal, max_tokens: maxTokens, - ...(opts.stream ? { stream: true } : {}), - ...(opts.schemaTransport ? { jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA } : {}), - ...(opts.effort ? { effort: opts.effort } : {}) })); - } catch (error) { - checkActive(); - if (error instanceof JudgeRefusalError && customInputMetadata) { - const approved = getCookieWorkflowManualReview(ROOT, { testName: opts.testName, prompt, - model: customInputMetadata.model, maxTokens, thresholds, attempt }, error.refusal); - checkActive(); - if (approved) { - manualReview = approved; - console.log(`[workflow-judge] ${opts.testName}: MANUAL ACCEPTANCE, no automated score; ${approved.approval.approval_url}`); - finish(false, error); - return; - } - } - throw error; - } - checkActive(); - samples = result; - console.log(`[workflow-judge] ${opts.testName}: ${reused ? `reused ${reused.reuse.source.runId} @ ${reused.reuse.source.revision} (${new Date(reused.reuse.source.completedAt).toISOString()})` : 'executed'}`); - stage = 'validation'; - if (opts.compactReasoning && !samples.every(sample => validWorkflowJudgeScore(sample as unknown as EvalCacheValue, { clarity: 1, completeness: 1, actionability: 1 }, true))) { - throw new Error('Workflow judge violated the compact response contract'); - } - scores = judgePanelMedian(samples, JUDGE_SCORE_DIMENSIONS); - console.log(`${opts.testName} panel:`, JSON.stringify({ gate: 'median', median: scores, mean: judgePanelMean(samples, JUDGE_SCORE_DIMENSIONS), samples }, null, 2)); - expect(scores.clarity).toBeGreaterThanOrEqual(thresholds.clarity); - expect(scores.completeness).toBeGreaterThanOrEqual(thresholds.completeness); - expect(scores.actionability).toBeGreaterThanOrEqual(thresholds.actionability); - checkActive(); - stage = 'recording'; - arm(); - const discardReceipt = reused ? undefined : cache.publish(samples, active); - try { checkActive(); finish(true); } - catch (error) { discardReceipt?.(); throw error; } - }; - try { await Promise.race([work(), stopped]); } - catch (error) { - const failure = finalized ? controller.signal.reason ?? error - : performance.now() >= deadline() ? deadlineError() : error; - if (!finalized) finish(false, failure); - throw failure; - } -} - -// Block 1: Ship & Release skills -describeIfSelected('Ship & Release skill evals', ['ship/SKILL.md workflow', 'document-release/SKILL.md workflow'], () => { - testIfSelected('ship/SKILL.md workflow', async () => { - await runWorkflowJudge({ - testName: 'ship/SKILL.md workflow', - schemaTransport: true, - compactReasoning: true, - maxTokens: 65_536, - stream: true, - // Default effort thought past JUDGE_MS in 3 of 18 measured samples; medium kept all 18 under 80 s. - effort: 'medium', - suite: 'Ship & Release skill evals', - agentCapability: 'frontier', - // The contract now precedes platform detection; keep the complete workflow. - skillPath: 'ship/SKILL.md', - startMarker: '# Ship:', - endMarker: '## Important Rules', - references: QA_DISCOVERY_REFERENCES, - judgeContext: 'a ship/release workflow document', - judgeGoal: 'how to create a PR: merge base, test, review and explore changed behavior, handle required blocked checks, bump metadata, finish and vet every-ship documentation, then verify stable inputs, push and create/update the PR with visible QA and docs outcomes', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('document-release/SKILL.md workflow', async () => { - await runWorkflowJudge({ - testName: 'document-release/SKILL.md workflow', - schemaTransport: true, - suite: 'Ship & Release skill evals', - skillPath: 'document-release/SKILL.md', - startMarker: '# Document Release:', - endMarker: '## Important Rules', - judgeContext: 'a release documentation audit workflow', - judgeGoal: 'how to audit relevant nested docs and authored sources; in ship-owned mode complete a bounded docs-only audit and return a typed result without Git/metadata authority, while standalone mode retains its approval and publication protections', - }); - }, WORKFLOW_JUDGE_TEST_MS); -}); - -// Block 2: Plan Review skills -describeIfSelected('Plan Review skill evals', [ - 'plan-ceo-review/SKILL.md modes', 'plan-eng-review/SKILL.md sections', 'plan-design-review/SKILL.md passes', -], () => { - testIfSelected('plan-ceo-review/SKILL.md modes', async () => { - await runWorkflowJudge({ - testName: 'plan-ceo-review/SKILL.md modes', - schemaTransport: true, - suite: 'Plan Review skill evals', - skillPath: 'plan-ceo-review/SKILL.md', - startMarker: '## Step 0: Nuclear Scope Challenge', - endMarker: '## Review Sections', - judgeContext: 'a CEO/founder plan review framework with 4 scope modes', - judgeGoal: 'how to conduct a CEO-perspective plan review: challenge scope, select a mode (Expansion, Selective Expansion, Hold Scope, Reduction), then review sections interactively', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('plan-eng-review/SKILL.md sections', async () => { - await runWorkflowJudge({ - testName: 'plan-eng-review/SKILL.md sections', - schemaTransport: true, - suite: 'Plan Review skill evals', - skillPath: ENG_REVIEW_EXCERPT.skillPath, - startMarker: '# Plan Review Mode', - endMarker: null, - judgeContext: 'an engineering plan review framework with 4 review sections', - judgeGoal: 'how to review a plan for architecture quality, code quality, test coverage, and performance — walking through each section interactively with AskUserQuestion', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('plan-design-review/SKILL.md passes', async () => { - await runWorkflowJudge({ - testName: 'plan-design-review/SKILL.md passes', - schemaTransport: true, - suite: 'Plan Review skill evals', - skillPath: 'plan-design-review/SKILL.md', - startMarker: '## Review Sections', - endMarker: ASK_QUESTIONS_HEADING, - judgeContext: 'a design plan review framework with 7 review passes', - judgeGoal: 'how to review a plan for design quality using a 0-10 rating method: rate each dimension, explain what a 10 looks like, edit the plan to fix gaps, then re-rate', - }); - }, WORKFLOW_JUDGE_TEST_MS); -}); - -// Block 3: Design skills -describeIfSelected('Design skill evals', ['design-review/SKILL.md fix loop', 'design-consultation/SKILL.md research'], () => { - testIfSelected('design-review/SKILL.md fix loop', async () => { - await runWorkflowJudge({ - testName: 'design-review/SKILL.md fix loop', - schemaTransport: true, - suite: 'Design skill evals', - skillPath: 'design-review/SKILL.md', - startMarker: '## Phase 7:', - endMarker: '## Additional Rules', - judgeContext: 'a design audit triage and fix loop workflow', - judgeGoal: 'how to triage design issues by severity, fix them atomically in source code, commit each fix, and re-verify with before/after screenshots', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('design-consultation/SKILL.md research', async () => { - await runWorkflowJudge({ - testName: 'design-consultation/SKILL.md research', - schemaTransport: true, - suite: 'Design skill evals', - skillPath: 'design-consultation/SKILL.md', - startMarker: '## Phase 0:', - endMarker: '## Phase 4:', - judgeContext: 'a design consultation research and proposal workflow', - judgeGoal: 'how to gather product context, research the competitive landscape, and produce a complete design system proposal with typography, color, spacing, and motion specifications', - }); - }, WORKFLOW_JUDGE_TEST_MS); -}); - -// Block 4: Deploy skills -describeIfSelected('Deploy skill evals', [ - 'land-and-deploy/SKILL.md workflow', 'canary/SKILL.md monitoring loop', - 'benchmark/SKILL.md perf collection', 'setup-deploy/SKILL.md platform setup', -], () => { - testIfSelected('land-and-deploy/SKILL.md workflow', async () => { - await runWorkflowJudge({ - testName: 'land-and-deploy/SKILL.md workflow', - schemaTransport: true, - suite: 'Deploy skill evals', - skillPath: 'land-and-deploy/SKILL.md', - startMarker: '## Step 1: Pre-flight', - endMarker: '## Important Rules', - judgeContext: 'a merge-deploy-verify workflow for landing PRs to production', - judgeGoal: 'how to merge a PR via GitHub CLI, wait for CI and deploy workflows (with platform-specific strategies for Fly.io/Render/Vercel/Netlify), run canary health checks on production, and offer revert if something breaks — with timing data logged for retrospectives', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('canary/SKILL.md monitoring loop', async () => { - await runWorkflowJudge({ - testName: 'canary/SKILL.md monitoring loop', - schemaTransport: true, - suite: 'Deploy skill evals', - skillPath: 'canary/SKILL.md', - startMarker: '### Phase 2: Baseline Capture', - endMarker: '## Important Rules', - judgeContext: 'a post-deploy canary monitoring workflow driving a real browser (Aside first, the gstack headless browser as fallback)', - judgeGoal: 'how to capture baseline screenshots and metrics before deploy, run a continuous monitoring loop checking each page every 60 seconds for console errors and performance regressions, fire alerts with evidence (screenshots), and produce a health report with per-page status and verdict', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('benchmark/SKILL.md perf collection', async () => { - await runWorkflowJudge({ - testName: 'benchmark/SKILL.md perf collection', - schemaTransport: true, - suite: 'Deploy skill evals', - skillPath: 'benchmark/SKILL.md', - startMarker: '### Phase 3: Performance Data Collection', - endMarker: '## Important Rules', - judgeContext: 'a performance regression detection workflow using browser-based Web Vitals measurement (Aside first, the gstack headless browser as fallback)', - judgeGoal: 'how to collect real performance metrics (TTFB, FCP, LCP, bundle sizes, request counts) via performance.getEntries(), compare against baselines with regression thresholds, produce a performance report with delta analysis, and track trends over time', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('setup-deploy/SKILL.md platform setup', async () => { - await runWorkflowJudge({ - testName: 'setup-deploy/SKILL.md platform setup', - schemaTransport: true, - suite: 'Deploy skill evals', - skillPath: 'setup-deploy/SKILL.md', - startMarker: '### Step 2: Detect platform', - endMarker: '## Important Rules', - judgeContext: 'a deployment configuration setup workflow that detects deploy platforms and writes config to CLAUDE.md', - judgeGoal: 'how to detect deploy platforms (Fly.io, Render, Vercel, Netlify, Heroku, GitHub Actions, custom), gather platform-specific configuration (URLs, status commands, health checks, custom hooks), and persist everything to CLAUDE.md for future automated use', - }); - }, WORKFLOW_JUDGE_TEST_MS); -}); - -// Block 5: Other skills -describeIfSelected('Other skill evals', [ - 'retro/SKILL.md instructions', 'qa-only/SKILL.md workflow', 'review/SKILL.md workflow', 'gstack-upgrade/SKILL.md upgrade flow', - 'sync-gbrain/SKILL.md read-only readiness', -], () => { - testIfSelected('review/SKILL.md workflow', async () => { - await runWorkflowJudge({ - testName: 'review/SKILL.md workflow', - suite: 'Other skill evals', - skillPath: 'review/SKILL.md', - startMarker: '## Step 0: Detect platform and base branch', - endMarker: null, - references: [...QA_DISCOVERY_REFERENCES, 'review/checklist.md', 'review/specialists/testing.md'], - judgeContext: 'a pre-landing review with bounded exploratory QA', - judgeGoal: 'how to review and explore changed behavior even for small diffs without a plan or server, preserve report-only discovery and the test_stub ASK gate, handle incomplete probes honestly, and rerun affected evidence after approved repairs', - agentCapability: 'frontier', - // This ~64k-token bundle is the largest workflow input. Thinking and the answer share max_tokens: - // at default effort, 24 measured samples used up to 6.9k output tokens and a CI sample spent all - // 8192 without an answer. The ship-sized streaming cap removes truncation; effort stays default. - maxTokens: 65_536, - stream: true, - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('sync-gbrain/SKILL.md read-only readiness', async () => { - await runWorkflowJudge({ - testName: 'sync-gbrain/SKILL.md read-only readiness', - schemaTransport: true, - suite: 'Other skill evals', - skillPath: 'sync-gbrain/SKILL.md', - startMarker: '## Step 4: Refresh', - endMarker: '## Concurrency note', - judgeContext: 'a source-scoped gbrain readiness and guidance workflow', - judgeGoal: 'how to verify the pinned worktree source using only bounded reads, preserve guidance when the read is unknown, and report the verdict without creating or deleting pages', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('retro/SKILL.md instructions', async () => { - await runWorkflowJudge({ - testName: 'retro/SKILL.md instructions', - schemaTransport: true, - suite: 'Other skill evals', - skillPath: 'retro/SKILL.md', - startMarker: '## Instructions', - endMarker: '## Tone', - judgeContext: 'an engineering retrospective data gathering and analysis workflow', - judgeGoal: 'how to gather git metrics (commit history, test counts, work patterns), analyze them, produce a structured retro report with praise, growth areas, and trend tracking', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('qa-only/SKILL.md workflow', async () => { - await runWorkflowJudge({ - testName: 'qa-only/SKILL.md workflow', - schemaTransport: true, - suite: 'Other skill evals', - skillPath: 'qa-only/SKILL.md', - startMarker: '# /qa-only:', - endMarker: null, - references: QA_DISCOVERY_REFERENCES.filter(file => file !== 'qa/sections/exploratory.md'), - judgeContext: 'a report-only QA testing workflow', - judgeGoal: 'how to select browser/native functional/mixed targets, explore safely with repository tools, report exact contract evidence and coverage limits, conditionally load browser/DX instructions and never mutate product/tests/Git through any tool', - }); - }, WORKFLOW_JUDGE_TEST_MS); - - testIfSelected('gstack-upgrade/SKILL.md upgrade flow', async () => { - await runWorkflowJudge({ - testName: 'gstack-upgrade/SKILL.md upgrade flow', - schemaTransport: true, - suite: 'Other skill evals', - skillPath: 'gstack-upgrade/SKILL.md', - startMarker: '## Inline upgrade flow', - endMarker: '## Standalone usage', - judgeContext: 'a version upgrade detection and execution workflow', - judgeGoal: 'how to detect install type, compare versions, back up current install, upgrade via git or fresh clone, run setup, and show what changed', - }); - }, WORKFLOW_JUDGE_TEST_MS); -}); - // Voice directive eval — tests that the voice section produces the right tone describeIfSelected('Voice directive eval', ['voice directive tone'], () => { testIfSelected('voice directive tone', async () => { @@ -783,20 +210,5 @@ describeIfSelected('Voice directive eval', ['voice directive tone'], () => { }, JUDGE_MS); }); -describeIfSelected('Cookie setup workflow quality', ['setup-browser-cookies/SKILL.md workflow'], () => { - testIfSelected('setup-browser-cookies/SKILL.md workflow', async () => { - await runWorkflowJudge({ - testName: 'setup-browser-cookies/SKILL.md workflow', - suite: 'Cookie setup workflow quality', - skillPath: 'setup-browser-cookies/SKILL.md', - startMarker: '# Setup Browser Cookies', - endMarker: null, - ...COOKIE_WORKFLOW_JUDGE, - schemaTransport: true, - readInput: () => buildCookieWorkflowJudgeInput(ROOT), - }); - }, WORKFLOW_JUDGE_TEST_MS); -}); - // Module-level afterAll — finalize eval collector after all tests complete afterAll(() => finalizeEvalCollector(evalCollector)); diff --git a/test/test-of-test-ratchet.test.ts b/test/test-of-test-ratchet.test.ts index f5e062259..5d6a40821 100644 --- a/test/test-of-test-ratchet.test.ts +++ b/test/test-of-test-ratchet.test.ts @@ -62,7 +62,6 @@ const BASELINE = [ 'test/codex-eval-recording.test.ts', 'test/codex-resume-flag-semantics.test.ts', 'test/context-restore-fragment-verdict.test.ts', // GSTA-91: replays three captured false reds (0/3 on main cb99df4) through the paid context-restore-fragment-match case's fragmentRestoreVerdict - 'test/cookie-workflow-judge-input.test.ts', // slices test/skill-llm-eval.test.ts source (recorded 2026-10 when the ratchet learned paid-source slicing) 'test/cso-ntfs-fixture.test.ts', 'test/cso-windows-build-contract.test.ts', 'test/dependency-security.test.ts', @@ -243,8 +242,6 @@ const BASELINE = [ 'test/ubicloud-runner-reattach.test.ts', // GSTA-151: split from test/ubicloud-runner.test.ts for shard time; runs scripts/ubicloud/ubi-runner.sh through test/helpers/ubi-runner-fake-vm.ts 'test/ubicloud-runner-recovery.test.ts', // GST-1220: recovery bundle and results sink; runs scripts/ubicloud/ubi-runner.sh through test/helpers/ubi-runner-fake-vm.ts 'test/ubicloud-runner-tiers.test.ts', // GSTA-151: split from test/ubicloud-runner.test.ts for shard time; runs scripts/ubicloud/ubi-runner.sh through test/helpers/ubi-runner-fake-vm.ts - 'test/workflow-judge-cache.test.ts', // slices test/skill-llm-eval.test.ts source (recorded 2026-10 when the ratchet learned paid-source slicing) - 'test/workflow-judge-input.test.ts', // slices test/skill-llm-eval.test.ts source (recorded 2026-10 when the ratchet learned paid-source slicing) ]; function freeTests(): string[] { diff --git a/test/touchfiles.test.ts b/test/touchfiles.test.ts index 7bd042652..cd39eb3e4 100644 --- a/test/touchfiles.test.ts +++ b/test/touchfiles.test.ts @@ -94,12 +94,6 @@ describe('selectTests', () => { expect(result.selected).not.toContain('outside-plan-disabled-no-fallback'); }); - test('learnings rendering selects the Eng judge that consumes the changed instruction', () => { - const result = selectTests(['scripts/resolvers/learnings.ts'], LLM_JUDGE_TOUCHFILES); - expect(result.reason).toBe('diff'); - expect(result.selected).toEqual(['plan-eng-review/SKILL.md sections']); - }); - test.each(['bin/gstack-paths', 'bin/gstack-slug', 'scripts/resolvers/design.ts'])( 'Design artifact dependencies select their native consumers: %s', (file) => { const result = selectTests([file], E2E_TOUCHFILES); @@ -176,12 +170,6 @@ describe('selectTests', () => { } }); - test('testing resolver source selects only judges that consume its generated workflow text', () => { - const result = selectTests(['scripts/resolvers/testing.ts'], LLM_JUDGE_TOUCHFILES); - expect(result.reason).toBe('diff'); - expect(result.selected.sort()).toEqual(['plan-eng-review/SKILL.md sections', 'ship/SKILL.md workflow']); - }); - test('bounded shared-code planning selects its consumed resolvers, excluding other Eng sections', () => { const entrypoint = readSkillEffective(path.join(ROOT, 'plan-eng-review/SKILL.md')); const review = fs.readFileSync(path.join(ROOT, 'plan-eng-review/sections/review-sections.md'), 'utf8'); @@ -209,35 +197,6 @@ describe('selectTests', () => { } }); - test.each([ - ['plan-eng-review/sections/review-sections.md', 'plan-eng-review/SKILL.md sections', - 'plan-eng-review/SKILL.md', '## Scope gate', '## Section self-check', '### REGRESSION RULE (mandatory)'], - ['ship/sections/tests.md', 'ship/SKILL.md workflow', - 'ship/SKILL.md', '# Ship:', '## Important Rules', '## Test Framework Bootstrap'], - ['ship/sections/test-coverage.md', 'ship/SKILL.md workflow', - 'ship/SKILL.md', '# Ship:', '## Important Rules', '### REGRESSION RULE (mandatory)'], - ['plan-design-review/sections/review-sections.md', 'plan-design-review/SKILL.md passes', - 'plan-design-review/SKILL.md', '## Review Sections', ASK_QUESTIONS_HEADING, - '## Review Sections (7 passes, after scope is agreed)'], - ])('expanded judge content remains selected by its section alone: %s', (file, judge, skill, start, end, marker) => { - const body = fs.readFileSync(path.join(ROOT, file), 'utf8').replace(/^\n/gm, '').trim(); - const paragraph = body.split(`${marker}\n\n`)[1]?.split('\n\n')[0]; - expect(paragraph?.length).toBeGreaterThan(100); - expect(readWorkflowExcerpt(skill, start, end)).toContain(`${marker}\n\n${paragraph}`); - for (const changed of [file, `${file}.tmpl`]) { - expect(selectTests([changed], LLM_JUDGE_TOUCHFILES).selected).toEqual([judge]); - } - }); - - test.each(['scripts/resolvers/design.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', - 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts'])( - 'Design rendering source selects its workflow judge: %s', (file) => { - const result = selectTests([file], LLM_JUDGE_TOUCHFILES); - expect(result.reason).toBe('diff'); - expect(result.selected).toContain('plan-design-review/SKILL.md passes'); - }); - - test('browse/src change selects browse and qa tests', () => { const result = selectTests(['browse/src/commands.ts'], E2E_TOUCHFILES); expect(result.selected).toContain('browse-basic'); @@ -383,10 +342,7 @@ describe('selectTests', () => { test('works with LLM_JUDGE_TOUCHFILES', () => { const result = selectTests(['qa/SKILL.md'], LLM_JUDGE_TOUCHFILES); - expect(result.selected.sort()).toEqual([ - 'qa/SKILL.md workflow', 'qa/SKILL.md health rubric', 'qa/SKILL.md anti-refusal', - 'qa-only/SKILL.md workflow', 'review/SKILL.md workflow', 'ship/SKILL.md workflow', - ].sort()); + expect(result.selected).toEqual(['qa/SKILL.md anti-refusal']); }); test('SKILL.md.tmpl root template selects root-dependent tests and routing tests', () => { @@ -530,7 +486,7 @@ describe('TOUCHFILES completeness', () => { ); const unique = registeredJudgeTestNames(llmContent); - expect(unique).toHaveLength(24); + expect(unique).toHaveLength(3); const missing = unique.filter(name => !(name in LLM_JUDGE_TOUCHFILES)); if (missing.length > 0) { @@ -549,7 +505,7 @@ describe('TOUCHFILES completeness', () => { testIfSelected('unmapped judge case', async () => {}, 120_000); `; const names = registeredJudgeTestNames(withUnmappedCase); - expect(names).toHaveLength(25); + expect(names).toHaveLength(4); expect(names.filter(name => !(name in LLM_JUDGE_TOUCHFILES))).toEqual(['unmapped judge case']); }); diff --git a/test/workflow-excerpt.test.ts b/test/workflow-excerpt.test.ts index c53726a47..13385079f 100644 --- a/test/workflow-excerpt.test.ts +++ b/test/workflow-excerpt.test.ts @@ -298,7 +298,7 @@ console.log(JSON.stringify({calls, results})); expect(() => readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '# missing')).toThrow('End marker not found'); }); - test('retro judge includes compare semantics and unambiguous report inputs', () => { + test('retro instructions include compare semantics and unambiguous report inputs', () => { const text = readWorkflowExcerpt('retro/SKILL.md', '## Instructions', '## Tone'); expect(text).toContain('## Compare Mode'); expectMentions(text, [['before', 'current', 'second']], 'text'); @@ -309,8 +309,6 @@ console.log(JSON.stringify({calls, results})); expect(text.indexOf('## Capture Learnings')).toBeGreaterThan(text.indexOf('### Step 14:')); expect(text).not.toContain('$(date'); expect(text.match(/today=""/g)).toHaveLength(2); - const judge = readFileSync(join(import.meta.dir, 'skill-llm-eval.test.ts'), 'utf8'); - expect(judge).toMatch(/skillPath: 'retro\/SKILL.md',[\s\S]*?endMarker: '## Tone'/); }); test('deploy gates and navigation timing formulas are executable as documented', () => { diff --git a/test/workflow-judge-cache.test.ts b/test/workflow-judge-cache.test.ts deleted file mode 100644 index 5ad942656..000000000 --- a/test/workflow-judge-cache.test.ts +++ /dev/null @@ -1,634 +0,0 @@ -import { afterEach, describe, expect, spyOn, test } from 'bun:test'; -import { Messages } from '@anthropic-ai/sdk/resources/messages'; -import { callJudge, JudgeRefusalError, DEFAULT_JUDGE_MAX_TOKENS, judgePanel, judgePanelMajority, judgePanelMean, judgePanelMedian, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS, JUDGE_PANEL_SAMPLES } from './helpers/llm-judge'; -import { EVAL_POLICY } from './helpers/periodic-exclude-data'; -import { getCookieWorkflowManualReview } from './helpers/cookie-workflow-manual-review'; -import { resolveEvalModel } from '../lib/eval-model'; -import * as fs from 'node:fs'; -import * as os from 'node:os'; -import * as path from 'node:path'; -import { execFileSync } from 'node:child_process'; -import { browseJudgeFloorsMet, prepareWorkflowJudgeCache, validWorkflowJudgePanel, validWorkflowJudgeScore, workflowJudgeDependencies, type WorkflowCacheOptions } from './helpers/workflow-judge-cache'; -import { readWorkflowJudgeInput, buildWorkflowJudgePrompt, QA_DISCOVERY_REFERENCES, WORKFLOW_JUDGE_RESPONSE_SCHEMA } from './helpers/workflow-judge-input'; - -const roots: string[] = []; -afterEach(() => { for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); }); -const scores = { clarity: 4, completeness: 5, actionability: 4, reasoning: 'Concrete steps' }; -const SAMPLES = JUDGE_PANEL_SAMPLES; -const panelOf = (sample: typeof scores) => Array.from({ length: SAMPLES }, () => sample); -const panel = panelOf(scores); -function fixture() { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-judge-cache-')); roots.push(root); - const files = { - 'package.json': '{"name":"cache-fixture","type":"module"}', 'bun.lock': 'fixed dependencies', - '.github/docker/Dockerfile.ci': 'FROM pinned-image', - '.github/workflows/evals.yml': 'name: Paid evaluation', - 'scripts/test-paid-shards.ts': '#!/usr/bin/env bun\nexport const runner = 1;', - 'scripts/test-strict-output.ts': 'export const output = 1;', - 'scripts/eval-select.ts': 'export const selection = 1;', - 'scripts/test-pr-profile.ts': 'export const profile = 1;', - 'test/skill-llm-eval.test.ts': 'import "./helpers/llm-judge";', - 'test/helpers/workflow-judge-cache.ts': 'export const adapter = 1;', - 'test/helpers/llm-judge.ts': 'import SDK from "@anthropic-ai/sdk"; import "./nested";', - 'test/helpers/nested.ts': 'export const actualRunnerDependency = 1;', - 'lib/eval-model.ts': 'export const model = "model-v1";', - 'test/helpers/eval-budgets.ts': 'export const JUDGE_MS = 120000;', - 'node_modules/@anthropic-ai/sdk/package.json': '{"name":"@anthropic-ai/sdk","main":"index.js"}', - 'node_modules/@anthropic-ai/sdk/index.js': 'module.exports = class SDK {};', - 'example/SKILL.md': '# Start\nRead sections/review.md\n# End\n', - 'example/sections/review.md': 'Complete review and preserve exact answers.\n', - }; - for (const [file, value] of Object.entries(files)) { - const target = path.join(root, file); fs.mkdirSync(path.dirname(target), { recursive: true }); - fs.writeFileSync(target, value); - } - for (const args of [['init', '-b', 'main'], ['config', 'user.name', 'Cache Test'], - ['config', 'user.email', 'cache@example.test'], ['add', '.'], ['commit', '-qm', 'seed']]) - execFileSync('git', args, { cwd: root, stdio: 'pipe', timeout: 5000 }); - const env = { EVALS_CACHE_DIR: path.join(root, 'cache'), EVALS_CACHE_REPOSITORY: 'owner/repo', - EVALS_CACHE_PR: '42', EVALS_CACHE_RUNTIME_ID: 'b'.repeat(64), EVALS_TIER: 'gate', EVALS_RUN_ID: 'free-cache-test' }; - const opts: WorkflowCacheOptions = { root, env, testName: 'example workflow', skillPath: 'example/SKILL.md', - startMarker: '# Start', endMarker: '# End', judgeContext: 'a workflow', judgeGoal: 'how to finish', - thresholds: { clarity: 4, completeness: 3, actionability: 4 }, prompt: '', attempt: 1 }; - const refreshPrompt = () => { opts.prompt = buildWorkflowJudgePrompt(opts, readWorkflowJudgeInput(opts)); }; - refreshPrompt(); - return { root, opts, env, refreshPrompt, cache: () => prepareWorkflowJudgeCache(opts), - entries: () => fs.existsSync(env.EVALS_CACHE_DIR) ? fs.readdirSync(env.EVALS_CACHE_DIR) : [] }; -} - -test('the audited adapter reuses only the exact completed score and original provenance', () => { - const f = fixture(); const first = f.cache(); expect(first.lookup()).toBeNull(); first.publish(panel); - expect(f.entries()).toHaveLength(1); - const reused = f.cache().lookup(); expect(reused?.samples).toEqual(panel); - expect(reused?.reuse.source.runId).toBe('free-cache-test'); - expect(reused?.reuse.source.revision).toMatch(/^[a-f0-9]{40}$/); - expect(reused?.reuse.source.completedAt).toBeLessThanOrEqual(Date.now()); -}); - -test('the dependency closure includes actual installed SDK bytes and local transitive imports', () => { - const f = fixture(); const files = workflowJudgeDependencies(f.root, ['example/SKILL.md']); - expect(files).toContain('node_modules/@anthropic-ai/sdk/index.js'); - expect(files).toContain('node_modules/@anthropic-ai/sdk/package.json'); - expect(files).toContain('test/helpers/nested.ts'); - expect(files).toContain('bun.lock'); - for (const file of ['scripts/test-paid-shards.ts', 'scripts/test-strict-output.ts', - 'scripts/eval-select.ts', 'scripts/test-pr-profile.ts', '.github/workflows/evals.yml']) expect(files).toContain(file); - expect(files).not.toContain('package.json'); -}); - -test('release-label changes preserve reuse; other package semantics invalidate it', () => { - const f = fixture(); f.cache().publish(panel); - const file = path.join(f.root, 'package.json'); - const original = JSON.parse(fs.readFileSync(file, 'utf8')); - fs.writeFileSync(file, JSON.stringify({ ...original, version: '2.0.0' }, null, 2)); - expect(f.cache().lookup()?.samples).toEqual(panel); - for (const change of [{ scripts: { 'test:gate': 'changed command' } }, { dependencies: { 'some-sdk': '2.0.0' } }]) { - fs.writeFileSync(file, JSON.stringify({ ...original, ...change, version: '2.0.0' })); - expect(f.cache().lookup()).toBeNull(); - } -}); - -for (const file of ['test/helpers/nested.ts', 'node_modules/@anthropic-ai/sdk/index.js', - 'node_modules/@anthropic-ai/sdk/package.json', 'bun.lock', 'example/sections/review.md', - 'scripts/test-paid-shards.ts', 'scripts/test-strict-output.ts', 'scripts/eval-select.ts', - 'scripts/test-pr-profile.ts', '.github/workflows/evals.yml']) { - test(`changes in ${file} require new evaluation`, () => { - const f = fixture(); f.cache().publish(panel); const target = path.join(f.root, file); - fs.appendFileSync(target, file.endsWith('.json') ? ' ' : '\n// changed'); - f.refreshPrompt(); expect(f.cache().lookup()).toBeNull(); - }); -} - -test('changed sources during an attempt and mismatched actual prompt cannot publish', () => { - const f = fixture(); const before = f.cache(); - fs.appendFileSync(path.join(f.root, 'example/sections/review.md'), 'new finding'); - before.publish(panel); expect(f.entries()).toHaveLength(0); - f.refreshPrompt(); f.opts.prompt += ' hidden new request'; f.cache().publish(panel); - expect(f.entries()).toHaveLength(0); -}); - -for (const [key, value] of Object.entries({ EVALS_FRESH: '1', EVALS_TIER: 'periodic', - EVALS_CACHE_PURPOSE: 'release', EVALS_CACHE_RUNTIME_ID: 'mutable:latest', EVALS_CACHE_PR: '', - EVALS_CACHE_REPOSITORY: '', NODE_OPTIONS: '--require=unknown', BUN_OPTIONS: '--preload=unknown', - ANTHROPIC_BASE_URL: 'https://custom-provider.example.test' })) { - test(`${key}=${value} is fresh or ineligible`, () => { - const f = fixture(); f.cache().publish(panel); - f.opts.env = { ...f.env, [key]: value }; const cache = f.cache(); - expect(cache.lookup()).toBeNull(); cache.publish(panel); expect(f.entries()).toHaveLength(1); - }); -} - -test('runtime/model/threshold changes miss, and retries never reuse or publish', () => { - const f = fixture(); f.cache().publish(panel); - for (const overrides of [{ GSTACK_EVAL_MODEL_JUDGE: 'different-model' }, { EVALS_CACHE_RUNTIME_ID: 'c'.repeat(64) }]) { - f.opts.env = { ...f.env, ...overrides }; expect(f.cache().lookup()).toBeNull(); - } - f.opts.env = f.env; f.opts.thresholds.clarity = 5; expect(f.cache().lookup()).toBeNull(); - f.opts.thresholds.clarity = 4; f.opts.attempt = 2; const retry = f.cache(); - expect(retry.lookup()).toBeNull(); retry.publish(panel); expect(f.entries()).toHaveLength(1); -}); - -test('frontier reader calibration cannot reuse a score from the unspecified-reader rubric', () => { - const f = fixture(); f.cache().publish(panel); - const original = f.opts.prompt; - f.opts.agentCapability = 'frontier'; f.refreshPrompt(); - expect(f.opts.prompt).not.toBe(original); - expect(f.cache().lookup()).toBeNull(); - f.cache().publish(panel); - expect(f.entries()).toHaveLength(2); - expect(f.cache().lookup()?.samples).toEqual(panel); - delete f.opts.agentCapability; f.refreshPrompt(); - expect(f.opts.prompt).toBe(original); - expect(f.cache().lookup()?.samples).toEqual(panel); -}); - -test('a pinned workflow judge model overrides the global model and changes the cache identity', () => { - const f = fixture(); - f.opts.model = 'claude-sonnet-4-6'; - f.cache().publish(panel); - expect(f.entries()).toHaveLength(1); - f.opts.env = { ...f.env, GSTACK_EVAL_MODEL_JUDGE: 'different-global-model' }; - expect(f.cache().lookup()?.samples).toEqual(panel); - f.opts.model = 'claude-opus-4-7'; - expect(f.cache().lookup()).toBeNull(); -}); - -test('failed assertions, missing provenance, and missing imported dependencies cannot supply a receipt', () => { - const f = fixture(); f.cache().publish(panelOf({ ...scores, clarity: 3 })); expect(f.entries()).toHaveLength(0); - f.opts.env = { ...f.env, EVALS_RUN_ID: '' }; f.cache().publish(panel); expect(f.entries()).toHaveLength(0); - f.opts.env = f.env; fs.unlinkSync(path.join(f.root, 'test/helpers/nested.ts')); - f.cache().publish(panel); expect(f.entries()).toHaveLength(0); -}); - -test('cached payload schema remains small and cannot carry operational fields', () => { - expect(validWorkflowJudgeScore(scores, { clarity: 4, completeness: 3, actionability: 4 })).toBe(true); - for (const changed of [{ ...scores, prompt: 'private request' }, { ...scores, clarity: '4' }, - { ...scores, reasoning: null }, { ...scores, clarity: 6 }]) - expect(validWorkflowJudgeScore(changed as any, { clarity: 4, completeness: 3, actionability: 4 })).toBe(false); -}); - -test('workflow registration preserves model work and reserves only terminal-recording grace', () => { - const source = fs.readFileSync(path.join(import.meta.dir, 'skill-llm-eval.test.ts'), 'utf8'); - const body = source.split('async function runWorkflowJudge')[1]!.split('// Block 1:')[0]!; - const stages = ['workflowJudgeAttempts.set', 'readWorkflowJudgeInput(', 'cache.lookup()', - 'judgePanel(() => callJudge(prompt, opts.model, { signal: controller.signal, max_tokens: maxTokens,', - 'scores = judgePanelMedian(samples, JUDGE_SCORE_DIMENSIONS);', - 'expect(scores.clarity)', 'expect(scores.completeness)', 'expect(scores.actionability)', 'cache.publish(samples, active)'] - .map(stage => body.indexOf(stage)); - expect(stages.every(position => position >= 0)).toBe(true); - expect(stages).toEqual([...stages].sort((a, b) => a - b)); - const calls = [...source.matchAll(/await runWorkflowJudge\(/g)].map(match => match.index!); - expect(calls.length).toBeGreaterThan(0); - for (const call of calls) { - const timeout = /\n\s*\}, (\w+)\);/.exec(source.slice(call))?.[1]; - expect(timeout, `runWorkflowJudge registration at offset ${call} must use WORKFLOW_JUDGE_TEST_MS`).toBe('WORKFLOW_JUDGE_TEST_MS'); - } -}); - -function actualCallback(f: ReturnType, overrides: { - judge?: (prompt: string, model: string | undefined, options: { signal: AbortSignal }) => Promise; - read?: typeof readWorkflowJudgeInput; - prepare?: typeof prepareWorkflowJudgeCache; - clock?: () => number; - budget?: number; - allowance?: number; - setTimer?: typeof setTimeout; - clearTimer?: typeof clearTimeout; -} = {}) { - // Execute the production callback with only its provider and collector replaced. - // Keep all prompt construction, cache decisions and Bun assertions intact. - const source = fs.readFileSync(path.join(import.meta.dir, 'skill-llm-eval.test.ts'), 'utf8'); - const callback = source.slice(source.indexOf('async function runWorkflowJudge'), source.indexOf('// Block 1:')); - const javascript = new Bun.Transpiler({ loader: 'ts' }).transformSync(callback); - const records: any[] = [], signals: AbortSignal[] = [], prompts: string[] = []; - const attempts = new Map(); - const run = new Function('ROOT', 'readWorkflowJudgeInput', - 'buildWorkflowJudgePrompt', 'prepareWorkflowJudgeCache', 'workflowJudgeAttempts', 'callJudge', - 'evalCollector', 'expect', 'console', 'performance', 'JUDGE_MS', 'WORKFLOW_JUDGE_RECORD_MS', - 'setTimeout', 'clearTimeout', 'JudgeRefusalError', 'getCookieWorkflowManualReview', 'DEFAULT_JUDGE_MAX_TOKENS', 'resolveEvalModel', - 'WORKFLOW_JUDGE_RESPONSE_SCHEMA', 'validWorkflowJudgeScore', - 'judgePanel', 'judgePanelMean', 'judgePanelMedian', 'judgePanelReasoning', 'JUDGE_SCORE_DIMENSIONS', - `${javascript}\nreturn runWorkflowJudge;`)( - f.root, overrides.read ?? readWorkflowJudgeInput, buildWorkflowJudgePrompt, - (options: WorkflowCacheOptions) => (overrides.prepare ?? prepareWorkflowJudgeCache)({ ...options, env: f.env }), - attempts, async (prompt: string, model: string | undefined, options: { signal: AbortSignal }) => { - prompts.push(prompt); signals.push(options.signal); - return overrides.judge ? overrides.judge(prompt, model, options) : scores; - }, { addTest: (entry: any) => records.push(entry) }, expect, { log() {} }, - overrides.clock ? { now: overrides.clock } : performance, overrides.budget ?? 120_000, overrides.allowance ?? 5_000, - overrides.setTimer ?? setTimeout, overrides.clearTimer ?? clearTimeout, - JudgeRefusalError, getCookieWorkflowManualReview, DEFAULT_JUDGE_MAX_TOKENS, resolveEvalModel, - WORKFLOW_JUDGE_RESPONSE_SCHEMA, validWorkflowJudgeScore, - judgePanel, judgePanelMean, judgePanelMedian, judgePanelReasoning, JUDGE_SCORE_DIMENSIONS); - return { run, records, signals, prompts, attempts, options: { ...f.opts, suite: 'Cache regression' } }; -} - -test('the actual workflow callback preserves the pinned model and frontier rubric for custom inputs', async () => { - const f = fixture(); - const models: Array = []; - const actual = actualCallback(f, { judge: async (_prompt, model) => { models.push(model); return scores; } }); - await actual.run({ ...actual.options, model: 'claude-sonnet-4-6', agentCapability: 'frontier', - readInput: () => readWorkflowJudgeInput(f.opts) }); - expect(models).toEqual(Array(SAMPLES).fill('claude-sonnet-4-6')); - expect(actual.prompts[0]).toContain('GPT-5.6 Sol-level capability or stronger'); - expect(actual.records[0]).toMatchObject({ passed: true, model: 'claude-sonnet-4-6', prompt: actual.prompts[0] }); -}); - -test.each(['ship', 'review'])('the registered %s callback sends the frontier rubric and still rejects subthreshold clarity', async skill => { - const source = fs.readFileSync(path.join(import.meta.dir, 'skill-llm-eval.test.ts'), 'utf8'); - const registration = source.match(new RegExp(`testIfSelected\\('${skill}/SKILL\\.md workflow',[\\s\\S]*?await runWorkflowJudge\\(\\{([\\s\\S]*?)\\n \\}\\);`)); - expect(registration).not.toBeNull(); - const registered = new Function('QA_DISCOVERY_REFERENCES', `return ({${registration![1]}});`)(QA_DISCOVERY_REFERENCES); - const f = fixture(); - Object.assign(f.env, { EVALS_FRESH: '1' }); - const options = { ...registered, skillPath: f.opts.skillPath, startMarker: f.opts.startMarker, - endMarker: f.opts.endMarker, references: [] }; - const passing = actualCallback(f, { judge: async () => ({ ...scores, clarity: 3 }) }); - await passing.run(options); - expect(passing.prompts).toHaveLength(SAMPLES); - expect(passing.prompts[0]).toContain('GPT-5.6 Sol-level capability or stronger'); - expect(passing.records[0]).toMatchObject({ passed: true, execution: 'executed', judge_scores: { clarity: 3 } }); - const failing = actualCallback(f, { judge: async () => ({ ...scores, clarity: 2 }) }); - await expect(failing.run(options)).rejects.toThrow(); - expect(failing.prompts).toEqual(passing.prompts); - expect(failing.records[0]).toMatchObject({ passed: false, execution: 'executed', - exit_reason: 'validation_failed', judge_scores: { clarity: 2 } }); - expect(f.entries()).toHaveLength(0); -}); - -test('the actual workflow callback executes once, reuses with provenance, and preserves assertion failures', async () => { - const f = fixture(); const first = actualCallback(f); - const options = { ...f.opts, suite: 'Cache regression' }; - await first.run(options); - expect(first.prompts).toEqual(Array(SAMPLES).fill(f.opts.prompt)); expect(f.entries()).toHaveLength(1); - expect(first.records[0]).toMatchObject({ passed: true, execution: 'executed', cost_usd: 0.02 * SAMPLES }); - expect(first.records[0]).not.toHaveProperty('prompt'); - expect(first.records[0]).not.toHaveProperty('model'); - const reused = actualCallback(f, { judge: async () => ({ ...scores, clarity: 1 }) }); - await reused.run(options); - expect(reused.prompts).toHaveLength(0); - expect(reused.records[0]).toMatchObject({ passed: true, execution: 'reused', cost_usd: 0, - reused_from: { run_id: 'free-cache-test' } }); - // Each original assertion still rejects failed model output before publication. - for (const field of ['clarity', 'completeness', 'actionability']) { - const changed = { ...options, testName: `failed ${field}` }; - const failed = actualCallback(f, { judge: async () => ({ ...scores, [field]: 1 }) }); - await expect(failed.run(changed)).rejects.toThrow(); - expect(failed.records).toHaveLength(1); - expect(failed.records[0]).toMatchObject({ passed: false, execution: 'executed', exit_reason: 'validation_failed' }); - expect(f.entries()).toHaveLength(1); - } -}); - -test('a timed-out provider records once, aborts, and cannot overwrite its successful fresh retry', async () => { - const f = fixture(); let completeLate!: (value: typeof scores) => void; let calls = 0, now = 0; - const timers = new Map void>(); let timerId = 0; - const h = actualCallback(f, { budget: 20, clock: () => now, - setTimer: ((callback: () => void) => { timers.set(++timerId, callback); return timerId; }) as any, - clearTimer: ((id: number) => { timers.delete(id); }) as any, - judge: async () => ++calls === 1 - ? new Promise(resolve => { completeLate = resolve; }) : scores }); - const expired = h.run(h.options).catch((error: Error) => error); - now = 20; - for (const callback of [...timers.values()]) callback(); - expect((await expired).message).toContain('deadline'); - expect(h.records).toHaveLength(1); - expect(h.records[0]).toMatchObject({ passed: false, exit_reason: 'timeout' }); - expect(h.signals[0].aborted).toBe(true); - expect(f.entries()).toHaveLength(0); - await h.run(h.options); - expect(h.records.map(record => record.passed)).toEqual([false, true]); - expect(h.attempts.get(f.opts.testName).attempt).toBe(2); - completeLate(scores); await new Promise(resolve => setImmediate(resolve)); - expect(h.records).toHaveLength(2); - expect(f.entries()).toHaveLength(0); // Retry passes never become receipts. -}); - -test('a superseding attempt cancels its predecessor before either can record a stale pass', async () => { - const f = fixture(); let completeLate!: (value: typeof scores) => void; let calls = 0; - const h = actualCallback(f, { judge: async () => ++calls === 1 - ? new Promise(resolve => { completeLate = resolve; }) : scores }); - const old = h.run(h.options).catch((error: Error) => error); - await h.run(h.options); - expect((await old).name).toBe('WorkflowJudgeSuperseded'); - expect(h.signals[0].aborted).toBe(true); - completeLate(scores); await new Promise(resolve => setImmediate(resolve)); - expect(h.records.map(record => [record.passed, record.exit_reason])).toEqual([[false, 'cancelled'], [true, undefined]]); - expect(f.entries()).toHaveLength(0); -}); - -test('a failed input read consumes attempt one and prevents a retry from borrowing or publishing a receipt', async () => { - const f = fixture(); f.cache().publish(panel); const receipt = fs.readFileSync(path.join(f.env.EVALS_CACHE_DIR, f.entries()[0]), 'utf8'); - let reads = 0; - const h = actualCallback(f, { read: options => { if (++reads === 1) throw new Error('Missing workflow fixture'); return readWorkflowJudgeInput(options); } }); - await expect(h.run(h.options)).rejects.toThrow('Missing workflow fixture'); - expect(h.records[0]).toMatchObject({ passed: false, exit_reason: 'harness_error' }); - await h.run(h.options); - expect(h.prompts).toHaveLength(SAMPLES); - expect(h.records.map(record => record.execution)).toEqual(['executed', 'executed']); - expect(h.attempts.get(f.opts.testName).attempt).toBe(2); - expect(fs.readFileSync(path.join(f.env.EVALS_CACHE_DIR, f.entries()[0]), 'utf8')).toBe(receipt); -}); - -test('monotonic expiry after a synchronous preparation or late model response refuses success', async () => { - for (const phase of ['preparation', 'response']) { - const f = fixture(); let now = 0; - const h = actualCallback(f, { budget: 20, clock: () => now, - prepare: options => { const cache = prepareWorkflowJudgeCache(options); if (phase === 'preparation') now = 21; return cache; }, - judge: async () => { now = 21; return scores; } }); - await expect(h.run(h.options)).rejects.toThrow('deadline'); - expect(h.records).toHaveLength(1); - expect(h.records[0]).toMatchObject({ passed: false, exit_reason: 'timeout', duration_ms: 21 }); - expect(h.prompts).toHaveLength(phase === 'preparation' ? 0 : SAMPLES); - expect(f.entries()).toHaveLength(0); - } -}); - -test('publication rechecks after input scanning and withdraws a receipt if recording expires', async () => { - const f = fixture(); let checks = 0; - f.cache().publish(panel, () => ++checks < 2); - expect(checks).toBe(2); expect(f.entries()).toHaveLength(0); - let now = 0; - const h = actualCallback(f, { budget: 20, allowance: 5, clock: () => now, - prepare: options => { - const cache = prepareWorkflowJudgeCache(options); - return { ...cache, publish: (result, active) => { const discard = cache.publish(result, active); now = 26; return discard; } }; - } }); - await expect(h.run(h.options)).rejects.toThrow('recording deadline'); - expect(h.records).toHaveLength(1); - expect(h.records[0]).toMatchObject({ passed: false, exit_reason: 'timeout' }); - expect(f.entries()).toHaveLength(0); -}); - -test('provider exceptions retain their diagnostic and produce one failed record without cache output', async () => { - const f = fixture(); const error = new Error('Provider connection failed'); - const h = actualCallback(f, { judge: async () => { throw error; } }); - await expect(h.run(h.options)).rejects.toBe(error); - expect(h.records).toHaveLength(1); - expect(h.records[0]).toMatchObject({ passed: false, exit_reason: 'harness_error' }); - expect(h.records[0].error).toContain(error.message); - expect(f.entries()).toHaveLength(0); -}); - -test('the actual workflow callback preserves the complete public API body; cancellation is only a request option', async () => { - const f = fixture(); - // Replace the SDK method before invoking the real helper: no network requests. - const create = spyOn(Messages.prototype, 'create').mockResolvedValue({ - stop_reason: 'end_turn', content: [{ type: 'text', text: JSON.stringify(scores) }], - } as any); - try { - const h = actualCallback(f, { judge: (prompt, model, options) => callJudge(prompt, model, options) }); - await h.run(h.options); - expect(create).toHaveBeenCalledTimes(SAMPLES); - expect(create.mock.calls[0]).toEqual([{ - model: resolveEvalModel('judge'), max_tokens: 8192, - messages: [{ role: 'user', content: f.opts.prompt }], - }, { signal: h.signals[0] }]); - expect(h.records).toHaveLength(1); - expect(h.records[0].passed).toBe(true); - } finally { create.mockRestore(); } -}); - -test('Ship sends its authorized 64k cap and compact response contract through the streaming SDK boundary', async () => { - const source = fs.readFileSync(path.join(import.meta.dir, 'skill-llm-eval.test.ts'), 'utf8'); - const registration = source.match(/testIfSelected\('ship\/SKILL\.md workflow',[\s\S]*?await runWorkflowJudge\(\{([\s\S]*?)\n \}\);/); - expect(registration).not.toBeNull(); - const options = new Function('QA_DISCOVERY_REFERENCES', `return ({${registration![1]}});`)(QA_DISCOVERY_REFERENCES); - expect(options.schemaTransport).toBe(true); - expect(options.compactReasoning).toBe(true); - expect(options.maxTokens).toBe(65_536); - expect(options.stream).toBe(true); - expect(options.effort).toBe('medium'); - expect(source.match(/^\s+effort: '/gm)).toHaveLength(1); - // The ~64k-token review bundle gets only ship's output cap and streaming; its effort and response format stay default. - const review = source.match(/testIfSelected\('review\/SKILL\.md workflow',[\s\S]*?await runWorkflowJudge\(\{([\s\S]*?)\n \}\);/); - expect(review).not.toBeNull(); - const reviewOptions = new Function('QA_DISCOVERY_REFERENCES', `return ({${review![1]}});`)(QA_DISCOVERY_REFERENCES); - expect([reviewOptions.maxTokens, reviewOptions.stream, reviewOptions.effort, reviewOptions.schemaTransport, reviewOptions.compactReasoning]) - .toEqual([65_536, true, undefined, undefined, undefined]); - expect(WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning).not.toHaveProperty('pattern'); - expect(WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning).not.toHaveProperty('maxLength'); - expect(source.match(/compactReasoning: true/g)).toHaveLength(1); - const f = fixture(); - const stream = spyOn(Messages.prototype, 'stream').mockReturnValue({ on() { return this; }, finalMessage: async () => ({ - stop_reason: 'end_turn', content: [{ type: 'text', text: JSON.stringify(scores) }], - }) } as any); - try { - const h = actualCallback(f, { judge: (prompt, model, request) => callJudge(prompt, model, request) }); - await h.run({ ...h.options, schemaTransport: options.schemaTransport, compactReasoning: options.compactReasoning, maxTokens: options.maxTokens, stream: options.stream, effort: options.effort }); - expect(stream.mock.calls[0]).toEqual([{ - model: resolveEvalModel('judge'), max_tokens: 65_536, - output_config: { format: { type: 'json_schema', schema: WORKFLOW_JUDGE_RESPONSE_SCHEMA }, effort: 'medium' }, - messages: [{ role: 'user', content: f.opts.prompt }], - }, { signal: h.signals[0] }]); - expect(h.records[0]).toMatchObject({ passed: true, judge_scores: { clarity: 4, completeness: 5, actionability: 4 } }); - expect(f.entries()).toHaveLength(1); - expect(f.cache().lookup()).toBeNull(); - f.opts.schemaTransport = true; - f.opts.compactReasoning = true; - f.opts.maxTokens = 65_536; - f.opts.stream = true; - expect(f.cache().lookup()).toBeNull(); - f.opts.effort = 'medium'; - expect(f.cache().lookup()?.samples).toEqual(panel); - } finally { stream.mockRestore(); } -}); - -test('changing response serialization misses the cache even when prompt and model match', () => { - const f = fixture(); f.cache().publish(panel); - f.opts.schemaTransport = true; - expect(f.cache().lookup()).toBeNull(); - f.cache().publish(panel); - expect(f.entries()).toHaveLength(2); - expect(f.cache().lookup()?.samples).toEqual(panel); - const description = WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning.description; - try { - WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning.description += ' Changed response contract.'; - expect(f.cache().lookup()).toBeNull(); - } finally { WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning.description = description; } - expect(f.cache().lookup()?.samples).toEqual(panel); - f.opts.schemaTransport = false; - expect(f.cache().lookup()?.samples).toEqual(panel); -}); - -test('schema transport and the compact-reasoning validator independently affect workflow cache identity', () => { - const f = fixture(); f.cache().publish(panel); - f.opts.schemaTransport = true; - expect(f.cache().lookup()).toBeNull(); - f.cache().publish(panel); - f.opts.compactReasoning = true; - expect(f.cache().lookup()).toBeNull(); - f.cache().publish(panel); - expect(f.entries()).toHaveLength(3); - f.opts.compactReasoning = false; - expect(f.cache().lookup()?.samples).toEqual(panel); - const long = { ...scores, reasoning: Array(150).fill('word').join(' ') }; - expect(validWorkflowJudgeScore(long, { clarity: 1, completeness: 1, actionability: 1 })).toBe(true); - expect(validWorkflowJudgeScore(long, { clarity: 1, completeness: 1, actionability: 1 }, true)).toBe(false); -}); - -test('schema transport alone sends the schema without applying the compact-reasoning validator', async () => { - const f = fixture(); - const long = { ...scores, reasoning: Array(150).fill('word').join(' ') }; - const create = spyOn(Messages.prototype, 'create').mockResolvedValue({ - stop_reason: 'end_turn', content: [{ type: 'text', text: JSON.stringify(long) }], - } as any); - try { - const h = actualCallback(f, { judge: (prompt, model, request) => callJudge(prompt, model, request) }); - await h.run({ ...h.options, schemaTransport: true }); - expect(create.mock.calls[0]).toEqual([{ - model: resolveEvalModel('judge'), max_tokens: 8192, - output_config: { format: { type: 'json_schema', schema: WORKFLOW_JUDGE_RESPONSE_SCHEMA } }, - messages: [{ role: 'user', content: f.opts.prompt }], - }, { signal: h.signals[0] }]); - expect(h.records[0]).toMatchObject({ passed: true }); - const validated = actualCallback(f, { judge: (prompt, model, request) => callJudge(prompt, model, request) }); - await expect(validated.run({ ...validated.options, testName: 'validated', schemaTransport: true, compactReasoning: true })).rejects.toThrow('compact response contract'); - } finally { create.mockRestore(); } -}); - -test('the actual cap and streaming transport independently affect workflow cache identity', () => { - const f = fixture(); f.cache().publish(panel); - f.opts.maxTokens = 65_536; - expect(f.cache().lookup()).toBeNull(); - f.cache().publish(panel); - f.opts.stream = true; - expect(f.cache().lookup()).toBeNull(); - f.cache().publish(panel); - expect(f.entries()).toHaveLength(3); - expect(f.cache().lookup()?.samples).toEqual(panel); - delete f.opts.maxTokens; - delete f.opts.stream; - expect(f.cache().lookup()?.samples).toEqual(panel); -}); - -test('the structured callback rejects incomplete, schema-invalid and below-threshold answers without cache credit', async () => { - const invalid = [null, {}, { ...scores, extra: 'not a score field' }, { ...scores, clarity: '4' }, - { ...scores, clarity: 4.5 }, { ...scores, clarity: 6 }, { ...scores, reasoning: '' }, - { ...scores, reasoning: Array(150).fill('word').join(' ') }, { ...scores, clarity: 2 }, - { ...scores, completeness: 2 }, { ...scores, actionability: 3 }]; - const responses: Array<{ stop_reason: string; value: unknown }> = [ - ...['max_tokens', 'refusal', 'stop_sequence'].map(stop_reason => ({ stop_reason, value: scores })), - ...invalid.map(value => ({ stop_reason: 'end_turn', value })), - ]; - const stream = spyOn(Messages.prototype, 'stream'); - const diagnostics = spyOn(console, 'error').mockImplementation(() => {}); - try { - for (const response of responses) { - const f = fixture(); - stream.mockReturnValue({ on() { return this; }, finalMessage: async () => ({ stop_reason: response.stop_reason, - content: [{ type: 'text', text: JSON.stringify(response.value) }] }) } as any); - const h = actualCallback(f, { judge: (prompt, model, request) => callJudge(prompt, model, request) }); - await expect(h.run({ ...h.options, schemaTransport: true, compactReasoning: true, maxTokens: 65_536, stream: true })).rejects.toThrow(); - expect(h.records).toHaveLength(1); - expect(h.records[0].passed).toBe(false); - expect(f.entries()).toHaveLength(0); - } - expect(validWorkflowJudgeScore({ ...scores, reasoning: Array(149).fill('word').join(' ') }, { clarity: 1, completeness: 1, actionability: 1 }, true)).toBe(true); - } finally { stream.mockRestore(); diagnostics.mockRestore(); } -}); - -// --- Judge panel policy (EVAL_POLICY.judge): fixed concurrent samples, per-dimension -// mean and boolean majority against unchanged thresholds, an erroring sample fails -// the whole panel and is never resampled. The provider is always a stub. -const panelScore = (clarity: number, completeness = 4, actionability = 4) => ({ clarity, completeness, actionability, reasoning: `c${clarity}` }); -const panelThresholds = { clarity: 3, completeness: 3, actionability: 4 }; -const panelRefusal = () => new JudgeRefusalError({ id: 'msg_1', _request_id: 'req_1', model: 'm', usage: { input_tokens: 1, output_tokens: 0 }, content: [] }); - -describe('judge panel', () => { - test('the pre-registered panel is three samples, and the helper restates EVAL_POLICY exactly', () => { - expect(EVAL_POLICY.judge.samples).toBe(3); - expect(JUDGE_PANEL_SAMPLES).toBe(EVAL_POLICY.judge.samples); - }); - - test('draws every sample concurrently before any resolves', async () => { - let started = 0; - const releases: Array<() => void> = []; - const panel = judgePanel(() => new Promise(resolve => { started += 1; releases.push(() => resolve(started)); })); - await Promise.resolve(); - expect(started).toBe(SAMPLES); - releases.forEach(release => release()); - expect(await panel).toHaveLength(SAMPLES); - }); - - test('an erroring sample fails the panel and is never resampled', async () => { - let calls = 0; - const panel = judgePanel(async () => { - calls += 1; - if (calls === 2) throw new Error('Judge returned non-JSON: nope'); - return panelScore(5); - }); - await expect(panel).rejects.toThrow('non-JSON'); - expect(calls).toBe(SAMPLES); - }); - - test('a refusal on every sample stays a provider refusal; a partial refusal is an ordinary failure', async () => { - await expect(judgePanel(async () => { throw panelRefusal(); })).rejects.toBeInstanceOf(JudgeRefusalError); - let calls = 0; - const partial = judgePanel(async () => { if (++calls === 1) throw panelRefusal(); return panelScore(4); }); - const error = await partial.then(() => null, (reason: unknown) => reason); - expect(error).toBeInstanceOf(Error); - expect(error).not.toBeInstanceOf(JudgeRefusalError); - expect(String(error)).toContain(`sample 1 of ${SAMPLES} failed beside scored samples`); - }); - - test('numeric dimensions gate on the per-dimension median: 2 of 3 samples decide, the mean is only reported', () => { - const outvoted = judgePanelMedian([panelScore(2), panelScore(4), panelScore(4)], JUDGE_SCORE_DIMENSIONS); - expect(outvoted.clarity).toBe(4); - expect(judgePanelMean([panelScore(2), panelScore(4), panelScore(4)], JUDGE_SCORE_DIMENSIONS).clarity).toBeCloseTo(10 / 3); - const low = judgePanelMedian([panelScore(2), panelScore(2), panelScore(4)], JUDGE_SCORE_DIMENSIONS); - expect(low.clarity).toBeLessThan(panelThresholds.clarity); - // No compensation across dimensions: each is gated on its own. - expect(judgePanelMedian([panelScore(5, 1), panelScore(5, 1), panelScore(5, 1)], JUDGE_SCORE_DIMENSIONS).completeness).toBe(1); - }); - - test('malformed sample fields fail the panel instead of averaging to NaN', () => { - expect(() => judgePanelMean([panelScore(4), { ...panelScore(4), clarity: '4' as unknown as number }, panelScore(4)], JUDGE_SCORE_DIMENSIONS)).toThrow('sample 2 has non-numeric clarity'); - expect(() => judgePanelMean([panelScore(4), null as unknown as ReturnType], JUDGE_SCORE_DIMENSIONS)).toThrow('sample 2'); - expect(() => judgePanelMean([], JUDGE_SCORE_DIMENSIONS)).toThrow('no samples'); - }); - - test('boolean fields gate on a strict majority', () => { - const vote = (...values: boolean[]) => judgePanelMajority(values.map(value => ({ ok: value })), 'ok'); - expect(vote(true, true, false)).toBe(true); - expect(vote(true, false, false)).toBe(false); - expect(vote(true, false)).toBe(false); - expect(() => judgePanelMajority([{ ok: true }, { ok: 'yes' }], 'ok')).toThrow('sample 2 has non-boolean ok'); - }); - - test('reasoning keeps every sample, numbered, even for malformed samples', () => { - expect(judgePanelReasoning([panelScore(4), null, { reasoning: 7 }])).toBe('[sample 1] c4\n[sample 2] \n[sample 3] '); - }); - - test('the cache stores and validates only a complete panel against the mean', () => { - expect(validWorkflowJudgePanel({ samples: [panelScore(2), panelScore(4), panelScore(4)] }, panelThresholds)).toBe(true); - expect(validWorkflowJudgePanel({ samples: [panelScore(2), panelScore(2), panelScore(4)] }, panelThresholds)).toBe(false); - expect(validWorkflowJudgePanel({ samples: [panelScore(4), panelScore(4)] }, panelThresholds)).toBe(false); - expect(validWorkflowJudgePanel({ samples: [panelScore(4), panelScore(4), panelScore(4), panelScore(4)] }, panelThresholds)).toBe(false); - expect(validWorkflowJudgePanel({ samples: [panelScore(4), panelScore(4), { ...panelScore(4), clarity: 6 }] }, panelThresholds)).toBe(false); - expect(validWorkflowJudgePanel({ samples: [panelScore(4), panelScore(4), panelScore(4)], prompt: 'x' }, panelThresholds)).toBe(false); - expect(validWorkflowJudgePanel(panelScore(4), panelThresholds)).toBe(false); - }); - - test('every judge in the quality file samples through the panel, never a lone call', () => { - const source = fs.readFileSync(path.join(import.meta.dir, 'skill-llm-eval.test.ts'), 'utf8'); - const calls = [...source.matchAll(/\b(?:callJudge<[^>(]*(?:<[^>]*>[^>(]*)*>|judge)\(/g)]; - expect(calls.length).toBeGreaterThanOrEqual(8); - for (const call of calls) { - expect(source.slice(Math.max(0, call.index! - 25), call.index), `unpaneled judge call at offset ${call.index}`).toMatch(/judgePanel\(\(\) => $/); - } - expect(source).not.toMatch(/\bscores\.reasoning\b|\bresult\.reasoning\b/); - }); -}); - -// Stored browse reference panel means (recorded before EVAL_POLICY v3) against the judge floors. -{ - type Scores = { clarity: number; completeness: number; actionability: number }; - const stored = JSON.parse(fs.readFileSync(path.join(import.meta.dir, 'fixtures/browse-judge/panel-means.json'), 'utf8')) as - { known_good: Record; known_bad: Record }; - - describe('browse judge floors', () => { - test.each(Object.entries(stored.known_good))('passes %s', (_name, scores) => expect(browseJudgeFloorsMet(scores)).toBe(true)); - test.each(Object.entries(stored.known_bad))('fails %s', (_name, scores) => expect(browseJudgeFloorsMet(scores)).toBe(false)); - }); -} diff --git a/test/workflow-judge-input.test.ts b/test/workflow-judge-input.test.ts index ee3c751b5..141b78a53 100644 --- a/test/workflow-judge-input.test.ts +++ b/test/workflow-judge-input.test.ts @@ -1,61 +1,15 @@ -/** Free regression coverage for the file bundle sent to workflow judges. */ +/** Free regression coverage for the skill file bundle readWorkflowJudgeInput builds. */ import { afterEach, describe, expect, test } from 'bun:test'; import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { dirname, join, resolve } from 'node:path'; -import { createHash } from 'node:crypto'; -import { readWorkflowJudgeInput, buildWorkflowJudgePrompt, QA_DISCOVERY_REFERENCES, WORKFLOW_JUDGE_RESPONSE_SCHEMA, WORKFLOW_JUDGE_REASONING_WORD_LIMIT } from './helpers/workflow-judge-input'; +import { readWorkflowJudgeInput } from './helpers/workflow-judge-input'; import { ASK_QUESTIONS_HEADING, ENG_REVIEW_EXCERPT } from './helpers/workflow-excerpt'; import { readSkillStartup, SHARED_PREAMBLE_FILE } from './helpers/skill-startup'; const ROOT = resolve(import.meta.dir, '..'); const scratchRoots: string[] = []; -test('cache extraction preserves every byte of the original workflow request and rubric', () => { - // Captured from runWorkflowJudge's pre-cache template literal, not from the - // new builder: moving request construction must not change the paid metric. - const prompt = buildWorkflowJudgePrompt({ judgeContext: 'test workflow', judgeGoal: 'test goal' }, - { files: [], text: 'full prompt\nincluding lines' }); - expect(createHash('sha256').update(prompt).digest('hex')).toBe('71cc9c777bf28ff0efd610259b411e3539852f83a0888fa0e92469331f8b9a43'); -}); - -test.each(['ship', 'review'])('%s clarity targets frontier readers without excusing missing decisions or authority', skill => { - const source = readFileSync(join(ROOT, 'test/skill-llm-eval.test.ts'), 'utf8'); - const registration = source.match(new RegExp(`testIfSelected\\('${skill}/SKILL\\.md workflow',[\\s\\S]*?await runWorkflowJudge\\(\\{([\\s\\S]*?)\\n \\}\\);`)); - expect(registration).not.toBeNull(); - const options = new Function('QA_DISCOVERY_REFERENCES', `return ({${registration![1]}});`)(QA_DISCOVERY_REFERENCES); - expect(options.agentCapability).toBe('frontier'); - expect(options.thresholds).toBeUndefined(); - const input = readWorkflowJudgeInput({ root: ROOT, ...options }); - const prompt = buildWorkflowJudgePrompt(options, input); - expect(prompt).toContain('GPT-5.6 Sol-level capability or stronger'); - expect(prompt).toContain('Length, technical vocabulary and multiple explicit recovery paths alone are not clarity defects'); - expect(prompt).toContain('Do not invent missing policies, permissions or evidence'); - expect(prompt).toContain('Clarity 4 means the target agent can determine the next permitted action on each applicable path'); - expect(prompt).toContain('Score clarity 3 or lower when execution still requires guessing'); - expect(prompt).toContain('conflicting order, undefined decisions, unclear authority or missing input/output handling'); - expect(prompt).toContain('cite the specific file/step and explain the competing actions or missing decision'); - expect(prompt.endsWith(input.text)).toBe(true); - expect(prompt).toContain('"clarity": N, "completeness": N, "actionability": N, "reasoning": "brief explanation"'); -}); - -test('frontier calibration bounds reporting without reducing the evaluated source bundle', () => { - const input = { files: [], text: 'Entire source bundle remains present.' }; - const prompt = buildWorkflowJudgePrompt({ judgeContext: 'a workflow', judgeGoal: 'how to finish', agentCapability: 'frontier' }, input); - expect(prompt).toContain('Evaluate the whole workflow, but keep the JSON reasoning under 120 words with at most two decisive examples'); - expect(prompt).toContain('For a clarity defect, cite the specific file/step and explain the competing actions or missing decision'); - expect(prompt).not.toContain('For each clarity defect'); - expect(prompt.endsWith(input.text)).toBe(true); -}); - -test('judges are asked for 120 words while the enforced reasoning limit stays below 150', () => { - const prompt = buildWorkflowJudgePrompt({ judgeContext: 'a workflow', judgeGoal: 'how to finish', agentCapability: 'frontier' }, { files: [], text: '' }); - expect(prompt).toContain('keep the JSON reasoning under 120 words'); - expect(prompt).not.toContain('150 words'); - expect(WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning.description).toStartWith('Under 120 words'); - expect(WORKFLOW_JUDGE_REASONING_WORD_LIMIT).toBe(150); -}); - afterEach(() => { for (const root of scratchRoots.splice(0)) rmSync(root, { recursive: true, force: true }); }); @@ -74,7 +28,7 @@ function occurrences(text: string, needle: string): number { return text.split(needle).length - 1; } -/** The judge's bundle: every generated section except runner-only profiles (manifest `runner_only`, plan B4). */ +/** The bundle: every generated section except runner-only profiles (manifest `runner_only`, plan B4). */ function sectionPaths(skill: string): string[] { const runnerOnly = new Set((JSON.parse(readFileSync(join(ROOT, skill, 'sections', 'manifest.json'), 'utf8')).sections as Array<{ file: string; runner_only?: boolean }>) .filter(section => section.runner_only).map(section => section.file)); @@ -84,12 +38,9 @@ function sectionPaths(skill: string): string[] { .map(name => `${skill}/sections/${name}`); } -describe('workflow judge file bundle', () => { - test('the actual Eng judge registration includes authorization and the referenced final gate', () => { - const source = readFileSync(join(ROOT, 'test/skill-llm-eval.test.ts'), 'utf8'); - const registration = source.match(/testIfSelected\('plan-eng-review\/SKILL\.md sections',[\s\S]*?await runWorkflowJudge\(\{([\s\S]*?)\n \}\);/); - expect(registration).not.toBeNull(); - const options = new Function('ENG_REVIEW_EXCERPT', `return ({${registration![1]}});`)(ENG_REVIEW_EXCERPT); +describe('skill file bundle', () => { + test('the Eng bundle includes authorization and the referenced final gate', () => { + const options = { skillPath: ENG_REVIEW_EXCERPT.skillPath, startMarker: '# Plan Review Mode', endMarker: null }; const input = readWorkflowJudgeInput({ root: ROOT, ...options }); const entry = input.files.find(file => file.kind === 'entrypoint')!; const complete = readSkillStartup(join(ROOT, options.skillPath)); @@ -168,38 +119,6 @@ describe('workflow judge file bundle', () => { expect(() => readWorkflowJudgeInput({ ...options, references: ['../outside.md'] })).toThrow('Reference outside root'); }); - for (const skill of ['ship', 'qa-only', 'document-release', 'review']) { - test(`actual ${skill} judge includes all referenced QA or documentation sections`, () => { - const caller = readFileSync(join(ROOT, 'test/skill-llm-eval.test.ts'), 'utf8'); - const name = `${skill}/SKILL.md workflow`; - const start = caller.indexOf(`testIfSelected('${name}'`); - expect(start).toBeGreaterThanOrEqual(0); - const registration = caller.slice(start).match(/await runWorkflowJudge\(\{([\s\S]*?)\n \}\);/); - expect(registration).not.toBeNull(); - const options = new Function('QA_DISCOVERY_REFERENCES', `return ({${registration![1]}});`)(QA_DISCOVERY_REFERENCES); - const input = readWorkflowJudgeInput({ root: ROOT, ...options }); - for (const file of options.references ?? []) { - expect(input.files.find(item => item.path === file)?.content).toBe(readFileSync(join(ROOT, file), 'utf8')); - } - if (skill === 'ship') { - expect(input.files.map(file => file.path)).toContain('ship/sections/documentation.md'); - expect(input.files.map(file => file.path)).toContain('qa/sections/exploratory.md'); - } else if (skill === 'qa-only') { - expect(input.files.map(file => file.path)).toContain('qa/sections/system-functional.md'); - expect(input.files.filter(file => file.path.endsWith('/exploratory.md'))).toHaveLength(1); - expect(input.files.find(file => file.kind === 'entrypoint')?.content).toContain('Never fix bugs or write product tests'); - } else if (skill === 'document-release') { - expect(input.files.map(file => file.path)).toContain('document-release/sections/audit-scope.md'); - expect(input.text).toContain('Ship-owned documentation mode'); - } else { - expect(input.files.map(file => file.path)).toContain('qa/sections/exploratory.md'); - expect(input.files.map(file => file.path)).toContain('review/checklist.md'); - expect(input.text).toContain('test_stub'); - expect(input.text).toContain('## Step 5: Fix-First Review'); - } - }); - } - test('retains section prelude and suffix exactly once when both markers are inside a section', () => { // Generated comments made the old 120-character prefix heuristic append // this whole file after its partial slice, duplicating every review pass. @@ -280,13 +199,10 @@ describe('workflow judge file bundle', () => { test('generated ship includes base-branch initialization and every lazy section once', () => { const skillPath = 'ship/SKILL.md'; const source = readFileSync(join(ROOT, skillPath), 'utf8'); - // Bind to the paid caller's actual slice and retain both the opening contract - // and initialization, regardless of their ordering in the authored workflow. - const caller = readFileSync(join(ROOT, 'test/skill-llm-eval.test.ts'), 'utf8'); - const markers = caller.match(/skillPath: 'ship\/SKILL\.md',\s+startMarker: '([^']+)',\s+endMarker: '([^']+)'/); - expect(markers).not.toBeNull(); - const [, startMarker, endMarker] = markers!; - expect(startMarker).toBe('# Ship:'); + // Retain both the opening contract and initialization, regardless of their + // ordering in the authored workflow. + const startMarker = '# Ship:'; + const endMarker = '## Important Rules'; const input = readWorkflowJudgeInput({ root: ROOT, skillPath, startMarker, endMarker }); const entrypoint = input.files.find(file => file.kind === 'entrypoint'); expect(entrypoint?.content).toBe(source.slice(source.indexOf(startMarker), source.indexOf(endMarker, source.indexOf(startMarker)))); @@ -317,11 +233,8 @@ describe('workflow judge file bundle', () => { test('generated engineering review includes the scope choices and readiness probe its steps reference', () => { const skillPath = 'plan-eng-review/SKILL.md'; - const caller = readFileSync(join(ROOT, 'test/skill-llm-eval.test.ts'), 'utf8'); - const registration = caller.match(/testIfSelected\('plan-eng-review\/SKILL\.md sections',[\s\S]*?await runWorkflowJudge\(\{([\s\S]*?)\n \}\);/); - expect(registration).not.toBeNull(); - const options = new Function('ENG_REVIEW_EXCERPT', `return ({${registration![1]}});`)(ENG_REVIEW_EXCERPT); expect(ENG_REVIEW_EXCERPT.skillPath).toBe(skillPath); + const options = { skillPath, startMarker: '# Plan Review Mode', endMarker: null }; const input = readWorkflowJudgeInput({ root: ROOT, ...options }); const entrypoint = input.files.find(file => file.kind === 'entrypoint')!; expect(entrypoint.content).toContain('B) A plan or design doc'); @@ -346,10 +259,8 @@ describe('workflow judge file bundle', () => { test('generated design consultation includes the prechecks its proposal and preview reference', () => { const skillPath = 'design-consultation/SKILL.md'; - const caller = readFileSync(join(ROOT, 'test/skill-llm-eval.test.ts'), 'utf8'); - const markers = caller.match(/skillPath: 'design-consultation\/SKILL\.md',\s+startMarker: '([^']+)',\s+endMarker: '([^']+)'/); - expect(markers).not.toBeNull(); - const [, startMarker, endMarker] = markers!; + const startMarker = '## Phase 0:'; + const endMarker = '## Phase 4:'; const input = readWorkflowJudgeInput({ root: ROOT, skillPath, startMarker, endMarker }); const entrypoint = input.files.find(file => file.kind === 'entrypoint')!; for (const prerequisite of ['## Phase 0: Pre-checks', 'DESIGN_MD_FORMAT:', 'DESIGN_READY', 'DESIGN_NOT_AVAILABLE']) {