test: PTY harness handles clipped reviews and bundled setup tabs; AUQ judge uses structured output; design-consultation carve declines optional outside voices

- ceo mode routing: a Submit review taller than the viewport, a setup tab
  bundled after the mode tab, and a clip through the mode question each hung
  or misread the run; the native answer is still verified after Submit.
- judgeRecommendation requests a 1-5 enum schema; a malformed Haiku reply had
  scored substance 0 for a 4/5 brief. Judge failures now propagate.
- carve section-loading for design-consultation declines the optional outside
  voices (a supported path) and treats DESIGN.md as the report; timeout unchanged.
The Step 0E handoff defect is not fixed (0/15 samples across four wordings,
none shipped) and is filed in TODOS.
This commit is contained in:
garrytan committed 2026-09-30 12:52:56 +00:00
1 parent dfe5e733fb
commit a27365bffe
16 files changed
+893 -28

No files matched your search

+7 -11
View File
@@ -13,17 +13,13 @@
2.1.251 in every recent run) and the HOLD SCOPE
routing case when its next brief happens not to name the mode (see the
handoff item below). Effort M each.
- **`/plan-ceo-review` skips its Step 0E mode handoff** — in 4 of 4 asked-mode
samples (census 36633323521 plus three local runs) the model went straight
to tools after the mode answer without the required `Mode: <mode>; approved
decisions: …` chat. The routing case still passes on other posture text; a
wording change moving the handoff ahead of the question log did not change
the behavior in two paid runs, so it was not shipped. Effort M.
- **`/ship` design-lite probe wording** — `scripts/resolvers/design.ts` still
says "Probe for a design detector the user installed", the wording that let
`/review` skip its probe in 5 of 6 captured trials before this release made it
mandatory. The ship union ratio is at 1.3966 of 1.397, so the same sentence
needs a trim elsewhere first. Effort S.
- **`/plan-ceo-review` skips its Step 0E mode handoff** — 0 of 15 answered
samples sent the required `Mode: <mode>; approved decisions: …` chat after
the mode answer, across four wording repairs (none shipped). The model writes
the handoff in its reasoning and later says it was "sent above". A prose fix
won't reach it; this needs a mechanism outside the prompt (a hook or a
tool-result gate). The HOLD SCOPE routing case fails whenever the handoff is
skipped and nothing else names the posture in time. Effort M.
- **Let pass-rate history decide the rest** — every census on this branch had
a different handful of single-trial reds. Once `eval:pass-rates` has 10 weekly
trials per case, apply the CASE_QUARANTINE entry rule instead of chasing one