test: PTY harness handles clipped reviews and bundled setup tabs; AUQ judge uses structured output; design-consultation carve declines optional outside voices

- ceo mode routing: a Submit review taller than the viewport, a setup tab
  bundled after the mode tab, and a clip through the mode question each hung
  or misread the run; the native answer is still verified after Submit.
- judgeRecommendation requests a 1-5 enum schema; a malformed Haiku reply had
  scored substance 0 for a 4/5 brief. Judge failures now propagate.
- carve section-loading for design-consultation declines the optional outside
  voices (a supported path) and treats DESIGN.md as the report; timeout unchanged.
The Step 0E handoff defect is not fixed (0/15 samples across four wordings,
none shipped) and is filed in TODOS.
This commit is contained in:
garrytan committed 2026-09-30 12:52:56 +00:00
1 parent dfe5e733fb
commit a27365bffe
16 files changed
+893 -28

No files matched your search

+11 -1
View File
@@ -397,6 +397,16 @@ ${text}`, undefined, { signal });
* Format spec: scripts/resolvers/preamble/generate-ask-user-format.ts
* Recommendation: <choice> because <one-line reason>
*/
export const RECOMMENDATION_JUDGE_SCHEMA = {
type: 'object',
properties: {
reason_substance: { type: 'integer', enum: [1, 2, 3, 4, 5] },
reasoning: { type: 'string' },
},
required: ['reason_substance', 'reasoning'],
additionalProperties: false,
};
export async function judgeRecommendation(askUserText: string, signal?: AbortSignal): Promise<RecommendationScore> {
signal?.throwIfAborted();
// Deterministic checks. The format spec requires:
@@ -472,7 +482,7 @@ Respond with ONLY valid JSON:
const out = await callJudge<{ reason_substance: number; reasoning: string }>(
prompt,
'claude-haiku-4-5-20251001',
{ signal },
{ signal, jsonSchema: RECOMMENDATION_JUDGE_SCHEMA },
);
// Defensive clamp: rubric is 1-5. If Haiku returns out-of-range or non-numeric,