mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-06 00:44:07 +01:00
fix(evals): plan-design-with-ui focus detector accepts Opus 5.5's 'Review focus' title with the step's rating and an all-7 option; replay run 39298269902
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
e56e3aa8e2
commit
34de9fb073
6 files changed
+264
-5
No files matched your search
@@ -14,6 +14,8 @@
|
||||
### Changed
|
||||
- **Gate integrity (train block only):** under a `train` block, PR pushes no longer start paid slices; trains, screens and the backstop gate instead. Train and bisect runs never reuse a verdict and never bind one for reuse. No threshold, trial count, case kind or selection changed.
|
||||
- **Gate integrity (always, GSTA-48 P3-1):** when the readiness job blocks a PR head, a new `readiness-comment` job replaces an existing `## E2E Evals` comment with "NOT RUN on <head>", so an earlier PASS no longer stands beside a head that never ran. It posts nothing when no such comment exists. A slice killed before its upload step counts as incomplete.
|
||||
- **Gate integrity, widening (`plan-design-with-ui-scope` focus detector, GStack Minder-approved):** the Step 0D focus question also counts when its title names the focus (the title ends at the first line break or before `Project/branch/task:`), its body carries the step's own rating sentence ("I('ve) rated this plan N/10 on design completeness"), and it offers at least two options, one of which starts "All 7" or "All seven" (design or review) dimensions or passes. Opus 5.5 titled the question only "Review focus" in both attempts of run 39298269902, so the detector missed it. In the second attempt the finding detector also counted the focus question as the review finding, because its one-line title names hierarchy and panels. A recognized focus question can no longer count as a finding. Free replays of both captured attempts (`test/fixtures/design-ui-focus-capture-39298269902-a1.json` and `-a2.json`, registered in the case's touchfiles) pass. Every existing negative control still fails, as do these new ones: an issue-labeled finding that carries the rating, a single-option focus question, a focus question without a leading all-7 option, and a 4/7 rating. No threshold, trial count, case kind or ceiling changed.
|
||||
|
||||
## [1.91.149.0] - 2026-10-19
|
||||
|
||||
**Paid eval cases can now run on the Codex host, and the Monday census runs two arms: Claude Opus 5.5 and Codex gpt-6.1-sol.** `EVALS_HOST=codex` runs the 52 adapter-only gate cases through a full Codex runtime per case. On its first pilot gpt-6.1-sol passed 48 of 52 with no retries, at $4.63 for the 52 cases, against $28.93 for the same cases on Fable (0.16×). Claude-only cases are skipped on Codex with a recorded reason and never count as failed. GSTA-49 TM2.
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
{
|
||||
"provenance": {
|
||||
"kind": "public native AskUserQuestion calls and answers only (observation.json fingerprints[].nativeCall, byte-exact)",
|
||||
"runId": 39298269902,
|
||||
"artifact": "paid-slice-4-a1",
|
||||
"headSha": "072d4b95e5f3b9cf329fbd91fa97e339388f5fe6",
|
||||
"recordedOutcome": "ceiling_reached",
|
||||
"recordedReviewCount": 1,
|
||||
"recordedStep0Count": 4,
|
||||
"recordedPassed": false,
|
||||
"note": "The Step 0D question was titled 'Review focus for the User Dashboard plan' (header Focus) and carried 'I rated this plan 4/10 on design completeness' in its body, with options 'All 7 dimensions (recommended)' and 'Specific areas only'. The runner reached the review ceiling on the issue-labeled D5 finding, but the focus predicate required 'all 7' in the title, so the gate failed."
|
||||
},
|
||||
"calls": [
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 41013,
|
||||
"call": {
|
||||
"sessionId": "8baee8e7-0997-47ec-9eb3-5e4487a80424",
|
||||
"toolUseId": "toolu_01JVwSk6LQqTFjxJegSV8r5v",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D1 — Add gstack skill routing rules to this project's CLAUDE.md?\nProject/branch/task: main branch, reviewing the User Dashboard Page plan.\nELI10: gstack can add a short \"Skill routing\" section to CLAUDE.md so future sessions automatically pick the right skill (e.g. design review, QA). Without it, you invoke skills by name yourself.\nStakes if we pick wrong: Either an unwanted CLAUDE.md edit + commit, or future sessions don't auto-route to skills.\nRecommendation: A because it is one-time and makes skill selection automatic.\nNote: options differ in kind, not coverage — no completeness score.\nNote: plan mode is active, so if you pick A the CLAUDE.md edit + commit is deferred until after plan mode ends; nothing is written now.\nPros / cons:\nA) Add routing rules (recommended)\n ✅ Future sessions route requests to matching skills without you naming them\n ✅ One-time setup, a small appended section at the end of CLAUDE.md\n ❌ Edits and commits CLAUDE.md, which in this repo is a fixture file\nB) No thanks\n ✅ CLAUDE.md stays exactly as it is, no commit is created\n ✅ You keep full manual control over which skill runs\n ❌ You must invoke skills by name every time going forward\nNet: automatic routing versus leaving CLAUDE.md untouched.",
|
||||
"header": "Routing",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Add routing rules (recommended)",
|
||||
"description": "Append the Skill routing section to CLAUDE.md and commit (deferred until plan mode ends)."
|
||||
},
|
||||
{
|
||||
"label": "No thanks",
|
||||
"description": "Set routing_declined; invoke skills manually."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D1 — Add gstack skill routing rules to this project's CLAUDE.md?\nProject/branch/task: main branch, reviewing the User Dashboard Page plan.\nELI10: gstack can add a short \"Skill routing\" section to CLAUDE.md so future sessions automatically pick the right skill (e.g. design review, QA). Without it, you invoke skills by name yourself.\nStakes if we pick wrong: Either an unwanted CLAUDE.md edit + commit, or future sessions don't auto-route to skills.\nRecommendation: A because it is one-time and makes skill selection automatic.\nNote: options differ in kind, not coverage — no completeness score.\nNote: plan mode is active, so if you pick A the CLAUDE.md edit + commit is deferred until after plan mode ends; nothing is written now.\nPros / cons:\nA) Add routing rules (recommended)\n ✅ Future sessions route requests to matching skills without you naming them\n ✅ One-time setup, a small appended section at the end of CLAUDE.md\n ❌ Edits and commits CLAUDE.md, which in this repo is a fixture file\nB) No thanks\n ✅ CLAUDE.md stays exactly as it is, no commit is created\n ✅ You keep full manual control over which skill runs\n ❌ You must invoke skills by name every time going forward\nNet: automatic routing versus leaving CLAUDE.md untouched.": "Add routing rules (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-19T18:09:26.151Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 88160,
|
||||
"call": {
|
||||
"sessionId": "8baee8e7-0997-47ec-9eb3-5e4487a80424",
|
||||
"toolUseId": "toolu_01WMTWHwmpvAHVSW4BS2UKZb",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D2 — Review focus for the User Dashboard plan\nProject/branch/task: main, design review of the User Dashboard Page plan (review-input.md).\nELI10: I rated this plan 4/10 on design completeness. The biggest gaps are page hierarchy per breakpoint, what each activity/notification row actually shows, and the real copy/behavior for empty, error, modal and toast states. Next I generate visual mockups, then walk all 7 design dimensions. You can narrow that.\nStakes if we pick wrong: Narrowing skips dimensions, so an unreviewed gap (e.g. accessibility or responsive) ships as whatever the implementer guesses.\nRecommendation: A because the plan has gaps in nearly every dimension, so a narrow pass would leave known holes.\nCompleteness: A=10/10, B=6/10\nPros / cons:\nA) All 7 dimensions (recommended)\n ✅ Every dimension gets rated and every gap gets its own decision\n ✅ Hierarchy, states, a11y and responsive all land in the plan before code\n ❌ More questions to answer, one per gap (human: ~1 hr / CC: ~15 min)\nB) Focus on specific areas only\n ✅ Fewer questions, faster to a revised plan for the areas you care about\n ✅ Lets you skip dimensions you already have opinions on\n ❌ Unreviewed dimensions stay at their current low scores in the report\nNet: full coverage versus fewer questions.",
|
||||
"header": "Focus",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "All 7 dimensions (recommended)",
|
||||
"description": "Mockups first, then full 7-pass review with per-gap decisions."
|
||||
},
|
||||
{
|
||||
"label": "Specific areas only",
|
||||
"description": "Tell me which dimensions (e.g. hierarchy, states, a11y) to focus on."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D2 — Review focus for the User Dashboard plan\nProject/branch/task: main, design review of the User Dashboard Page plan (review-input.md).\nELI10: I rated this plan 4/10 on design completeness. The biggest gaps are page hierarchy per breakpoint, what each activity/notification row actually shows, and the real copy/behavior for empty, error, modal and toast states. Next I generate visual mockups, then walk all 7 design dimensions. You can narrow that.\nStakes if we pick wrong: Narrowing skips dimensions, so an unreviewed gap (e.g. accessibility or responsive) ships as whatever the implementer guesses.\nRecommendation: A because the plan has gaps in nearly every dimension, so a narrow pass would leave known holes.\nCompleteness: A=10/10, B=6/10\nPros / cons:\nA) All 7 dimensions (recommended)\n ✅ Every dimension gets rated and every gap gets its own decision\n ✅ Hierarchy, states, a11y and responsive all land in the plan before code\n ❌ More questions to answer, one per gap (human: ~1 hr / CC: ~15 min)\nB) Focus on specific areas only\n ✅ Fewer questions, faster to a revised plan for the areas you care about\n ✅ Lets you skip dimensions you already have opinions on\n ❌ Unreviewed dimensions stay at their current low scores in the report\nNet: full coverage versus fewer questions.": "All 7 dimensions (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-19T18:10:17.876Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 123730,
|
||||
"call": {
|
||||
"sessionId": "8baee8e7-0997-47ec-9eb3-5e4487a80424",
|
||||
"toolUseId": "toolu_01PDpzEdw9gtfax268HSxc9a",
|
||||
"questions": [
|
||||
{
|
||||
"question": "Want outside design voices before the detailed review?\nD3 — Outside design voices\nProject/branch/task: main, User Dashboard Page plan design review.\nELI10: Before the 7 passes, I can get second opinions: Codex scores the plan against hard design rules and litmus checks, and a separate Claude agent does an independent completeness read. It costs a few minutes and gives a cross-check.\nStakes if we pick wrong: Skipping means one reviewer's judgment; running costs time and Codex may be unavailable.\nRecommendation: B because the plan's gaps are already concrete and a single native pass covers them; this is a taste call.\nNote: options differ in kind, not coverage — no completeness score.\nNet: a cross-model check versus a faster path to the fixed plan.",
|
||||
"header": "Voices",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Yes, run outside design voices",
|
||||
"description": "✅ Codex hard-rule + litmus scorecard plus an independent Claude completeness read. ✅ Disagreements surface with both perspectives in the relevant pass. ❌ Adds a few minutes; Codex may be missing here, in which case only the Claude voice runs and it's tagged single-model."
|
||||
},
|
||||
{
|
||||
"label": "No, proceed without (recommended)",
|
||||
"description": "✅ Straight into the 7 passes with the approved mockup as anchor. ✅ Fewest steps to a fixed plan. ❌ Findings rest on one reviewer; no cross-model check on slop risk or hierarchy."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"Want outside design voices before the detailed review?\nD3 — Outside design voices\nProject/branch/task: main, User Dashboard Page plan design review.\nELI10: Before the 7 passes, I can get second opinions: Codex scores the plan against hard design rules and litmus checks, and a separate Claude agent does an independent completeness read. It costs a few minutes and gives a cross-check.\nStakes if we pick wrong: Skipping means one reviewer's judgment; running costs time and Codex may be unavailable.\nRecommendation: B because the plan's gaps are already concrete and a single native pass covers them; this is a taste call.\nNote: options differ in kind, not coverage — no completeness score.\nNet: a cross-model check versus a faster path to the fixed plan.": "No, proceed without (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-19T18:10:53.393Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 140130,
|
||||
"call": {
|
||||
"sessionId": "8baee8e7-0997-47ec-9eb3-5e4487a80424",
|
||||
"toolUseId": "toolu_0116C69RgYwN7PJ2H7FpKdqq",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D4 — Search learnings across your other projects?\nProject/branch/task: main, User Dashboard plan design review.\nELI10: gstack keeps local notes (\"learnings\") from past sessions. It can search notes from your other projects on this machine too, which helps when patterns repeat. Nothing leaves your machine.\nStakes if we pick wrong: Cross-project on can mix in patterns from unrelated client codebases; off misses reusable lessons.\nRecommendation: A because for a solo developer reusable lessons outweigh cross-contamination risk.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\nA) Enable cross-project learnings (recommended)\n ✅ Past lessons from other repos on this machine can inform this review\n ✅ Stays fully local, one-time setting stored in gstack config\n ❌ Patterns from unrelated client projects could leak into recommendations\nB) Project-scoped only\n ✅ Each project's learnings stay isolated, no cross-client contamination\n ✅ Simpler mental model of where advice comes from\n ❌ Misses lessons already learned elsewhere on this machine\nNet: broader recall versus strict per-project isolation.",
|
||||
"header": "Learnings",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Enable cross-project (recommended)",
|
||||
"description": "Set cross_project_learnings=true and search across local projects."
|
||||
},
|
||||
{
|
||||
"label": "Project-scoped only",
|
||||
"description": "Set cross_project_learnings=false."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D4 — Search learnings across your other projects?\nProject/branch/task: main, User Dashboard plan design review.\nELI10: gstack keeps local notes (\"learnings\") from past sessions. It can search notes from your other projects on this machine too, which helps when patterns repeat. Nothing leaves your machine.\nStakes if we pick wrong: Cross-project on can mix in patterns from unrelated client codebases; off misses reusable lessons.\nRecommendation: A because for a solo developer reusable lessons outweigh cross-contamination risk.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\nA) Enable cross-project learnings (recommended)\n ✅ Past lessons from other repos on this machine can inform this review\n ✅ Stays fully local, one-time setting stored in gstack config\n ❌ Patterns from unrelated client projects could leak into recommendations\nB) Project-scoped only\n ✅ Each project's learnings stay isolated, no cross-client contamination\n ✅ Simpler mental model of where advice comes from\n ❌ Misses lessons already learned elsewhere on this machine\nNet: broader recall versus strict per-project isolation.": "Enable cross-project (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-19T18:11:10.623Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": false,
|
||||
"observedAtMs": 168630,
|
||||
"call": {
|
||||
"sessionId": "8baee8e7-0997-47ec-9eb3-5e4487a80424",
|
||||
"toolUseId": "toolu_01HhdTUnvoyntgagQCG8UjBH",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D5 — Issue 1: Which content leads the dashboard, and how do panels lay out?\nProject/branch/task: main, User Dashboard plan, Pass 1 Information Architecture.\nELI10: The plan names three panels but never says which one the user sees first. Without that, the engineer picks an order (usually the order in the spec) and every panel gets equal weight, so nothing stands out. Unread notifications are the only thing here that asks the user to do something, so they should win.\nStakes if we pick wrong: Users landing after login scan an undifferentiated grid and miss unread items, or the page is rebuilt after launch.\nRecommendation: 1A because notifications are the only actionable content and \"hierarchy as service\" says the actionable thing goes first.\nCompleteness: 1A=9/10, 1B=8/10, 1C=5/10\nPros / cons:\n1A) Notifications primary, Activity secondary (recommended)\n ✅ Unread items are the first thing seen on every viewport, matching the user's real job\n ✅ lg: two columns (Notifications 7/12 left, Activity 5/12 right); base/sm/md: single column Notifications then Activity\n ❌ Activity gets less width on desktop, so long activity descriptions truncate sooner\n1B) Single column at all widths, Notifications then Activity\n ✅ Simplest layout, one reading path, easiest to make accessible and responsive\n ✅ Same priority order on every device, nothing to re-learn between phone and desktop\n ❌ Wastes horizontal space on lg screens; Activity sits below the fold on desktop\n1C) Leave the order to the implementer\n ✅ No spec time now, implementer has freedom (human: 0 / CC: 0)\n ✅ Can be tuned later once real data exists\n ❌ Most likely result is three equal-weight panels with no clear first read\nNet: a ranked, actionable-first layout versus simplicity or deferral.",
|
||||
"header": "Hierarchy",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "1A Notifications first (recommended)",
|
||||
"description": "lg: 2 columns, Notifications 7/12 + Activity 5/12; below lg: single column Notifications → Activity.",
|
||||
"preview": "lg (≥1024px)\n+----------------------------------------------+\n| App header: logo | Dashboard* Workspace | ava|\n+----------------------------------------------+\n| Dashboard |\n+---------------------------+------------------+\n| Notifications 3 unread | Recent activity |\n| [Mark all as read] | row |\n| ● row | row |\n| ● row | row |\n| row | View all → |\n+---------------------------+------------------+\n\nbase/sm/md\n+------------------+\n| Dashboard |\n| Notifications |\n| rows... |\n| Recent activity |\n| rows... |\n+------------------+"
|
||||
},
|
||||
{
|
||||
"label": "1B Single column always",
|
||||
"description": "Notifications then Activity, one column at every width (max-w ~720px centered).",
|
||||
"preview": "all widths\n+--------------------------+\n| Dashboard |\n| Notifications 3 unread |\n| rows... |\n| Recent activity |\n| rows... |\n+--------------------------+"
|
||||
},
|
||||
{
|
||||
"label": "1C Leave to implementer",
|
||||
"description": "No hierarchy spec; implementer decides order and layout."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D5 — Issue 1: Which content leads the dashboard, and how do panels lay out?\nProject/branch/task: main, User Dashboard plan, Pass 1 Information Architecture.\nELI10: The plan names three panels but never says which one the user sees first. Without that, the engineer picks an order (usually the order in the spec) and every panel gets equal weight, so nothing stands out. Unread notifications are the only thing here that asks the user to do something, so they should win.\nStakes if we pick wrong: Users landing after login scan an undifferentiated grid and miss unread items, or the page is rebuilt after launch.\nRecommendation: 1A because notifications are the only actionable content and \"hierarchy as service\" says the actionable thing goes first.\nCompleteness: 1A=9/10, 1B=8/10, 1C=5/10\nPros / cons:\n1A) Notifications primary, Activity secondary (recommended)\n ✅ Unread items are the first thing seen on every viewport, matching the user's real job\n ✅ lg: two columns (Notifications 7/12 left, Activity 5/12 right); base/sm/md: single column Notifications then Activity\n ❌ Activity gets less width on desktop, so long activity descriptions truncate sooner\n1B) Single column at all widths, Notifications then Activity\n ✅ Simplest layout, one reading path, easiest to make accessible and responsive\n ✅ Same priority order on every device, nothing to re-learn between phone and desktop\n ❌ Wastes horizontal space on lg screens; Activity sits below the fold on desktop\n1C) Leave the order to the implementer\n ✅ No spec time now, implementer has freedom (human: 0 / CC: 0)\n ✅ Can be tuned later once real data exists\n ❌ Most likely result is three equal-weight panels with no clear first read\nNet: a ranked, actionable-first layout versus simplicity or deferral.": "1A Notifications first (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-19T18:11:38.592Z"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"provenance": {
|
||||
"kind": "public native AskUserQuestion calls and answers only (observation.json fingerprints[].nativeCall, byte-exact)",
|
||||
"runId": 39298269902,
|
||||
"artifact": "paid-slice-4-a2",
|
||||
"headSha": "072d4b95e5f3b9cf329fbd91fa97e339388f5fe6",
|
||||
"recordedOutcome": "ceiling_reached",
|
||||
"recordedReviewCount": 1,
|
||||
"recordedStep0Count": 0,
|
||||
"recordedPassed": false,
|
||||
"note": "The Step 0D question arrived on one line, 'Review focus. Project/branch/task: ...', with 'I rated this plan 4/10 on design completeness' in the body and options 'All 7 passes (recommended)' and 'States + hierarchy only'. The focus predicate missed it, and the finding predicate counted it as the review question (its one-line title names hierarchy and panels), so the run stopped at the ceiling on the focus question itself."
|
||||
},
|
||||
"calls": [
|
||||
{
|
||||
"preReview": false,
|
||||
"observedAtMs": 63729,
|
||||
"call": {
|
||||
"sessionId": "84af3460-cb15-42a6-a835-364da70c0be4",
|
||||
"toolUseId": "toolu_01JaP4QULFteQDRNuaxtMvmz",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D1 — Review focus. Project/branch/task: User Dashboard plan on main. ELI10: I rated this plan 4/10 on design completeness. Biggest gaps: no visual hierarchy (which panel wins on first view), panel states named but never designed (empty copy, skeleton shape, error/retry layout), and no responsive layout per breakpoint. Mockups come next, then all 7 review dimensions unless you want a narrower focus. Stakes if we pick wrong: a narrow focus leaves accessibility or state gaps for the implementer to guess. Recommendation: A because the plan is early and every dimension has open gaps. Completeness: A=10/10, B=6/10. Pros / cons: A) All 7 passes (recommended) ✅ Covers hierarchy, states, journey, slop risk, specificity, responsive and accessibility in one sitting ✅ Each gap still gets its own decision, so you can skip any fix ❌ More questions to answer, roughly one per real gap found. B) Focus on states + hierarchy only ✅ Faster; targets the two gaps most likely to ship looking broken ✅ Fewer decisions to make today ❌ Responsive, modal/toast accessibility and copy stay unspecified for implementer. Net: thoroughness now versus fewer questions today.",
|
||||
"header": "Focus",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "All 7 passes (recommended)",
|
||||
"description": "✅ Full coverage of every design dimension, one decision per gap. ✅ Nothing left for the implementer to guess. ❌ More questions overall."
|
||||
},
|
||||
{
|
||||
"label": "States + hierarchy only",
|
||||
"description": "✅ Faster, targets the riskiest gaps. ✅ Fewer decisions. ❌ Responsive, a11y and copy stay unspecified."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D1 — Review focus. Project/branch/task: User Dashboard plan on main. ELI10: I rated this plan 4/10 on design completeness. Biggest gaps: no visual hierarchy (which panel wins on first view), panel states named but never designed (empty copy, skeleton shape, error/retry layout), and no responsive layout per breakpoint. Mockups come next, then all 7 review dimensions unless you want a narrower focus. Stakes if we pick wrong: a narrow focus leaves accessibility or state gaps for the implementer to guess. Recommendation: A because the plan is early and every dimension has open gaps. Completeness: A=10/10, B=6/10. Pros / cons: A) All 7 passes (recommended) ✅ Covers hierarchy, states, journey, slop risk, specificity, responsive and accessibility in one sitting ✅ Each gap still gets its own decision, so you can skip any fix ❌ More questions to answer, roughly one per real gap found. B) Focus on states + hierarchy only ✅ Faster; targets the two gaps most likely to ship looking broken ✅ Fewer decisions to make today ❌ Responsive, modal/toast accessibility and copy stay unspecified for implementer. Net: thoroughness now versus fewer questions today.": "All 7 passes (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-19T18:23:49.901Z"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -379,7 +379,7 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
|
||||
|
||||
|
||||
"test/fixtures/plan-scope-recovery-av.json",
|
||||
"test/fixtures/design-scope-checkpoint-at.json",'plan-design-review/**', 'test/fixtures/plans/ui-heavy-feature.md', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/fixtures/design-ui-boxed-question.json', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/pty-trust-dialog.ts', 'test/skill-e2e-plan-design-with-ui.test.ts', 'test/fixtures/eng-d2-truncated-border-0bcd.json', 'test/fixtures/eng-d1-clipped-elision-1579.json', 'test/fixtures/eng-d2-planning-prelude-4d.json', 'test/fixtures/ceo-approach-z-call.json', 'test/fixtures/ceo-approach-z-screen.txt',
|
||||
"test/fixtures/design-scope-checkpoint-at.json",'plan-design-review/**', 'test/fixtures/plans/ui-heavy-feature.md', 'test/fixtures/design-ui-focus-capture-39298269902-a1.json', 'test/fixtures/design-ui-focus-capture-39298269902-a2.json', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/fixtures/design-ui-boxed-question.json', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/pty-trust-dialog.ts', 'test/skill-e2e-plan-design-with-ui.test.ts', 'test/fixtures/eng-d2-truncated-border-0bcd.json', 'test/fixtures/eng-d1-clipped-elision-1579.json', 'test/fixtures/eng-d2-planning-prelude-4d.json', 'test/fixtures/ceo-approach-z-call.json', 'test/fixtures/ceo-approach-z-screen.txt',
|
||||
|
||||
|
||||
"scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts",
|
||||
|
||||
@@ -83,6 +83,10 @@ const captured = JSON.parse(fs.readFileSync(${JSON.stringify(path.join(ROOT, 'te
|
||||
// Run 39199692993 slice 4 (521fd755): the Step 0D title omitted 'design'; its body carried the rating.
|
||||
const capturedRated = JSON.parse(fs.readFileSync(${JSON.stringify(path.join(ROOT, 'test/fixtures/design-ui-focus-capture-39199692993.json'))}, 'utf8'))
|
||||
.calls.map(({ preReview, observedAtMs, call }) => nativePlanCallFingerprint(call, observedAtMs, preReview));
|
||||
// Run 39298269902 slice 4 (072d4b9), attempts 1 and 2: Opus 5.5 titled Step 0D only
|
||||
// "Review focus", with the rating in the body and an all-7 option (a2 on one line).
|
||||
const captured39298269902 = [1, 2].map(attempt => JSON.parse(fs.readFileSync(${JSON.stringify(ROOT)} + '/test/fixtures/design-ui-focus-capture-39298269902-a' + attempt + '.json', 'utf8'))
|
||||
.calls.map(({ preReview, observedAtMs, call }) => nativePlanCallFingerprint(call, observedAtMs, preReview)));
|
||||
const unnumberedFinding = fp('unnumbered-finding', "Which region is the dashboard's primary anchor, and how is the page composed?", false);
|
||||
unnumberedFinding.nativeCall.questions[0].options = nativeFinding.nativeCall.questions[0].options;
|
||||
const finding = fp('finding', 'Which loading feedback should Save show?', false);
|
||||
@@ -198,6 +202,22 @@ mock.module(${JSON.stringify(path.join(ROOT, 'test/helpers/claude-pty-runner.ts'
|
||||
['D14 — Record this follow-up?', ['1A) Record a TODO']],
|
||||
['D15 — Want outside design voices before the detailed review?', ['1A) Yes, run outside voices', '1B) No, proceed without']],
|
||||
]) expect(opts.isReviewAUQ(labeled('not-a-finding', question, labels))).toBe(false);
|
||||
const [focusA1, focusA2] = [captured39298269902[0][1], captured39298269902[1][0]];
|
||||
for (const focusCall of [focusA1, focusA2]) {
|
||||
expect(opts.isLastStep0AUQ(focusCall)).toBe(true);
|
||||
expect(opts.isReviewAUQ(focusCall)).toBe(false);
|
||||
}
|
||||
expect(opts.isReviewAUQ(captured39298269902[0][4])).toBe(true);
|
||||
const ratedBody = 'Project/branch/task: main. ELI10: I rated this plan 4/10 on design completeness.';
|
||||
const ratedFinding = labeled('rated-finding', 'D5 — Issue 1: Which panel leads the review focus? ' + ratedBody, ['1A All 7 dimensions first', '1B Activity first']);
|
||||
expect(opts.isLastStep0AUQ(ratedFinding)).toBe(false);
|
||||
expect(opts.isReviewAUQ(ratedFinding)).toBe(true);
|
||||
for (const [label, labels] of [['single-option-focus', ['All 7 dimensions (recommended)']], ['no-all-seven-option', ['Every dimension (recommended)', 'States + hierarchy only']],
|
||||
['all-seven-not-leading', ['Review all 7 dimensions (recommended)', 'States + hierarchy only']]]) {
|
||||
expect(opts.isLastStep0AUQ(labeled(label, 'D1 — Review focus. ' + ratedBody, labels))).toBe(false);
|
||||
}
|
||||
expect(opts.isLastStep0AUQ(labeled('unrated-focus', 'D1 — Review focus. Project/branch/task: main. ELI10: I rated this plan 4/7 on design completeness.',
|
||||
['All 7 passes (recommended)', 'States + hierarchy only']))).toBe(false);
|
||||
const chosenFocus = mode.startsWith('native-') ? nativeFocus : mode === 'paraphrase' ? paraphrase : fp('focus', focus);
|
||||
const pendingCall = {...chosenFocus.nativeCall, answered: false, unansweredQuestionIndices: [0]};
|
||||
const pending = nativePlanCallFingerprint(pendingCall, 1000, true);
|
||||
@@ -240,6 +260,9 @@ mock.module(${JSON.stringify(path.join(ROOT, 'test/helpers/claude-pty-runner.ts'
|
||||
: mode === 'captured-rated' ? capturedRated
|
||||
: mode === 'captured-rated-no-focus' ? capturedRated.filter((_, index) => index !== 1)
|
||||
: mode === 'captured-no-focus' ? captured.filter((_, index) => index !== 1)
|
||||
: mode === 'captured-39298269902' ? captured39298269902[0]
|
||||
: mode === 'captured-39298269902-no-focus' ? captured39298269902[0].filter((_, index) => index !== 1)
|
||||
: mode === 'captured-39298269902-focus-only' ? captured39298269902[1]
|
||||
: mode === 'captured-focus-after-finding' ? [...captured.filter((_, index) => index !== 1), captured[1]]
|
||||
: mode.startsWith('target-menu') ? [target, fp('other', 'Which artifact should I inspect?', false)]
|
||||
: mode === 'early-exit' ? [] : mode === 'focus-only' ? [chosenFocus]
|
||||
@@ -284,12 +307,12 @@ await import(${JSON.stringify(path.join(ROOT, 'test/skill-e2e-plan-design-with-u
|
||||
} finally { fs.rmSync(directory, { recursive: true, force: true }); }
|
||||
}
|
||||
|
||||
test.each(['source', 'paraphrase', 'native-sequence', 'native-unnumbered', 'native-passes', 'captured', 'captured-rated'])('UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (%s)', mode => {
|
||||
test.each(['source', 'paraphrase', 'native-sequence', 'native-unnumbered', 'native-passes', 'captured', 'captured-rated', 'captured-39298269902'])('UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (%s)', mode => {
|
||||
const result = exercise(mode);
|
||||
expect(result.code, result.output).toBe(0);
|
||||
}, 20_000);
|
||||
|
||||
test.each(['target-menu', 'target-menu-design-system', 'focus-only', 'captured-no-focus', 'captured-focus-after-finding', 'captured-rated-no-focus', 'native-setup-only', 'native-voices-only', 'early-exit', 'timeout', 'exited', 'unanswered-finding', 'failed-finding', 'no-native-finding'])('UI gate rejects %s and removes its fixture', mode => {
|
||||
test.each(['target-menu', 'target-menu-design-system', 'focus-only', 'captured-no-focus', 'captured-focus-after-finding', 'captured-rated-no-focus', 'captured-39298269902-no-focus', 'captured-39298269902-focus-only', 'native-setup-only', 'native-voices-only', 'early-exit', 'timeout', 'exited', 'unanswered-finding', 'failed-finding', 'no-native-finding'])('UI gate rejects %s and removes its fixture', mode => {
|
||||
const result = exercise(mode);
|
||||
expect(result.code, result.output).toBe(1);
|
||||
expect(result.output).toContain('plan-design-review with UI scope FAILED');
|
||||
|
||||
@@ -24,7 +24,7 @@ const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const FIXTURE = path.join(ROOT, 'test', 'fixtures', 'plans', 'ui-heavy-feature.md');
|
||||
|
||||
const designFocusBoundary = (fp: AskUserQuestionFingerprint): boolean =>
|
||||
fp.nativeCall?.answered === true && !fp.nativeCall.failed && fp.nativeCall.questions.some(({ question }) => {
|
||||
fp.nativeCall?.answered === true && !fp.nativeCall.failed && fp.nativeCall.questions.some(({ question, options }) => {
|
||||
const text = question.trim().replace(/^D\d+(?:\.\d+)?\s*[—–:-]\s*/i, '');
|
||||
// A target menu can mention a design system without reviewing this plan.
|
||||
// The source Step 0D question, or a native title (first line) that offers all
|
||||
@@ -33,9 +33,15 @@ const designFocusBoundary = (fp: AskUserQuestionFingerprint): boolean =>
|
||||
// body carries the step's own design-completeness rating (run 39199692993).
|
||||
const title = text.split(/\r?\n/, 1)[0]!;
|
||||
const rated = /\bI(?:['’]ve| have)? rated this plan (?:10(?:\.0+)?|[0-9](?:\.\d+)?)\/10 on design completeness\b/i;
|
||||
// Runs 39298269902 a1/a2: a title that only names the focus ("Review focus.",
|
||||
// on one line before Project/branch/task:), with the step's own rating in the
|
||||
// body and an option that offers all seven dimensions against a narrower one.
|
||||
const lead = text.split(/\r?\n|\s+(?=Project\/branch\/task:)/, 1)[0]!;
|
||||
return /^I(?:['’]ve| have) rated this plan (?:10(?:\.0+)?|[0-9](?:\.\d+)?)\/10 on design completeness\.[\s\S]*\bWant me to focus on specific areas instead(?: of all (?:7|seven))?\?/i.test(text)
|
||||
|| (/\ball (?:7|seven) (?:design )?(?:dimensions|passes)\b/i.test(title) && /\bfocus\b/i.test(title)
|
||||
&& (/\bdesign\b/i.test(title) || rated.test(text.slice(title.length))) && /\?\s*$/.test(title));
|
||||
&& (/\bdesign\b/i.test(title) || rated.test(text.slice(title.length))) && /\?\s*$/.test(title))
|
||||
|| (/\bfocus\b/i.test(lead) && rated.test(text) && options.length >= 2
|
||||
&& options.some(({ label }) => /^\s*all (?:7|seven) (?:design |review )?(?:dimensions|passes)\b/i.test(label)));
|
||||
});
|
||||
|
||||
// Require a choice about the supplied UI, not a workflow offer after focus.
|
||||
|
||||
Reference in new issue
Block a user