mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-13 09:40:21 +02:00
* fix(evals): align plan-eng/design plan-mode + finding-floor smokes to their declared periodic tier The #2077 demotion of these four stochastic tests to 'periodic' was inert: E2E_TIERS declared periodic but the files self-gated on EVALS_TIER === 'gate', so they kept running in the blocking gate lane and never in the weekly lane. Flip the four self-gates to 'periodic' (headers/describe labels updated), add a free static tier-alignment invariant test (dep-list filename mapping; unmapped self-gated files are reported, never silently skipped), and name the two plan-mode test files in their own touchfiles dep lists so the invariant binds for them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pty-runner): scope-gate question/auto-select detectors + observation flags Two render-shape-anchored detectors (whitespace-squished, like the Pattern-4/5 collapsed-form handling): isScopeGateQuestionVisible requires the question text PLUS option-body text (native AskUserQuestion renders numbered options, prose fallback renders lettered — the option body appears in both; narration doesn't), and isScopeGateAutoSelectVisible requires the announcement prefix PLUS the selected-B token. runPlanSkillObservation gains scopeGateQuestionObserved / scopeGateAutoSelectObserved high-water flags (attached at every return path) so paid smokes can assert gate behavior across the whole run instead of the lossy 2KB evidence tail. runPlanSkillFloorCheck no longer counts a scope-gate render toward auqObserved (tail-scoped exclusion) — the floor measures FINDING-driven questions, and the gate could fire inside the 3s pre-target window. Unit fixtures pin clean/native/collapsed positives, narration negatives, and the verbatim template announcement string (template rewording fails here first, before the paid smokes degrade to vacuous asserts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(plan-eng/design-review): auto-select B in plan mode at the scope gate In plan mode the scope gate's "What should I review? A/B/C" question is pure friction: there is no branch diff and the target is the plan being drafted. Both gates gain an ordered exceptions block, checked BEFORE asking: 1. Plan mode → auto-select B: review the active plan (in context or pasted), announce it in one line ("Scope gate: plan mode — auto-selected B (reviewing <target>)") so the user can interrupt; an explicitly different user-named target still wins; no plan drafted yet → ask as normal. 2. User-named target (outside plan mode): explicit-only — a path, a pasted doc, or the literal words "branch diff". A passing mention is not naming; when in doubt, ask. Outside plan mode with no explicitly-named target, nothing changes. Plan-mode is checked FIRST because the PTY harness seeds drafts as pasted user messages (claude-pty-runner.ts:1600) — ordering makes the seeded smokes deterministic. Pinning: seeded plan-mode smokes assert no gate render + announcement rendered (eng test 2; new design seeded test); plan-mode-no-op extends to eng/design (bypass must not misfire outside plan mode; first question must be the gate) plus a named-target case proving the pasted target is consumed; a drift-guard asserts the two hand-duplicated exceptions blocks stay identical modulo the two variant slots and carry the announcement string the detectors pin. Skeleton ceilings ratcheted with comments (eng 68k, design 89k; eng union ratio 1.08→1.09) — measured 67,006 B / 88,226 B after regen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(autoplan): skip the scope gate when following loaded review skills autoplan Step 3 reads plan-eng-review / plan-design-review SKILL.md verbatim, and its section skip list omitted the scope gate — so autoplan ingested a hard-STOP AskUserQuestion that contradicts its every-question-auto-decides contract. One skip-list line fixes it; a static toContain pin in skill-validation keeps the entry load-bearing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: file scope-gate resolver-extraction TODO (eng-review D5 follow-up) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pty-runner): positional floor exclusion, flag builder, outcome union, token tracking Review-army + adversarial findings on the scope-gate observability work, all verified before fixing: - Floor check: acceptance scanned the CUMULATIVE buffer while the scope-gate exclusion scanned only the 1500-byte tail, so an early gate render satisfied the floor vacuously once ~1.5KB of output accumulated (found independently by 4 review passes; predicate reproduced). Acceptance now scans only content APPENDED after the first gate render (positional anchor), and the LLM-judge 'waiting' shortcut no longer fires while the gate menu is the pending render. - High-water flags are built once and spread at every return path — the hand-spread pattern had already drifted (judge-waiting return omitted two flags), which made must-stay-false asserts vacuous on those paths. - isScopeGateAutoSelectVisible: tense-tolerant selected/selecting/selects token (must-be-TRUE asserts shouldn't fail semantically-perfect paraphrases) and quoted-occurrence rejection (a model verbatim-quoting the announcement while declining must not trip must-stay-FALSE asserts). Fixtures added for both directions. - PlanSkillObservation outcome union gains 'wrote_findings_before_asking' (returned at runtime via classifyVisible but missing from the type). - trackTokens/tokensObserved: cumulative-buffer token high-water for consumption asserts (the 2KB evidence tail is lossy and the plan-file fallback is unreachable outside plan mode). - New scope-gate-floor unit pins (from the ship coverage audit): both gate render forms trip acceptance and exclusion; a genuine finding AUQ is not excluded; tail-scoping semantics pinned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(evals): harden no-op asserts, close tier-invariant fail-open holes, pin gate question strings - no-op regression: gate-must-ask is now UNCONDITIONAL for eng/design (the outcome==='asked' conditional let a silent-bypass plan_ready run sail through); eng/design cases force --disallowedTools so the pinned prose shape is contractual rather than hoping native AUQ renders match; the named-target case uses trackTokens for consumption and lists wrote_findings_before_asking in its diagnostic throw branch. - tier-alignment invariant: both quote styles matched; zero-self-gate, mixed-tier, and owning-keys-without-E2E_TIERS-entries are all REPORTED instead of silently skipped (the fail-open holes three reviewers found). - drift-guard: the generated gate menus must carry the exact question/option strings the PTY question detector anchors on — free CI fails before the paid smokes can go vacuous on a menu reword. - touchfiles: corrected the no-op cost note for CI concurrency + retry semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): register plan-eng/design-review skills in PTY eval containers The extended plan-mode-no-op smoke invokes /plan-eng-review and /plan-design-review, but the fresh CI containers registered only office-hours and plan-ceo-review — both new runs would return 'Unknown command' and fail every PR's gate job (Codex structured review P1, verified against evals.yml). Registration loops, the dangling-target fail-fast list, and the frontmatter checks (now a loop over the same skill list, so the lists can't drift) all cover the two skills. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(plan-eng/design-review): harden scope-gate exceptions against injection and ambiguity Adversarial-review wording fixes (Claude adversarial F1-F8 + Codex cross-confirmation), applied to both gate templates + regen: - Host-anchored mode signal: only the host's own system messages (plan-mode reminder or active plan file path) arm the auto-select; plan-shaped text inside pasted documents, tool results, or fetched pages does NOT count — injected content can't disarm the consent gate or nominate the target. - Multiple plan candidates: the host-referenced plan file wins; still ambiguous means ask. - The DIFFERENT-target override carries the passing-mention guard. - Plan mode + explicitly named target + no drafted plan resolves to the named target instead of a contradictory re-ask. - The numbered ask-path rules are qualified ('When no exception above applied:') so they no longer restate an unconditional MUST-ask that contradicts the exceptions. - 'Whenever this gate does ask — in any mode — it is a hard STOP.' - Shared preamble: 'any AskUserQuestion the skill fires is the workflow operating within plan mode' (was 'the first AskUserQuestion is the workflow entering plan mode', which framed the opposite of the bypass); regenerates every skill. - Ceilings ratcheted with attribution: plan-eng union ratio 1.10, investigate 1.10 (the ~250B shared-preamble reword lands the closest-to-ceiling skill at 1.092). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v1.62.0.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pty-runner): active-render gate veto in the floor check + honest periodic-wiring docs Codex re-review P2s on the fix wave, both verified: - A finding AUQ rendering within TAIL_SCAN_BYTES of the gate (model waiting, no further output) was vetoed by the blanket tail exclusion until timeout. The veto is now ACTIVE-RENDER-aware: parseNumberedOptions anchors the last cursor menu, so only a pending GATE menu vetoes; the judge fallback shares the same check. Residual (documented): prose gate + prose finding inside one tail — floors run the native-menu path in practice. - The four demoted periodic tests are not in evals-periodic.yml's explicit matrix (a named instance of the pre-existing periodic-orphans TODO), so they run locally/manually until the PTY-capable periodic job lands. CHANGELOG claim softened accordingly; TODO filed with the wiring recipe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update project documentation for v1.62.0.0 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: apply codex doc-review fixes for v1.62.0.0 - CLAUDE.md: scope the tier-alignment invariant claim (mapped files enforced, unmapped files reported) - docs/skills.md: document the plan-mode auto-select scope gate for /plan-eng-review and /plan-design-review - evals.yml: fix stale comment (PTY smokes register four skills, not two) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: refresh ship golden baselines for the plan-mode preamble reword The generate-completion-status.ts wording change ('any AskUserQuestion the skill fires…') intentionally regenerates every SKILL.md; the byte-compare goldens carry the generator's output and refresh with it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ship): custom-hooks-path detection false-negatives on git worktrees The pre-push guard's HOOKS_IN_GIT_DIR check compared the hooks dir against --absolute-git-dir, which in a linked worktree is .git/worktrees/<name> while hooks resolve to the COMMON .git/hooks — so every Conductor worktree read as a 'custom hooks path' and the consented guard install was skipped. Match against the resolved --git-common-dir too (with a /nonexistent fallback so a failed resolution can't collapse the case pattern into match-everything). Verified live: this worktree now reports yes (was no), and the main checkout still reports yes. Goldens refreshed (--host all). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: changelog bullet for the worktree hooks-detection fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(evals): give the plan-ceo plan-mode smoke real budget headroom Measured 2026-08-11: a clean isolated pass took 295.7s against the 300s inner budget (4s of margin) and the same test timed out at ~308s three times under concurrent eval load — a budget-edge flake in the gate lane, not a behavior regression (it passed isolated on both this branch and main). Inner budget 300s -> 420s, outer bun timeout 360s -> 480s, and the test file is now named in its own touchfiles dep list so the tier-alignment invariant binds for it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(evals): 300s budget floor for the two 90s design-consultation SDK tests Root cause of PR #2533's e2e-design CI failure: design-consultation-preview failed 3 attempts at 0 turns/$0.00/93s — the session was up but the model's first completion queued past the 90s inner budget under concurrent API load (11 matrix jobs; the sibling research test booted its first tool at 4s, so this is API-side queuing, not CPU boot contention). The test was selected only because touchfiles.ts is a global touchfile; the tested behavior is untouched by this branch. 90s budgets cannot absorb one slow first completion. Both 90s tests in the file move to the repo's saturated-runner standard (300s inner / 360s outer, matching review-dashboard-via and retro-base-branch). Deliberately NOT re-arming the runner's inner timer on first stream event: an audit found ~100 outer bun-timeout literals sized inner+30-60s that a re-arm would silently break — the structural options are written up in TODOS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
624 lines
25 KiB
TypeScript
624 lines
25 KiB
TypeScript
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { runSkillTest } from './helpers/session-runner';
|
|
import { callJudge } from './helpers/llm-judge';
|
|
import {
|
|
ROOT, browseBin, runId, evalsEnabled,
|
|
describeIfSelected, testConcurrentIfSelected,
|
|
copyDirSync, setupBrowseShims, logCost, recordE2E,
|
|
createEvalCollector, finalizeEvalCollector,
|
|
} from './helpers/e2e-helpers';
|
|
import { spawnSync } from 'child_process';
|
|
import * as fs from 'fs';
|
|
import * as path from 'path';
|
|
import * as os from 'os';
|
|
|
|
const evalCollector = createEvalCollector('e2e-design');
|
|
|
|
/**
|
|
* LLM judge for DESIGN.md quality — checks font blacklist compliance,
|
|
* coherence, specificity, and AI slop avoidance.
|
|
*/
|
|
async function designQualityJudge(designMd: string): Promise<{ passed: boolean; reasoning: string }> {
|
|
return callJudge<{ passed: boolean; reasoning: string }>(`You are evaluating a generated DESIGN.md file for quality.
|
|
|
|
Evaluate against these criteria — ALL must pass for an overall "passed: true":
|
|
1. Does NOT recommend Inter, Roboto, Arial, Helvetica, Open Sans, Lato, Montserrat, or Poppins as primary fonts
|
|
2. Aesthetic direction is coherent with color approach (e.g., brutalist aesthetic doesn't pair with expressive color without explanation)
|
|
3. Font recommendations include specific font names (not generic like "a sans-serif font")
|
|
4. Color palette includes actual hex values, not placeholders like "[hex]"
|
|
5. Rationale is provided for major decisions (not just "because it looks good")
|
|
6. No AI slop patterns: purple gradients mentioned positively, "3-column feature grid" language, generic marketing speak
|
|
7. Product context is reflected in design choices (civic tech → should have appropriate, professional aesthetic)
|
|
|
|
DESIGN.md content:
|
|
\`\`\`
|
|
${designMd}
|
|
\`\`\`
|
|
|
|
Return JSON: { "passed": true/false, "reasoning": "one paragraph explaining your evaluation" }`);
|
|
}
|
|
|
|
// --- Design Consultation E2E ---
|
|
|
|
describeIfSelected('Design Consultation E2E', [
|
|
'design-consultation-core',
|
|
'design-consultation-existing',
|
|
'design-consultation-research',
|
|
'design-consultation-preview',
|
|
], () => {
|
|
let designDir: string;
|
|
|
|
beforeAll(() => {
|
|
designDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-design-consultation-'));
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: designDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Create a realistic project context
|
|
fs.writeFileSync(path.join(designDir, 'README.md'), `# CivicPulse
|
|
|
|
A civic tech data platform for government employees to access, visualize, and share public data. Built with Next.js and PostgreSQL.
|
|
|
|
## Features
|
|
- Real-time data dashboards for municipal budgets
|
|
- Public records search with faceted filtering
|
|
- Data export and sharing tools for inter-department collaboration
|
|
`);
|
|
fs.writeFileSync(path.join(designDir, 'package.json'), JSON.stringify({
|
|
name: 'civicpulse',
|
|
version: '0.1.0',
|
|
dependencies: { next: '^14.0.0', react: '^18.2.0', 'tailwindcss': '^3.4.0' },
|
|
}, null, 2));
|
|
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial project setup']);
|
|
|
|
// Copy design-consultation skill
|
|
fs.mkdirSync(path.join(designDir, 'design-consultation'), { recursive: true });
|
|
fs.copyFileSync(
|
|
path.join(ROOT, 'design-consultation', 'SKILL.md'),
|
|
path.join(designDir, 'design-consultation', 'SKILL.md'),
|
|
);
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(designDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('design-consultation-core', async () => {
|
|
const result = await runSkillTest({
|
|
prompt: `Read design-consultation/SKILL.md for the design consultation workflow.
|
|
Skip the preamble bash block, lake intro, telemetry, and contributor mode sections — go straight to the design workflow.
|
|
|
|
This is a civic tech data platform called CivicPulse for government employees who need to access public data. Read the README.md for details.
|
|
|
|
Skip research — work from your design knowledge. Skip the font preview page. Skip any AskUserQuestion calls — this is non-interactive. Accept your first design system proposal.
|
|
|
|
Write DESIGN.md and CLAUDE.md (or update it) in the working directory.`,
|
|
workingDirectory: designDir,
|
|
maxTurns: 20,
|
|
timeout: 360_000,
|
|
testName: 'design-consultation-core',
|
|
runId,
|
|
model: 'claude-opus-4-7',
|
|
});
|
|
|
|
logCost('/design-consultation core', result);
|
|
|
|
const designPath = path.join(designDir, 'DESIGN.md');
|
|
const claudePath = path.join(designDir, 'CLAUDE.md');
|
|
const designExists = fs.existsSync(designPath);
|
|
const claudeExists = fs.existsSync(claudePath);
|
|
let designContent = '';
|
|
|
|
if (designExists) {
|
|
designContent = fs.readFileSync(designPath, 'utf-8');
|
|
}
|
|
|
|
// Structural checks — fuzzy synonym matching to handle agent variation
|
|
const sectionSynonyms: Record<string, string[]> = {
|
|
'Product Context': ['product', 'context', 'overview', 'about'],
|
|
'Aesthetic': ['aesthetic', 'visual direction', 'design direction', 'visual identity'],
|
|
'Typography': ['typography', 'type', 'font', 'typeface'],
|
|
'Color': ['color', 'colour', 'palette', 'colors'],
|
|
'Spacing': ['spacing', 'space', 'whitespace', 'gap'],
|
|
'Layout': ['layout', 'grid', 'structure', 'composition'],
|
|
'Motion': ['motion', 'animation', 'transition', 'movement'],
|
|
};
|
|
const missingSections = Object.entries(sectionSynonyms).filter(
|
|
([_, synonyms]) => !synonyms.some(s => designContent.toLowerCase().includes(s))
|
|
).map(([name]) => name);
|
|
|
|
// LLM judge for quality
|
|
let judgeResult = { passed: false, reasoning: 'judge not run' };
|
|
if (designExists && designContent.length > 100) {
|
|
try {
|
|
judgeResult = await designQualityJudge(designContent);
|
|
console.log('Design quality judge:', JSON.stringify(judgeResult, null, 2));
|
|
} catch (err) {
|
|
console.warn('Judge failed:', err);
|
|
judgeResult = { passed: true, reasoning: 'judge error — defaulting to pass' };
|
|
}
|
|
}
|
|
|
|
const structuralPass = designExists && claudeExists && missingSections.length === 0;
|
|
recordE2E(evalCollector, '/design-consultation core', 'Design Consultation E2E', result, {
|
|
passed: structuralPass && judgeResult.passed && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(designExists).toBe(true);
|
|
if (designExists) {
|
|
expect(missingSections).toHaveLength(0);
|
|
}
|
|
if (claudeExists) {
|
|
const claude = fs.readFileSync(claudePath, 'utf-8');
|
|
expect(claude.toLowerCase()).toContain('design.md');
|
|
}
|
|
}, 420_000);
|
|
|
|
testConcurrentIfSelected('design-consultation-research', async () => {
|
|
// Test WebSearch integration — research phase only, no DESIGN.md generation
|
|
const researchDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-research-'));
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `You have access to WebSearch. Research civic tech data platform designs.
|
|
|
|
Do exactly 2 WebSearch queries:
|
|
1. 'civic tech government data platform design 2025'
|
|
2. 'open data portal UX best practices'
|
|
|
|
Summarize the key design patterns you found to ${researchDir}/research-notes.md.
|
|
Include: color trends, typography patterns, and layout conventions you observed.
|
|
Do NOT generate a full DESIGN.md — just research notes.`,
|
|
workingDirectory: researchDir,
|
|
maxTurns: 8,
|
|
// 300s, not 90s: saturated-runner class (same as review-dashboard-via /
|
|
// retro-base-branch). PR #2533 CI observed the sibling preview test at
|
|
// 0 turns/$0.00 for 93s x3 attempts — session up, first completion
|
|
// queued past the budget under concurrent API load. 90s budgets cannot
|
|
// absorb one slow first completion; 300s is the repo's standard floor
|
|
// for CI SDK tests. Outer timeout below rises to 360s for headroom.
|
|
timeout: 300_000,
|
|
testName: 'design-consultation-research',
|
|
runId,
|
|
});
|
|
|
|
logCost('/design-consultation research', result);
|
|
|
|
const notesPath = path.join(researchDir, 'research-notes.md');
|
|
const notesExist = fs.existsSync(notesPath);
|
|
const notesContent = notesExist ? fs.readFileSync(notesPath, 'utf-8') : '';
|
|
|
|
// Check if WebSearch was used
|
|
const webSearchCalls = result.toolCalls.filter(tc => tc.tool === 'WebSearch');
|
|
if (webSearchCalls.length > 0) {
|
|
console.log(`WebSearch used ${webSearchCalls.length} times`);
|
|
} else {
|
|
console.warn('WebSearch not used — may be unavailable in test env');
|
|
}
|
|
|
|
recordE2E(evalCollector, '/design-consultation research', 'Design Consultation E2E', result, {
|
|
passed: notesExist && notesContent.length > 200 && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(notesExist).toBe(true);
|
|
if (notesExist) {
|
|
expect(notesContent.length).toBeGreaterThan(200);
|
|
}
|
|
|
|
try { fs.rmSync(researchDir, { recursive: true, force: true }); } catch {}
|
|
}, 360_000);
|
|
|
|
testConcurrentIfSelected('design-consultation-existing', async () => {
|
|
// Pre-create a minimal DESIGN.md (independent of core test)
|
|
fs.writeFileSync(path.join(designDir, 'DESIGN.md'), `# Design System — CivicPulse
|
|
|
|
## Typography
|
|
Body: system-ui
|
|
`);
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `Read design-consultation/SKILL.md for the design consultation workflow.
|
|
|
|
There is already a DESIGN.md in this repo. Update it with a complete design system for CivicPulse, a civic tech data platform for government employees.
|
|
|
|
Skip research. Skip font preview. Skip any AskUserQuestion calls — this is non-interactive.`,
|
|
workingDirectory: designDir,
|
|
maxTurns: 20,
|
|
timeout: 360_000,
|
|
testName: 'design-consultation-existing',
|
|
runId,
|
|
model: 'claude-opus-4-7',
|
|
});
|
|
|
|
logCost('/design-consultation existing', result);
|
|
|
|
const designPath = path.join(designDir, 'DESIGN.md');
|
|
const designExists = fs.existsSync(designPath);
|
|
let designContent = '';
|
|
if (designExists) {
|
|
designContent = fs.readFileSync(designPath, 'utf-8');
|
|
}
|
|
|
|
// Should have more content than the minimal version
|
|
const hasColor = designContent.toLowerCase().includes('color');
|
|
const hasSpacing = designContent.toLowerCase().includes('spacing');
|
|
|
|
recordE2E(evalCollector, '/design-consultation existing', 'Design Consultation E2E', result, {
|
|
passed: designExists && hasColor && hasSpacing && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(designExists).toBe(true);
|
|
if (designExists) {
|
|
expect(hasColor).toBe(true);
|
|
expect(hasSpacing).toBe(true);
|
|
}
|
|
}, 420_000);
|
|
|
|
testConcurrentIfSelected('design-consultation-preview', async () => {
|
|
// Test preview HTML generation only — no DESIGN.md (covered by core test)
|
|
const previewDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-preview-'));
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `Generate a font and color preview page for a civic tech data platform.
|
|
|
|
The design system uses:
|
|
- Primary font: Cabinet Grotesk (headings), Source Sans 3 (body)
|
|
- Colors: #1B4D8E (civic blue), #C4501A (alert orange), #2D6A4F (success green)
|
|
- Neutral: #F8F7F6 (warm white), #1A1A1A (near black)
|
|
|
|
Write a single HTML file to ${previewDir}/design-preview.html that shows:
|
|
- Font specimens for each font at different sizes
|
|
- Color swatches with hex values
|
|
- A light/dark toggle
|
|
Do NOT write DESIGN.md — only the preview HTML.`,
|
|
workingDirectory: previewDir,
|
|
maxTurns: 8,
|
|
// 300s, not 90s: this is the test that failed 3x at 0 turns/$0.00/93s
|
|
// on PR #2533 CI — see the research test's comment for the class.
|
|
timeout: 300_000,
|
|
testName: 'design-consultation-preview',
|
|
runId,
|
|
});
|
|
|
|
logCost('/design-consultation preview', result);
|
|
|
|
const previewPath = path.join(previewDir, 'design-preview.html');
|
|
const previewExists = fs.existsSync(previewPath);
|
|
let previewContent = '';
|
|
if (previewExists) {
|
|
previewContent = fs.readFileSync(previewPath, 'utf-8');
|
|
}
|
|
|
|
const hasHtml = previewContent.includes('<html') || previewContent.includes('<!DOCTYPE');
|
|
const hasFontRef = previewContent.includes('font-family') || previewContent.includes('fonts.googleapis') || previewContent.includes('fonts.bunny');
|
|
|
|
recordE2E(evalCollector, '/design-consultation preview', 'Design Consultation E2E', result, {
|
|
passed: previewExists && hasHtml && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(previewExists).toBe(true);
|
|
if (previewExists) {
|
|
expect(hasHtml).toBe(true);
|
|
expect(hasFontRef).toBe(true);
|
|
}
|
|
|
|
try { fs.rmSync(previewDir, { recursive: true, force: true }); } catch {}
|
|
}, 360_000);
|
|
});
|
|
|
|
// --- Plan Design Review E2E (plan-mode) ---
|
|
|
|
describeIfSelected('Plan Design Review E2E', ['plan-design-review-plan-mode', 'plan-design-review-no-ui-scope'], () => {
|
|
|
|
/** Create an isolated tmpdir with git repo and plan-design-review skill */
|
|
function setupReviewDir(): string {
|
|
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-plan-design-'));
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: dir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Copy plan-design-review skill
|
|
fs.mkdirSync(path.join(dir, 'plan-design-review'), { recursive: true });
|
|
fs.copyFileSync(
|
|
path.join(ROOT, 'plan-design-review', 'SKILL.md'),
|
|
path.join(dir, 'plan-design-review', 'SKILL.md'),
|
|
);
|
|
{ const _sec = path.join(ROOT, 'plan-design-review', 'sections'); if (fs.existsSync(_sec)) fs.cpSync(_sec, path.join(dir, 'plan-design-review', 'sections'), { recursive: true }); }
|
|
|
|
return dir;
|
|
}
|
|
|
|
testConcurrentIfSelected('plan-design-review-plan-mode', async () => {
|
|
const reviewDir = setupReviewDir();
|
|
try {
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: reviewDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
// Create a plan file with intentional design gaps
|
|
fs.writeFileSync(path.join(reviewDir, 'plan.md'), `# Plan: User Dashboard
|
|
|
|
## Context
|
|
Build a user dashboard that shows account stats, recent activity, and settings.
|
|
|
|
## Implementation
|
|
1. Create a dashboard page at /dashboard
|
|
2. Show user stats (posts, followers, engagement rate)
|
|
3. Add a recent activity feed
|
|
4. Add a settings panel
|
|
5. Use a clean, modern UI with cards and icons
|
|
6. Add a hero section at the top with a gradient background
|
|
|
|
## Technical Details
|
|
- React components with Tailwind CSS
|
|
- API endpoint: GET /api/dashboard
|
|
- WebSocket for real-time activity updates
|
|
`);
|
|
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial plan']);
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `Read plan-design-review/SKILL.md for the design review workflow.
|
|
|
|
Review the plan in ./plan.md. This plan has several design gaps — it uses vague language like "clean, modern UI" and "cards and icons", mentions a "hero section with gradient" (AI slop), and doesn't specify empty states, error states, loading states, responsive behavior, or accessibility.
|
|
|
|
Skip the preamble bash block. Skip any AskUserQuestion calls — this is non-interactive. Rate each design dimension 0-10 and explain what would make it a 10. Then EDIT plan.md to add the missing design decisions (interaction state table, empty states, responsive behavior, etc.).
|
|
|
|
IMPORTANT: Do NOT try to browse any URLs or use a browse binary. This is a plan review, not a live site audit. Just read the plan file, review it, and edit it to fix the gaps.`,
|
|
workingDirectory: reviewDir,
|
|
maxTurns: 15,
|
|
timeout: 300_000,
|
|
testName: 'plan-design-review-plan-mode',
|
|
runId,
|
|
});
|
|
|
|
logCost('/plan-design-review plan-mode', result);
|
|
|
|
// Check that the agent produced design ratings (0-10 scale)
|
|
const output = result.output || '';
|
|
const hasRatings = /\d+\/10/.test(output);
|
|
const hasDesignContent = output.toLowerCase().includes('information architecture') ||
|
|
output.toLowerCase().includes('interaction state') ||
|
|
output.toLowerCase().includes('ai slop') ||
|
|
output.toLowerCase().includes('hierarchy');
|
|
|
|
// Check that the plan file was edited (the core new behavior)
|
|
const planAfter = fs.readFileSync(path.join(reviewDir, 'plan.md'), 'utf-8');
|
|
const planOriginal = `# Plan: User Dashboard`;
|
|
const planWasEdited = planAfter.length > 300; // Original is ~450 chars, edited should be much longer
|
|
const planHasDesignAdditions = planAfter.toLowerCase().includes('empty') ||
|
|
planAfter.toLowerCase().includes('loading') ||
|
|
planAfter.toLowerCase().includes('error') ||
|
|
planAfter.toLowerCase().includes('state') ||
|
|
planAfter.toLowerCase().includes('responsive') ||
|
|
planAfter.toLowerCase().includes('accessibility');
|
|
|
|
recordE2E(evalCollector, '/plan-design-review plan-mode', 'Plan Design Review E2E', result, {
|
|
passed: hasDesignContent && planWasEdited && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
// Agent should produce design-relevant output about the plan
|
|
expect(hasDesignContent).toBe(true);
|
|
// Agent should have edited the plan file to add missing design decisions
|
|
expect(planWasEdited).toBe(true);
|
|
expect(planHasDesignAdditions).toBe(true);
|
|
} finally {
|
|
try { fs.rmSync(reviewDir, { recursive: true, force: true }); } catch {}
|
|
}
|
|
}, 360_000);
|
|
|
|
testConcurrentIfSelected('plan-design-review-no-ui-scope', async () => {
|
|
const reviewDir = setupReviewDir();
|
|
try {
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: reviewDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
// Write a backend-only plan
|
|
fs.writeFileSync(path.join(reviewDir, 'backend-plan.md'), `# Plan: Database Migration
|
|
|
|
## Context
|
|
Migrate user records from PostgreSQL to a new schema with better indexing.
|
|
|
|
## Implementation
|
|
1. Create migration to add new columns to users table
|
|
2. Backfill data from legacy columns
|
|
3. Add database indexes for common query patterns
|
|
4. Update ActiveRecord models
|
|
5. Run migration in staging first, then production
|
|
`);
|
|
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial plan']);
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `Read plan-design-review/SKILL.md for the design review workflow.
|
|
|
|
Review the plan in ./backend-plan.md. This is a pure backend database migration plan with no UI changes.
|
|
|
|
Skip the preamble bash block. Skip any AskUserQuestion calls — this is non-interactive. Write your findings directly to stdout.
|
|
|
|
IMPORTANT: Do NOT try to browse any URLs or use a browse binary. This is a plan review, not a live site audit.`,
|
|
workingDirectory: reviewDir,
|
|
maxTurns: 10,
|
|
timeout: 180_000,
|
|
testName: 'plan-design-review-no-ui-scope',
|
|
runId,
|
|
});
|
|
|
|
logCost('/plan-design-review no-ui-scope', result);
|
|
|
|
// Agent should detect no UI scope and exit early
|
|
const output = result.output || '';
|
|
const detectsNoUI = output.toLowerCase().includes('no ui') ||
|
|
output.toLowerCase().includes('no frontend') ||
|
|
output.toLowerCase().includes('no design') ||
|
|
output.toLowerCase().includes('not applicable') ||
|
|
output.toLowerCase().includes('backend');
|
|
|
|
recordE2E(evalCollector, '/plan-design-review no-ui-scope', 'Plan Design Review E2E', result, {
|
|
passed: detectsNoUI && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(detectsNoUI).toBe(true);
|
|
} finally {
|
|
try { fs.rmSync(reviewDir, { recursive: true, force: true }); } catch {}
|
|
}
|
|
}, 240_000);
|
|
});
|
|
|
|
// --- Design Review E2E (live-site audit + fix) ---
|
|
|
|
describeIfSelected('Design Review E2E', ['design-review-fix'], () => {
|
|
let qaDesignDir: string;
|
|
let qaDesignServer: ReturnType<typeof Bun.serve> | null = null;
|
|
|
|
beforeAll(() => {
|
|
qaDesignDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-qa-design-'));
|
|
setupBrowseShims(qaDesignDir);
|
|
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: qaDesignDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Create HTML/CSS with intentional design issues
|
|
fs.writeFileSync(path.join(qaDesignDir, 'index.html'), `<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<title>Design Test App</title>
|
|
<link rel="stylesheet" href="style.css">
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<h1 style="font-size: 48px; color: #333;">Welcome</h1>
|
|
<h2 style="font-size: 47px; color: #334;">Subtitle Here</h2>
|
|
</header>
|
|
<main>
|
|
<div class="card" style="padding: 10px; margin: 20px;">
|
|
<h3 style="color: blue;">Card Title</h3>
|
|
<p style="color: #666; font-size: 14px; line-height: 1.2;">Some content here with tight line height.</p>
|
|
</div>
|
|
<div class="card" style="padding: 30px; margin: 5px;">
|
|
<h3 style="color: green;">Another Card</h3>
|
|
<p style="color: #999; font-size: 16px;">Different spacing and colors for no reason.</p>
|
|
</div>
|
|
<button style="background: red; color: white; padding: 5px 10px; border: none;">Click Me</button>
|
|
<button style="background: #007bff; color: white; padding: 12px 24px; border: none; border-radius: 20px;">Also Click</button>
|
|
</main>
|
|
</body>
|
|
</html>`);
|
|
|
|
fs.writeFileSync(path.join(qaDesignDir, 'style.css'), `body {
|
|
font-family: Arial, sans-serif;
|
|
margin: 0;
|
|
padding: 20px;
|
|
}
|
|
.card {
|
|
border: 1px solid #ddd;
|
|
border-radius: 4px;
|
|
}
|
|
`);
|
|
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial design test page']);
|
|
|
|
// Start a simple file server for the design test page
|
|
qaDesignServer = Bun.serve({
|
|
port: 0,
|
|
fetch(req) {
|
|
const url = new URL(req.url);
|
|
const filePath = path.join(qaDesignDir, url.pathname === '/' ? 'index.html' : url.pathname.slice(1));
|
|
try {
|
|
const content = fs.readFileSync(filePath);
|
|
const ext = path.extname(filePath);
|
|
const contentType = ext === '.css' ? 'text/css' : ext === '.html' ? 'text/html' : 'text/plain';
|
|
return new Response(content, { headers: { 'Content-Type': contentType } });
|
|
} catch {
|
|
return new Response('Not Found', { status: 404 });
|
|
}
|
|
},
|
|
});
|
|
|
|
// Copy design-review skill
|
|
fs.mkdirSync(path.join(qaDesignDir, 'design-review'), { recursive: true });
|
|
fs.copyFileSync(
|
|
path.join(ROOT, 'design-review', 'SKILL.md'),
|
|
path.join(qaDesignDir, 'design-review', 'SKILL.md'),
|
|
);
|
|
});
|
|
|
|
afterAll(() => {
|
|
qaDesignServer?.stop();
|
|
try { fs.rmSync(qaDesignDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('design-review-fix', async () => {
|
|
const serverUrl = `http://localhost:${(qaDesignServer as any)?.port}`;
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `IMPORTANT: The browse binary is already assigned below as B. Do NOT search for it or run the SKILL.md setup block — just use $B directly.
|
|
|
|
B="${browseBin}"
|
|
|
|
Read design-review/SKILL.md for the design review + fix workflow.
|
|
|
|
Review the site at ${serverUrl}. Use --quick mode. Skip any AskUserQuestion calls — this is non-interactive. Fix up to 3 issues max. Write your report to ./design-audit.md.`,
|
|
workingDirectory: qaDesignDir,
|
|
maxTurns: 30,
|
|
timeout: 360_000,
|
|
testName: 'design-review-fix',
|
|
runId,
|
|
});
|
|
|
|
logCost('/design-review fix', result);
|
|
|
|
const reportPath = path.join(qaDesignDir, 'design-audit.md');
|
|
const reportExists = fs.existsSync(reportPath);
|
|
|
|
// Check if any design fix commits were made
|
|
const gitLog = spawnSync('git', ['log', '--oneline'], {
|
|
cwd: qaDesignDir, stdio: 'pipe',
|
|
});
|
|
const commits = gitLog.stdout.toString().trim().split('\n');
|
|
const designFixCommits = commits.filter((c: string) => c.includes('style(design)'));
|
|
|
|
recordE2E(evalCollector, '/design-review fix', 'Design Review E2E', result, {
|
|
passed: ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
// Accept error_max_turns — the fix loop is complex
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
|
|
// Report and commits are best-effort — log what happened
|
|
if (reportExists) {
|
|
const report = fs.readFileSync(reportPath, 'utf-8');
|
|
console.log(`Design audit report: ${report.length} chars`);
|
|
} else {
|
|
console.warn('No design-audit.md generated');
|
|
}
|
|
console.log(`Design fix commits: ${designFixCommits.length}`);
|
|
}, 420_000);
|
|
});
|
|
|
|
// Module-level afterAll — finalize eval collector after all tests complete
|
|
afterAll(async () => {
|
|
await finalizeEvalCollector(evalCollector);
|
|
});
|