mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 06:24:41 +01:00
* test(qa-callers): bind QA section names in the caller prompt instead of teaching helper forms (W3-4)
The caller session prompt taught full capture/checkpoint syntax but only
mentioned materialize, so two ship-exploratory-small-cli sessions (paid
gate runs 38973329985, 38993076767) skipped qa/sections/exploratory.md and
system-functional.md and guessed a bare materialize. The prompt now
declares only fixture bindings (DEADLINE_TOOL, EVIDENCE_TOOL, PROBE_DIR,
DEADLINE_FILE, --public approval, --timeout-ms 10000 for plan probes) and
points to the section for every form. The read requirement is unchanged;
both captured transcripts are kept as negative controls.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(review-army): run the state-machine detector in the scope block and check its exit status (W3-5)
Specialist 9 named gstack-state-matrix detect only in selection prose, and
small diffs keep only State Machines, so agents ran an undeclared command
(paid gate run 39002269281). The generated scope block now runs the
detector with its exit status checked: on failure it keeps the detector's
diagnostics and sets SCOPE_STATE_MACHINE=unknown, and selection dispatches
State Machines when the signal is true or unknown. The check step becomes a
declared heredoc block. The clarity test requires every non-diff-scope
signal to come from a detect helper the scope block runs; a free test runs
the block against a failing detector stub; the captured standalone call
stays outside the caller interface.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(third-party-actions): letter the absent-Aside consent options B/C/D (W3-7)
The contract fixed letters only when Aside is detected, so a tpa-absent-linux
agent numbered its consent question 1/2/3 (paid run 39011101113), which the
lettered-offer inspection cannot read. The absent branch now offers
B) gstack's own visible browser, C) manual instructions, D) defer and omits
A). The paid test's assertion is unchanged; the captured numbered answer is
a documented fail example in the unit test.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(plan-reality): say tier 1 rows are never n/a (W3-8)
The plan-review reality block offered pass|finding|n/a without saying tier 1
rows may never be n/a, so a plan-mode-no-op session wrote
`REALITY: deferred-asks n/a ...`, check returned repair, and the repair loop
ran past the 300 s capture (paid run 39004466422). The block now says tier
1 rows are never n/a and that with no linked issue deferred-asks is pass
with a receipt showing none. The captured line is pinned through check as
missing=deferred-asks verdict=repair.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(qa-functional): webhook fix runs re-verify the repaired invariant and rerun superseded captures (W3-6, W3-14)
W3-6: stage 2 of the fix-run prompt asked only for the original, happy,
cancel and dependency probes while verifyQANativeRegression reruns all
eight, so a status-only repair that left partial broken reached the verifier
(paid run 38986120559). The webhook fix prompt now states the capture budget
up front (at most 5 pre-repair captures, then the 7 post-repair scenarios:
duplicate, partial, concurrent-ab, concurrent-ba, happy, cancel, dependency;
12 in total; report incomplete if 5 cannot establish the defect). That list
feeds the capture nudge (GSTACK_QA_REQUIRED_PROBES);
QA_WEBHOOK_REQUIRED_SCENARIOS and the verdict are unchanged.
W3-14: the fix prompt now says to rerun every pre-repair capture the
verdict relies on after the final repair, and that a superseded row stays
open until its command reran on the repaired source (paid run 39053431488
labelled a pre-repair duplicate superseded and never reran it). The
detector and its negative test are unchanged; the captured shape joins them
as a negative control.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(qa): tie the exploratory stop rule to per-rule input probes; fix runs close superseded captures (W3-9, W3-14)
W3-9: two qa-functional-cli-report sessions stopped at 5-7 of 12 probes
without probing a non-digit amount (paid runs 38983306684, 39001107574),
though system-functional.md requires one probe per documented input rule.
The single stop rule shared by both exploratory templates now says each
documented input rule needs its own probe breaking only that rule, and
rules left uncovered at a probe or time cap go in limits.
W3-14: for fix runs (not qa-only), every capture taken before a fix is
superseded even when its client prints no snapshot; before completion the
agent reruns each one its verdict relies on or reports it as an open
remainder. Production superseded/open semantics are unchanged.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(qa-functional): keep the captured webhook-report cat attempts as denied negative controls (Q3)
Paid run 39006896632 (qa-functional-webhook-report) attempted
`cat package.json probe.ts .gitignore` and `cat src/*.ts test/*.ts`; the
PreToolUse hook denied both. This is a model miss, not a detector defect, so
both commands are pinned as denied by the registered hook and outside
qaCommandAllowed. No detector change.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(docs-candidate): per-subcommand --help and a compare reject that names the fix (W3-2)
snapshot --help and compare --help print the usage and exit 0 before any side
effect; the docsync observer admits exactly those two forms. compare names the
record path, each missing or wrong field (schema_version, base, hashes) and
the snapshot command that writes a valid record, keeping exit 1. The captured
ship-docsync-stale-after commands from PR gate run 38990111230 replay inside
the interface; help with any other argument stays rejected.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(docsync-fault): prepare writes the production candidate through the installed snapshot (W3-1)
The fault actor prepare wrote its own {base_sha, content_hashes, ...} record,
which the installed gstack-docs-candidate compare rejects, while
ship/sections/documentation.md requires running compare; 79 of 198 docsync
captures (and the red in PR gate run 38990111230) were pushed off the
documented path by it.
prepare now runs the installed snapshot under a 30 s timeout for the same
selection, returns its exact command as snapshot_command so the post-child
rerun is literal, publishes prompt and event only after a valid record
exists, and removes a record its own failed snapshot left. dispatch observes
the candidate with the same installed snapshot; inspect returns base, index
and per-file git blob ids in the record representation; the legacy re-audit
check reads selected/hashes. The fault prompt names snapshot_command and the
new field names.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(todos): file the qa-quick capture headroom follow-up (fix wave 3, T)
qa-quick timed out once at 300 s (paid run 38980005793) against passing
runs of 164-272 s. Trimming what the Quick tier loads is a product call, so
it is filed, not repaired; no budget change.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(docsync): keep the two docsync model misses as negative controls (W3-3)
The Bash grep of ship-docsync-missing-asset (PR gate run 38983306684,
bash#13) and the composed cat; cat of ship-docsync-failure (38984977679,
bash#12) were correct rejections. They join the stored known_bad commands
(with a {home} placeholder for the fixture home); no detector change.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(golden): refresh the ship goldens for the W3-5 scope block and W3-7 consent letters
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(third-party-actions): state the absent-Aside letters in fewer characters (W3-7)
Same contract (B), C), D) when Aside is not detected; the next sentence
already omits the Aside option), 67 fewer characters on the pinned
land-and-deploy and office-hours skills.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(skill-md-size): re-pin 31 over-limit skills grown by fix wave 3 prompt repairs (owner sign-off needed)
CREW rule 11: these skills were already over 100,000 characters and grow
by the minimum text each repair needs; none crosses the limit for the
first time.
- gstack-review on cursor, factory, kiro, opencode, slate (+657..+676):
W3-5 runs gstack-state-matrix detect inside the scope block with its exit
status checked and declares the check step as a heredoc block; the new
fence carries the env-var-host runtime prelude (about 360 characters).
- gstack-plan-eng-review on 12 hosts (+104): W3-8 says tier 1 reality rows
are never n/a (paid run 39004466422 looped on a rejected n/a row).
- gstack-land-and-deploy on 12 hosts and gstack-office-hours on 4 hosts
(+21): W3-7 letters the absent-Aside consent options B), C), D).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(skills): fit the W3-5, W3-7 and W3-8 prompt repairs inside the parity budgets
The parity suite left 2-42 bytes of headroom on ship, review,
plan-eng-review, plan-ceo-review and land-and-deploy, so each repair now
pays for itself in the same resolver:
- W3-5: the scope block runs `source <(gstack-state-matrix detect --base
<base> || echo SCOPE_STATE_MACHINE=unknown)` (a failed detector's
stdout is overridden by the later unknown assignment; its stderr still
shows), the check block uses a short heredoc, and DIFF_LINES comes from one
`git diff --numstat | awk` sum instead of two --stat greps (same count:
insertions plus deletions, binary files zero).
- W3-7: the absent branch reads "offer only options B), C) and D)."
- W3-8: "never n/a" joins the tier 1 parenthesis, the no-linked-issue rule
joins the receipt sentence, and the deep header says the same in fewer
words; the redundant "(it prints the PLAN_REALITY: line)" is dropped.
Size pins: land-and-deploy (12 hosts, -20), office-hours (4 hosts, -20)
and plan-eng-review (12 hosts, -10) are lowered to their new sizes;
gstack-review on cursor, factory, kiro, opencode and slate stays above its
main pin (+355..+374) because the declared check block is a new fence that
carries the env-var-host runtime prelude. Raising those five pins needs
owner sign-off.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(design-e2e): the shim-dom scan check resolves the command's literal _REPORT/_RUN (W3-10)
PR gate run 38980005793 went red because the agent ran the generated DOM
block, scanning "$_REPORT/dom/$_RUN" with _RUN="run1" assigned in the same
command, and the check looked for the literal dom/run1 anywhere in a command.
domScanOfRun substitutes that command's own literal _REPORT=/_RUN=
assignments and credits a scan only when one of its own arguments is the run
directory. scan --changed, another directory, quoted heredoc report text and
non-literal or conflicting assignments stay negative controls. All 15
captured shim-dom runs in the window pass the new check (14 passed the old).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(pty): keep the last non-blank viewport for failure dumps (W3-11)
autoplan-guard-pty (PR gate run 38976253774) stalled at 489 s and its screen
dump, the tail of visibleText(), was only spinner redraws. Live PTY sessions
now remember the last non-blank viewport a screen read returned
(session.lastScreen()) and the harness wait-timeout dumps append it.
Diagnostic only: visibleText(), matching and verdicts are unchanged.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(autoplan): explicit 30 s timeout for each answer-invalidation test (W3-12)
The eng-only override test ran 4.1-4.4 s on a 4-core machine against Bun's
5 s default (7.8-9.4 s per file). Each test now names a 30 s budget.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(review-army): size the nested bun test budgets for windows-latest (W3-15)
windows-free-tests run 39053431430 (shard 5): the nested bun test exited 1 in
5.58 s because Bun's default 5 s hook timeout fired on the real Red Team
beforeAll (8 git spawns plus skill-section copies). Both nested argvs pass
--timeout 30000, which raises only the nested default; the registrations
under test keep their explicit timeouts. Spawn timeouts move from 8 s to
40 s and outer test timeouts to 45 s, including the consensus test that had
none. With a 6 s beforeAll injected, the lifecycle still matches; the old
argv reproduces the Windows hook timeout.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(todos): close the createQAFunctionalFixture entry and file D6 (W3-13)
The git auto-maintenance entry was fixed by 0bf9d50 (#3190). D6: nested bun
test output pollutes free-runner failure attribution in
scripts/lib/shard-engine.ts (windows-free-tests 39053431430).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(free-fixtures): declare the W3-4 captured caller transcript as a free-only fixture
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(pty): strip cursor-visibility escapes and print the last screen on autoplan-guard-pty failure (W3-11)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(review-army): keep the state-machine detector's output only on exit 0 (W3-5 confirm round)
A failing gstack-state-matrix detect prints SCOPE_STATE_MACHINE=false before
exiting non-zero. The scope block now captures its output and sources it only
when the detector succeeds; otherwise it discards the output, sets
SCOPE_STATE_MACHINE=unknown (dispatching the State Machines specialist) and
leaves the detector's stderr visible. Free tests cover false-output/non-zero,
empty-output/non-zero and a late failure after printing true. The stack
comment is shortened so gstack-review stays 4 chars under its pin; the five
host pins are lowered to match.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(qa-functional): reserve webhook post-repair reruns first inside the unchanged 12-capture ceiling (W3-6/W3-14 confirm round)
The fix prompt now states precedence: post-repair reruns (the 7 stage-2
scenarios plus each distinct pre-repair command the verdict relies on, such as
a reject probe) are reserved first, pre-repair exploration gets what remains,
and reruns that cannot fit are reported as open. A free test fits a pre-repair
reject plus the reproduced defect and its replay within 12 (11 captures).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(docs-candidate): the compare reject never sends an agent to re-snapshot after the child audit (W3-2)
A post-audit snapshot over the candidate replaces the before-audit baseline, so
the ownership check would compare current files with themselves. The fix line
now says the candidate must be the record the Prepare snapshot wrote, that the
Prepare snapshot is rerun only before dispatching the child audit, and that the
post-child snapshot uses a different --out (<audit-id>-post.json). The helper
test pins the text.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(docsync-fault): diagnosable prepare snapshot failures with one fresh-id retry (W3-1)
A failed installed snapshot now returns exit 25 with a stable reason code
(snapshot_exit, snapshot_timeout, snapshot_spawn_error, snapshot_invalid_record),
the helper's stderr tail redacted through lib/redact-engine with the fixture
root as <HOME> and bounded to 800 characters, and keeps the attempt's files: the
candidate the helper left, if any, and prepare-<audit-id>-failure.json. The
failure is a prepare-failed event, not a rejected interaction. One retry is
allowed under a fresh audit id; reusing the failed id, dispatching it or a third
prepare is rejected. Tested before and after candidate creation. The two
snapshot tests get an explicit 30 s timeout: the compare/inspect agreement
test runs 4.3-4.8 s alone and timed out at Bun's 5 s default under load.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(qa-evidence): count required post-repair probes only after the fixture-declared final source edit (W3-6/W3-14 confirm round)
GSTACK_QA_REQUIRED_PROBES reminders counted any complete capture, so a
pre-repair duplicate or partial capture looked like post-repair coverage. The
caller may now declare GSTACK_QA_REQUIRED_AFTER_EDIT with absolute paths and a
clean-fixture time: only captures started after the last edit under those
paths count, none count before the first edit, and a malformed declaration
counts none. Without the declaration every capture counts as before. The
webhook fix fixture declares its src/ directory. A free test covers
pre-repair-only coverage (nudges), complete post-repair coverage (no nudge)
and a second edit (nudges again).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(harness): bump HARNESS_VERSION to 5 for the W3-11 stripAnsi change
stripAnsi now removes cursor-visibility escapes (ESC[?25h/l). Every PTY case
matches patterns against its output, so a pattern spanning one of those escapes
can now match where it did not before: a verdict can change, so this is a
behavioral bump, not a non-behavioral entry. The launch.ts change (last
non-blank screen in failure messages) is diagnostic only.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(review-army): pass the state-machines output to check as a written file, not a heredoc placeholder (W3-5)
The generated-bash lint (env-host-fence-conformance INV-3) rejects free text in
a heredoc body: <its complete output> reached the shell. The agent now writes
the specialist's full output to .gstack/tmp/state-machines.md and the fence
runs gstack-state-matrix check on that file. Two stack-detection comments and
the parenthetical are trimmed to stay inside the review/ship parity ratios;
gstack-review shrinks by 33 on the five pinned hosts and their pins are
lowered to match.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(autoplan-guard-pty): the scripted actor ends its turn after a background CEO dispatch before reading the close packet
PR gate 39070276914 (dc739be) failed at the close-packet check while gate
39069171159 (b789898) passed with byte-identical autoplan inputs. The evidence
differs only in order: the failing actor read the close packet in the dispatch
turn, so the first tool call after the CEO notice was the publication. The
product guard allowed all four guarded calls in both runs. The fixture step 5
told the actor to end its turn only conditionally; step 5 now ends the turn as
soon as the result says the agent runs in the background, and step 6 reads the
close packet only after the reviewer finished. The close-packet check moves to
closePacketReadAfterNotice, replayed in a free test against both captured
orders; a second free test pins the actor's steps 5 and 6.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore(release): v1.91.127.0 fix wave 3 paid-gate reliability round 2
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(qa): input-rule probes are per field, and a value breaking two rules covers neither (W3-9)
PR gate 39073934326 (93449bc) failed qa-functional-cli-report with no observed
unannounced defect, the same class as 38983306684 and 39001107574. The agent
read 'allowed characters' as the ID rule only, counted -5 as its sign probe
and never probed the amount's digits-only rule with an otherwise valid value,
so the seeded parseInt defect stayed hidden. The stop rule now says each
field's documented input rule needs a probe breaking only it, and that a
signed amount breaks sign and digits-only and covers neither; the functional
table row names input rules per field. The qa union stays inside its parity
ratio.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(qa): state the input-rule principle instead of a worked example (W3-9)
The signed-amount example in 0fe616e was misread in the focused rerun as 'a
signed amount breaks only the digits-only rule', and the run again missed
the seeded non-digit defect. The stop rule now states the principle: each
field's documented input rule needs a probe that breaks only it and satisfies
every other rule.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(tpa): the Darwin pitch detector accepts the canonical phrase anywhere, short forms only at a sentence start
PR gate 39075977549 (d6d82fb) failed tpa-absent-darwin on a correct pitch:
'One-time note: you can download Aside (macOS 15+) at aside.com, then open it,
sign in, and re-run'. The contract is to deliver the pitch once; the
sentence-start anchor exists only so narration is not counted. The full
canonical phrase now counts anywhere unless a negation precedes it, and the
short forms (download it / download Aside at aside.com) keep the
sentence-start rule. The detector moves to test/helpers/third-party-actions.ts
with a count helper. Free tests replay the captured red (counts once), the
four captured passing Darwin pitches (once each), five Linux and five READY
narrations plus both recovery-public responses (zero), a mid-sentence short
form and negated phrases (zero), and a duplicated phrase (two).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(qa-callers): only a required plan check may swap the deadline for --timeout-ms; reruns keep the deadline
PR gate 39077524802 (0f71a55) failed ship-exploratory-late-input: after the
late input change the actor reran captures 001 and 002 with --timeout-ms
10000, reading the fixture's 'explicit plan probes outside the smoke budget'
clause as covering revalidation reruns, and the verifier reported both as
smoke probes missing a trusted deadline run. The binding now says only an
explicit plan check this phase requires may use --timeout-ms, and every other
capture, including a rerun or revalidation, uses the deadline.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(third-party-actions): count the Darwin pitch only at a recommendation boundary
/review delta on 51f1e02 (Claude and GPT-6.1 Sol voices): the 0f71a55
detector accepted the canonical phrase anywhere unless the adjacent word was
a negation, so a Darwin refusal or a quoted policy mention ('I will not ask
you to download Aside ...', 'The Darwin policy says to "download Aside ..."',
'You cannot download ...') satisfied the paid case's positive match, and a
free assertion even expected "You don't need to download ..." to count.
The canonical phrase now counts only where a recommendation can start: a
sentence, clause (, : ; dash), bullet, line or bold start, optionally led by
then/please/just/so/first/now and you can/could/may/should/need to, with no
negation directly before it across a line break. Quoted, parenthesised,
reported and clause-negated mentions fall outside that boundary. Every probe
from both voices is a free test; the five captured Darwin pitches (including
the 39075977549 red) still count 1 each and the Linux/READY captures 0.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(review-army): write the state-machines output to a fresh private file from gstack-state-matrix outfile
/review delta on 51f1e02 (Claude F2): dc739be wrote the specialist output to
a fixed .gstack/tmp/state-machines.md with no exclude. On a second review
Claude Code's Write refuses an existing unread file, so check could validate
the previous run's output, and git add -A could commit it.
gstack-state-matrix outfile creates .gstack/tmp (0700) at the repo top,
adds /.gstack/tmp/ to info/exclude once, and prints a new empty 0600 file
per call; the selection prose has the agent Read that file, write the full
output there and check that path. The prose moved the check command inline,
so gstack-review shrinks and its pins are lowered (cursor 141010->140637,
factory 144792->144416, kiro 140816->140449, opencode 143258->142872,
slate 140913->140543).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(qa-evidence): pin the post-repair edit's mtime to capture 002's completion
The GPT voice's /review run on 51f1e02 failed qa-evidence.test.ts:547 once:
capture 003 reported no required probe remaining, so capture 002 (started
before the edit) counted as post-edit. The test wrote the edit right after
002 returned, a gap of one child run (7-27 ms measured here; the voice's
machine ran this test in 59 ms against 183 ms here, so a few ms there),
while an ext4 mtime comes from the kernel's coarse clock and lags
Date.now() by up to a tick (0.5-4.5 ms measured here).
The edit's mtime is now set to 002's recorded completedAt, so 002 started
strictly before the boundary and 003 starts after it on any machine. The
product boundary is unchanged: tick-scale imprecision only matters when a
capture starts within milliseconds before an edit, and a stricter margin
would refuse a real edit && capture chain.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
1342 lines
73 KiB
TypeScript
1342 lines
73 KiB
TypeScript
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { CAPTURE_MS, CAPTURE_LONG_MS } from './helpers/eval-budgets';
|
|
import { runSkillTest, type SkillTestResult } from './helpers/session-runner';
|
|
import { OFFICE_HOURS_BUN_GRACE_MS, runRecordedOfficeHoursAttempt } from './helpers/office-hours-attempt';
|
|
import { resolveEvalModel } from '../lib/eval-model';
|
|
import { getProjectEvalDir } from './helpers/eval-store';
|
|
import { callJudge } from './helpers/llm-judge';
|
|
import {
|
|
ROOT, runId, evalsEnabled, selectedTests,
|
|
describeIfSelected, testConcurrentIfSelected,
|
|
copyDirSync, logCost, recordE2E,
|
|
createEvalCollector, finalizeEvalCollector, browseBin,
|
|
} from './helpers/e2e-helpers';
|
|
import { asideAvailable } from './helpers/aside-available';
|
|
import { installFakeImpeccable, DETECT_SAMPLE } from './helpers/fake-impeccable';
|
|
import { hermeticChildEnv } from './helpers/hermetic-env';
|
|
import { sliceBetween, extractDesignResearchContract } from './helpers/skill-fixture';
|
|
import { spawnSync } from 'child_process';
|
|
import * as fs from 'fs';
|
|
import * as path from 'path';
|
|
import * as os from 'os';
|
|
|
|
const evalCollector = createEvalCollector('e2e-design');
|
|
|
|
/**
|
|
* LLM judge for DESIGN.md quality — checks font blacklist compliance,
|
|
* coherence, specificity, and AI slop avoidance.
|
|
*/
|
|
async function designQualityJudge(designMd: string): Promise<{ passed: boolean; reasoning: string }> {
|
|
return callJudge<{ passed: boolean; reasoning: string }>(`You are evaluating a generated DESIGN.md file for quality.
|
|
|
|
Evaluate against these criteria — ALL must pass for an overall "passed: true":
|
|
1. Does NOT recommend Inter, Roboto, Arial, Helvetica, Open Sans, Lato, Montserrat, or Poppins as primary fonts
|
|
2. Aesthetic direction is coherent with color approach (e.g., brutalist aesthetic doesn't pair with expressive color without explanation)
|
|
3. Font recommendations include specific font names (not generic like "a sans-serif font")
|
|
4. Color palette includes actual hex values, not placeholders like "[hex]"
|
|
5. Rationale is provided for major decisions (not just "because it looks good")
|
|
6. No AI slop patterns: purple gradients mentioned positively, "3-column feature grid" language, generic marketing speak
|
|
7. Product context is reflected in design choices (civic tech → should have appropriate, professional aesthetic)
|
|
|
|
DESIGN.md content:
|
|
\`\`\`
|
|
${designMd}
|
|
\`\`\`
|
|
|
|
Return JSON: { "passed": true/false, "reasoning": "one paragraph explaining your evaluation" }`);
|
|
}
|
|
|
|
// --- Design Consultation E2E ---
|
|
|
|
describeIfSelected('Design Consultation E2E', [
|
|
'design-consultation-core',
|
|
'design-consultation-existing',
|
|
'design-consultation-research',
|
|
'design-consultation-preview',
|
|
], () => {
|
|
let designDir: string;
|
|
|
|
beforeAll(() => {
|
|
designDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-design-consultation-'));
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: designDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Create a realistic project context
|
|
fs.writeFileSync(path.join(designDir, 'README.md'), `# CivicPulse
|
|
|
|
A civic tech data platform for government employees to access, visualize, and share public data. Built with Next.js and PostgreSQL.
|
|
|
|
## Features
|
|
- Real-time data dashboards for municipal budgets
|
|
- Public records search with faceted filtering
|
|
- Data export and sharing tools for inter-department collaboration
|
|
`);
|
|
fs.writeFileSync(path.join(designDir, 'package.json'), JSON.stringify({
|
|
name: 'civicpulse',
|
|
version: '0.1.0',
|
|
dependencies: { next: '^14.0.0', react: '^18.2.0', 'tailwindcss': '^3.4.0' },
|
|
}, null, 2));
|
|
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial project setup']);
|
|
|
|
// Copy design-consultation skill — INCLUDING sections/. The skill has
|
|
// been carved since v1.57.0.0 (e722c5bf): Phases 3-6, where the DESIGN.md
|
|
// structure (the "AESTHETIC: [direction]" proposal template) is
|
|
// prescribed, live in sections/proposal-and-preview.md behind a STOP-read.
|
|
// Without the dir the agent improvises structure from the skeleton
|
|
// ("Visual thesis" vocabulary) and the section-synonym check becomes a
|
|
// coin flip (observed: CI run 33090283032 failed both attempts with
|
|
// "no sections dir" in the trace; the skeleton-only pass on 32899975845
|
|
// was lucky vocabulary).
|
|
fs.mkdirSync(path.join(designDir, 'design-consultation'), { recursive: true });
|
|
fs.copyFileSync(
|
|
path.join(ROOT, 'design-consultation', 'SKILL.md'),
|
|
path.join(designDir, 'design-consultation', 'SKILL.md'),
|
|
);
|
|
fs.cpSync(
|
|
path.join(ROOT, 'design-consultation', 'sections'),
|
|
path.join(designDir, 'design-consultation', 'sections'),
|
|
{ recursive: true },
|
|
);
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(designDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('design-consultation-core', async () => {
|
|
const result = await runSkillTest({
|
|
prompt: `Read design-consultation/SKILL.md for the design consultation workflow.
|
|
Skip the preamble bash block, lake intro, telemetry, and contributor mode sections — go straight to the design workflow.
|
|
|
|
This is a civic tech data platform called CivicPulse for government employees who need to access public data. Read the README.md for details.
|
|
|
|
Skip research — work from your design knowledge. Skip the font preview page. Skip any AskUserQuestion calls — this is non-interactive. Accept your first design system proposal.
|
|
|
|
Write DESIGN.md and CLAUDE.md (or update it) in the working directory.`,
|
|
workingDirectory: designDir,
|
|
maxTurns: 20,
|
|
timeout: CAPTURE_LONG_MS,
|
|
testName: 'design-consultation-core',
|
|
runId,
|
|
model: 'claude-opus-4-7',
|
|
});
|
|
|
|
logCost('/design-consultation core', result);
|
|
|
|
const designPath = path.join(designDir, 'DESIGN.md');
|
|
const claudePath = path.join(designDir, 'CLAUDE.md');
|
|
const designExists = fs.existsSync(designPath);
|
|
const claudeExists = fs.existsSync(claudePath);
|
|
let designContent = '';
|
|
|
|
if (designExists) {
|
|
designContent = fs.readFileSync(designPath, 'utf-8');
|
|
}
|
|
|
|
// Structural checks — fuzzy synonym matching to handle agent variation
|
|
const sectionSynonyms: Record<string, string[]> = {
|
|
'Product Context': ['product', 'context', 'overview', 'about'],
|
|
// Widened 2026-08-27: two CI runs produced judge-praised DESIGN.md files
|
|
// that articulated the direction as "design principles" / "design
|
|
// language" prose without any of the original four literals (run
|
|
// 33090283032, both attempts; inputs identical to the prior passing
|
|
// run 32899975845 — vocabulary variance, not a generation regression).
|
|
// Widened again 2026-09-08: the open DESIGN.md format's Overview opens with a
|
|
// "Creative North Star" and "Key characteristics" instead of an Aesthetic
|
|
// Direction heading (the judge passed both CI attempts on the vocabulary).
|
|
'Aesthetic': ['aesthetic', 'visual direction', 'design direction', 'visual identity', 'design language', 'visual language', 'design principle', 'look and feel', 'art direction', 'north star', 'key characteristics', '## overview'],
|
|
'Typography': ['typography', 'type', 'font', 'typeface'],
|
|
'Color': ['color', 'colour', 'palette', 'colors'],
|
|
'Spacing': ['spacing', 'space', 'whitespace', 'gap'],
|
|
'Layout': ['layout', 'grid', 'structure', 'composition'],
|
|
'Motion': ['motion', 'animation', 'transition', 'movement', 'easing', 'duration', 'micro-interaction'],
|
|
};
|
|
const missingSections = Object.entries(sectionSynonyms).filter(
|
|
([_, synonyms]) => !synonyms.some(s => designContent.toLowerCase().includes(s))
|
|
).map(([name]) => name);
|
|
|
|
// LLM judge for quality
|
|
let judgeResult = { passed: false, reasoning: 'judge not run' };
|
|
if (designExists && designContent.length > 100) {
|
|
try {
|
|
judgeResult = await designQualityJudge(designContent);
|
|
console.log('Design quality judge:', JSON.stringify(judgeResult, null, 2));
|
|
} catch (err) {
|
|
console.warn('Judge failed:', err);
|
|
judgeResult = { passed: true, reasoning: 'judge error — defaulting to pass' };
|
|
}
|
|
}
|
|
|
|
const structuralPass = designExists && claudeExists && missingSections.length === 0;
|
|
recordE2E(evalCollector, '/design-consultation core', 'Design Consultation E2E', result, {
|
|
passed: structuralPass && judgeResult.passed && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(designExists).toBe(true);
|
|
if (designExists) {
|
|
// join() so a failure names the offending section(s) — a bare
|
|
// toHaveLength(0) failure never prints WHICH synonym set missed.
|
|
expect(missingSections.join(', ')).toBe('');
|
|
}
|
|
if (claudeExists) {
|
|
const claude = fs.readFileSync(claudePath, 'utf-8');
|
|
expect(claude.toLowerCase()).toContain('design.md');
|
|
}
|
|
}, CAPTURE_LONG_MS);
|
|
|
|
testConcurrentIfSelected('design-consultation-research', async () => {
|
|
// Research phase only, no DESIGN.md generation. Web research runs in Aside
|
|
// first, WebSearch second ({{ASIDE_RESEARCH}}, rendered into
|
|
// design-consultation/SKILL.md). With Aside live the agent MUST search
|
|
// through `aside exec` — a straight-to-WebSearch run is the ordering bug
|
|
// this case pins. Without Aside (CI, or GSTACK_SKIP_ASIDE=1) it must use
|
|
// the WebSearch tool, or say the fallback sentence when that is missing
|
|
// too, and write the notes from in-distribution knowledge. Either way the
|
|
// notes file must exist.
|
|
const researchDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-research-'));
|
|
|
|
// Extract only the research contract (CLAUDE.md: extract, don't copy). The tree's
|
|
// SKILL.md unless GSTACK_E2E_DOCS_ROOT points at a `gen:skill-docs --out-dir` render.
|
|
const skill = fs.readFileSync(path.join(process.env.GSTACK_E2E_DOCS_ROOT || ROOT, 'design-consultation', 'SKILL.md'), 'utf-8');
|
|
fs.writeFileSync(path.join(researchDir, 'research-contract.md'), extractDesignResearchContract(skill));
|
|
const live = asideAvailable();
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `Read ${researchDir}/research-contract.md first and follow it exactly: it says how web research runs in this project.
|
|
|
|
Research civic tech data platform designs. Run exactly 2 research queries:
|
|
1. 'civic tech government data platform design 2025'
|
|
2. 'open data portal UX best practices'
|
|
|
|
Summarize the key design patterns you found to ${researchDir}/research-notes.md.
|
|
Include: color trends, typography patterns, and layout conventions you observed.
|
|
Do NOT generate a full DESIGN.md — just research notes.`,
|
|
workingDirectory: researchDir,
|
|
maxTurns: 10,
|
|
allowedTools: ['Bash', 'Read', 'Write', 'WebSearch'],
|
|
// 300s, not 90s: saturated-runner class (same as review-dashboard-via /
|
|
// retro-base-branch). PR #2533 CI observed the sibling preview test at
|
|
// 0 turns/$0.00 for 93s x3 attempts — session up, first completion
|
|
// queued past the budget under concurrent API load. 90s budgets cannot
|
|
// absorb one slow first completion; 300s is the repo's standard floor
|
|
// for CI SDK tests. Outer timeout below rises to 360s for headroom.
|
|
timeout: CAPTURE_MS,
|
|
testName: 'design-consultation-research',
|
|
runId,
|
|
});
|
|
|
|
logCost('/design-consultation research', result);
|
|
|
|
const notesPath = path.join(researchDir, 'research-notes.md');
|
|
const notesExist = fs.existsSync(notesPath);
|
|
const notesContent = notesExist ? fs.readFileSync(notesPath, 'utf-8') : '';
|
|
|
|
// Aside live: research went through `aside exec` in a Bash tool call (WebSearch
|
|
// alone is the wrong order). Aside absent: WebSearch tool, or the fallback sentence.
|
|
const asideExecCalls = result.toolCalls.filter(tc => tc.tool === 'Bash' && /\baside exec\b/.test(String(tc.input?.command ?? '')));
|
|
const webSearchCalls = result.toolCalls.filter(tc => tc.tool === 'WebSearch');
|
|
const searched = asideExecCalls.length > 0 || webSearchCalls.length > 0;
|
|
// Neither: the agent SAID the fallback. Assistant text blocks only — the
|
|
// contract file the agent Reads contains the same sentence, so tool_result
|
|
// content must not count.
|
|
const assistantText = result.transcript
|
|
.filter((e: any) => e?.type === 'assistant')
|
|
.flatMap((e: any) => (e.message?.content ?? []).filter((c: any) => c?.type === 'text').map((c: any) => String(c.text)))
|
|
.join('\n');
|
|
const saidFallback = assistantText.includes('Search unavailable');
|
|
const researchOk = live ? asideExecCalls.length > 0 : (searched || saidFallback);
|
|
console.log(`aside exec issued ${asideExecCalls.length} times; WebSearch called ${webSearchCalls.length} times; Aside live: ${live}; fallback said: ${saidFallback}`);
|
|
|
|
recordE2E(evalCollector, '/design-consultation research', 'Design Consultation E2E', result, {
|
|
passed: researchOk && notesExist && notesContent.length > 200 && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
if (live) expect(asideExecCalls.length).toBeGreaterThan(0);
|
|
else expect(searched || saidFallback).toBe(true);
|
|
expect(notesExist).toBe(true);
|
|
if (notesExist) {
|
|
expect(notesContent.length).toBeGreaterThan(200);
|
|
}
|
|
|
|
try { fs.rmSync(researchDir, { recursive: true, force: true }); } catch {}
|
|
}, CAPTURE_LONG_MS);
|
|
|
|
testConcurrentIfSelected('design-consultation-existing', async () => {
|
|
// Pre-create a LEGACY-format DESIGN.md (gstack's pre-spec shape, no marker) so
|
|
// Phase 0's format check has a real decision to make.
|
|
fs.writeFileSync(path.join(designDir, 'DESIGN.md'), `# Design System — CivicPulse
|
|
|
|
## Product Context
|
|
- **What this is:** Civic data platform
|
|
|
|
## Aesthetic Direction
|
|
- **Direction:** Industrial/Utilitarian
|
|
|
|
## Typography
|
|
- **Body:** system-ui
|
|
|
|
## Color
|
|
- **Primary:** #1D4ED8
|
|
`);
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `Read design-consultation/SKILL.md for the design consultation workflow.
|
|
|
|
There is already a DESIGN.md in this repo. Update it with a complete design system for CivicPulse, a civic tech data platform for government employees.
|
|
|
|
Run Phase 0's DESIGN.md format check exactly as written (the gstack bin directory is ${ROOT}/bin). Skip research. Skip font preview. Skip any AskUserQuestion calls — this is non-interactive: where the skill asks whether to convert the legacy file, take option A (convert) without asking.`,
|
|
workingDirectory: designDir,
|
|
maxTurns: 20,
|
|
timeout: CAPTURE_LONG_MS,
|
|
testName: 'design-consultation-existing',
|
|
runId,
|
|
model: 'claude-opus-4-7',
|
|
});
|
|
|
|
logCost('/design-consultation existing', result);
|
|
|
|
const designPath = path.join(designDir, 'DESIGN.md');
|
|
const designExists = fs.existsSync(designPath);
|
|
let designContent = '';
|
|
if (designExists) {
|
|
designContent = fs.readFileSync(designPath, 'utf-8');
|
|
}
|
|
|
|
// Should have more content than the minimal version
|
|
const hasColor = designContent.toLowerCase().includes('color');
|
|
const hasSpacing = designContent.toLowerCase().includes('spacing');
|
|
|
|
// Phase 0 format decision: the check ran, and the file left behind is either
|
|
// converted to the open format (marker on line 2) or explicitly kept legacy
|
|
// (marker on line 1). Either is the persisted-choice contract; "neither" is the bug.
|
|
const bash = result.toolCalls.filter(c => c.tool === 'Bash').map(c => String(c.input?.command ?? ''));
|
|
const ranCheck = bash.some(c => c.includes('gstack-design-md.ts check'));
|
|
const marked = /^---\n# gstack: design-md-format=spec/.test(designContent) || designContent.startsWith('<!-- gstack: design-md-format=legacy-keep -->');
|
|
console.log(`design-consultation-existing: ranCheck=${ranCheck} marked=${marked}`);
|
|
|
|
recordE2E(evalCollector, '/design-consultation existing', 'Design Consultation E2E', result, {
|
|
passed: designExists && hasColor && hasSpacing && ranCheck && marked && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(ranCheck).toBe(true);
|
|
expect(marked).toBe(true);
|
|
expect(designExists).toBe(true);
|
|
if (designExists) {
|
|
expect(hasColor).toBe(true);
|
|
expect(hasSpacing).toBe(true);
|
|
}
|
|
}, CAPTURE_LONG_MS);
|
|
|
|
testConcurrentIfSelected('design-consultation-preview', async () => {
|
|
// Test preview HTML generation only — no DESIGN.md (covered by core test)
|
|
const previewDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-preview-'));
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `Generate a font and color preview page for a civic tech data platform.
|
|
|
|
The design system uses:
|
|
- Primary font: Cabinet Grotesk (headings), Source Sans 3 (body)
|
|
- Colors: #1B4D8E (civic blue), #C4501A (alert orange), #2D6A4F (success green)
|
|
- Neutral: #F8F7F6 (warm white), #1A1A1A (near black)
|
|
|
|
Write a single HTML file to ${previewDir}/design-preview.html that shows:
|
|
- Font specimens for each font at different sizes
|
|
- Color swatches with hex values
|
|
- A light/dark toggle
|
|
Do NOT write DESIGN.md — only the preview HTML.`,
|
|
workingDirectory: previewDir,
|
|
maxTurns: 8,
|
|
// 300s, not 90s: this is the test that failed 3x at 0 turns/$0.00/93s
|
|
// on PR #2533 CI — see the research test's comment for the class.
|
|
timeout: CAPTURE_MS,
|
|
testName: 'design-consultation-preview',
|
|
runId,
|
|
});
|
|
|
|
logCost('/design-consultation preview', result);
|
|
|
|
const previewPath = path.join(previewDir, 'design-preview.html');
|
|
const previewExists = fs.existsSync(previewPath);
|
|
let previewContent = '';
|
|
if (previewExists) {
|
|
previewContent = fs.readFileSync(previewPath, 'utf-8');
|
|
}
|
|
|
|
const hasHtml = previewContent.includes('<html') || previewContent.includes('<!DOCTYPE');
|
|
const hasFontRef = previewContent.includes('font-family') || previewContent.includes('fonts.googleapis') || previewContent.includes('fonts.bunny');
|
|
|
|
recordE2E(evalCollector, '/design-consultation preview', 'Design Consultation E2E', result, {
|
|
passed: previewExists && hasHtml && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(previewExists).toBe(true);
|
|
if (previewExists) {
|
|
expect(hasHtml).toBe(true);
|
|
expect(hasFontRef).toBe(true);
|
|
}
|
|
|
|
try { fs.rmSync(previewDir, { recursive: true, force: true }); } catch {}
|
|
}, CAPTURE_LONG_MS);
|
|
});
|
|
|
|
// --- Plan Design Review E2E (plan-mode) ---
|
|
|
|
describeIfSelected('Plan Design Review E2E', ['plan-design-review-plan-mode', 'plan-design-review-no-ui-scope'], () => {
|
|
|
|
/** Create an isolated tmpdir with git repo and plan-design-review skill */
|
|
function setupReviewDir(): string {
|
|
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-plan-design-'));
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: dir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Copy plan-design-review skill
|
|
fs.mkdirSync(path.join(dir, 'plan-design-review'), { recursive: true });
|
|
fs.copyFileSync(
|
|
path.join(ROOT, 'plan-design-review', 'SKILL.md'),
|
|
path.join(dir, 'plan-design-review', 'SKILL.md'),
|
|
);
|
|
{ const _sec = path.join(ROOT, 'plan-design-review', 'sections'); if (fs.existsSync(_sec)) fs.cpSync(_sec, path.join(dir, 'plan-design-review', 'sections'), { recursive: true }); }
|
|
|
|
return dir;
|
|
}
|
|
|
|
testConcurrentIfSelected('plan-design-review-plan-mode', async () => {
|
|
let reviewDir: string | undefined;
|
|
let planBefore = '';
|
|
const cleanup = () => {
|
|
if (reviewDir) try { fs.rmSync(reviewDir, { recursive: true, force: true }); } catch {}
|
|
};
|
|
try {
|
|
await runRecordedOfficeHoursAttempt({
|
|
collector: evalCollector, name: '/plan-design-review plan-mode', suite: 'Plan Design Review E2E',
|
|
model: process.env.EVALS_MODEL ?? resolveEvalModel('capture'),
|
|
// Keep recording/cancellation grace inside the existing 600s Bun wall.
|
|
budgetMs: CAPTURE_LONG_MS - OFFICE_HOURS_BUN_GRACE_MS,
|
|
run: async signal => {
|
|
try {
|
|
reviewDir = setupReviewDir();
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: reviewDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
// Preserve the original complete seed for byte-level edit evidence.
|
|
fs.writeFileSync(path.join(reviewDir, 'plan.md'), `# Plan: User Dashboard
|
|
|
|
## Context
|
|
Build a user dashboard that shows account stats, recent activity, and settings.
|
|
|
|
## Implementation
|
|
1. Create a dashboard page at /dashboard
|
|
2. Show user stats (posts, followers, engagement rate)
|
|
3. Add a recent activity feed
|
|
4. Add a settings panel
|
|
5. Use a clean, modern UI with cards and icons
|
|
6. Add a hero section at the top with a gradient background
|
|
|
|
## Technical Details
|
|
- React components with Tailwind CSS
|
|
- API endpoint: GET /api/dashboard
|
|
- WebSocket for real-time activity updates
|
|
`);
|
|
planBefore = fs.readFileSync(path.join(reviewDir, 'plan.md'), 'utf8');
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial plan']);
|
|
signal.throwIfAborted();
|
|
return await runSkillTest({
|
|
prompt: `Read plan-design-review/SKILL.md for the design review workflow.
|
|
|
|
Review the plan in ./plan.md. Its design gaps are vague "clean, modern UI" and "cards and icons", a "hero section with gradient" (AI slop), and missing empty, error, loading, responsive, and accessibility behavior.
|
|
|
|
Use this non-interactive delivery sequence:
|
|
1. Skip the preamble bash block and any AskUserQuestion calls. Read every lazy section the workflow requires: in one response, natively Read plan-design-review/SKILL.md (Read the remainder with offset if the first Read stops early), plan-design-review/sections/review-sections.md (the one lazy section this review requires) and plan.md. No cat, sed, ls, manifest or git exploration is needed. Review all 7 design passes. Rate each scored design dimension 0-10 and explain what would make it a 10; preserve the unresolved-decisions pass and every required design decision.
|
|
2. EDIT plan.md with the missing design decisions (interaction state table, empty states, responsive behavior, etc.) and the full required review report. Keep the saved review compact: use the canonical tables and decision IDs. Specify each design requirement once; refer to its section or decision ID from other pass rationales, tasks, and report cells instead of repeating that specification. Give concise score rationales and 10/10 explanations. Retain all required report fields, design decisions, diagrams, ratings, and explanations. Draft your plan.md additions, including the report, to about 10,000 characters; this is a drafting target, not a check, so do not count characters or trim after saving.
|
|
3. Persist that complete plan and review with Write before publishing a completed walkthrough or saying a fix is applied. Read plan.md back to verify the saved changes.
|
|
4. Then return a brief, concrete summary of the design changes in at most ten lines; do not repeat the full review in the response. This changes presentation only: execute every required pass and lazy-section Read.
|
|
|
|
IMPORTANT: Do NOT try to browse any URLs or use a browse binary. This is a plan review, not a live site audit.`,
|
|
workingDirectory: reviewDir,
|
|
maxTurns: 15,
|
|
timeout: CAPTURE_MS,
|
|
publicStreamDiagnostics: true,
|
|
signal,
|
|
testName: 'plan-design-review-plan-mode',
|
|
runId,
|
|
});
|
|
} catch (error) {
|
|
cleanup();
|
|
throw error;
|
|
}
|
|
},
|
|
validate: result => {
|
|
try {
|
|
logCost('/plan-design-review plan-mode', result);
|
|
|
|
// Check that the agent produced design ratings (0-10 scale)
|
|
const output = result.output || '';
|
|
const hasRatings = /\d+\/10/.test(output);
|
|
|
|
// Check that the plan file was edited (the core new behavior)
|
|
const planAfter = fs.readFileSync(path.join(reviewDir, 'plan.md'), 'utf-8');
|
|
// A rated summary may leave the design terms to the plan it wrote
|
|
// (census 37179171083); a plan term counts only when the review added it.
|
|
const count = (text: string, term: string) => text.toLowerCase().split(term).length - 1;
|
|
const hasDesignContent = ['information architecture', 'interaction state', 'ai slop', 'hierarchy'].some(term =>
|
|
output.toLowerCase().includes(term) || (hasRatings && count(planAfter, term) > count(planBefore, term)));
|
|
const planWasEdited = planAfter !== planBefore && planAfter.length > 300;
|
|
const planHasDesignAdditions = planAfter.toLowerCase().includes('empty') ||
|
|
planAfter.toLowerCase().includes('loading') ||
|
|
planAfter.toLowerCase().includes('error') ||
|
|
planAfter.toLowerCase().includes('state') ||
|
|
planAfter.toLowerCase().includes('responsive') ||
|
|
planAfter.toLowerCase().includes('accessibility');
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
// Agent should produce design-relevant output about the plan
|
|
expect(hasDesignContent).toBe(true);
|
|
// Agent should have edited the plan file to add missing design decisions
|
|
expect(planWasEdited).toBe(true);
|
|
expect(planHasDesignAdditions).toBe(true);
|
|
|
|
} finally { cleanup(); }
|
|
},
|
|
});
|
|
} finally { cleanup(); }
|
|
}, CAPTURE_LONG_MS);
|
|
|
|
testConcurrentIfSelected('plan-design-review-no-ui-scope', async () => {
|
|
const reviewDir = setupReviewDir();
|
|
try {
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: reviewDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
// Write a backend-only plan
|
|
fs.writeFileSync(path.join(reviewDir, 'backend-plan.md'), `# Plan: Database Migration
|
|
|
|
## Context
|
|
Migrate user records from PostgreSQL to a new schema with better indexing.
|
|
|
|
## Implementation
|
|
1. Create migration to add new columns to users table
|
|
2. Backfill data from legacy columns
|
|
3. Add database indexes for common query patterns
|
|
4. Update ActiveRecord models
|
|
5. Run migration in staging first, then production
|
|
`);
|
|
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial plan']);
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `Read plan-design-review/SKILL.md for the design review workflow.
|
|
|
|
Review the plan in ./backend-plan.md. This is a pure backend database migration plan with no UI changes.
|
|
|
|
Skip the preamble bash block. Skip any AskUserQuestion calls — this is non-interactive. Write your findings directly to stdout.
|
|
|
|
IMPORTANT: Do NOT try to browse any URLs or use a browse binary. This is a plan review, not a live site audit.`,
|
|
workingDirectory: reviewDir,
|
|
maxTurns: 10,
|
|
timeout: CAPTURE_MS,
|
|
testName: 'plan-design-review-no-ui-scope',
|
|
runId,
|
|
});
|
|
|
|
logCost('/plan-design-review no-ui-scope', result);
|
|
|
|
// Agent should detect no UI scope and exit early
|
|
const output = result.output || '';
|
|
const detectsNoUI = output.toLowerCase().includes('no ui') ||
|
|
output.toLowerCase().includes('no frontend') ||
|
|
output.toLowerCase().includes('no design') ||
|
|
output.toLowerCase().includes('not applicable') ||
|
|
output.toLowerCase().includes('backend');
|
|
|
|
recordE2E(evalCollector, '/plan-design-review no-ui-scope', 'Plan Design Review E2E', result, {
|
|
passed: detectsNoUI && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(detectsNoUI).toBe(true);
|
|
} finally {
|
|
try { fs.rmSync(reviewDir, { recursive: true, force: true }); } catch {}
|
|
}
|
|
}, CAPTURE_MS);
|
|
});
|
|
|
|
// --- Design Review E2E (live-site audit + fix) ---
|
|
|
|
/**
|
|
* Concatenated tool_result text from the stream-json transcript. runSkillTest
|
|
* leaves toolCalls[].output empty, and the agent's Bash INPUT also contains
|
|
* the sentinel string — only the tool_result proves the script printed it.
|
|
*/
|
|
function toolOutput(result: SkillTestResult): string {
|
|
const parts: string[] = [];
|
|
for (const e of result.transcript) {
|
|
if (e?.type !== 'user') continue;
|
|
for (const item of e.message?.content ?? []) {
|
|
if (item?.type !== 'tool_result') continue;
|
|
parts.push(typeof item.content === 'string' ? item.content : JSON.stringify(item.content ?? ''));
|
|
}
|
|
}
|
|
return parts.join('\n');
|
|
}
|
|
|
|
// /design-review drives the Aside browser; without it the skill's BROWSER SETUP stops at
|
|
// NEEDS_ASIDE, so the block self-skips (CI runners have no Aside).
|
|
describeIfSelected('Design Review E2E', ['design-review-fix'], () => {
|
|
// bun runs describe.skip callbacks too — probe only when this block is actually selected,
|
|
// so an unrelated eval run never pays the up-to-30s `aside repl` probe.
|
|
const selected = evalsEnabled && (selectedTests === null || selectedTests.includes('design-review-fix'));
|
|
if (selected && !asideAvailable()) { test.skip('needs Aside', () => {}); return; }
|
|
|
|
let qaDesignDir: string;
|
|
let qaDesignServer: ReturnType<typeof Bun.serve> | null = null;
|
|
|
|
beforeAll(() => {
|
|
qaDesignDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-qa-design-'));
|
|
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: qaDesignDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Create HTML/CSS with intentional design issues
|
|
fs.writeFileSync(path.join(qaDesignDir, 'index.html'), `<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<title>Design Test App</title>
|
|
<link rel="stylesheet" href="style.css">
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<h1 style="font-size: 48px; color: #333;">Welcome</h1>
|
|
<h2 style="font-size: 47px; color: #334;">Subtitle Here</h2>
|
|
</header>
|
|
<main>
|
|
<div class="card" style="padding: 10px; margin: 20px;">
|
|
<h3 style="color: blue;">Card Title</h3>
|
|
<p style="color: #666; font-size: 14px; line-height: 1.2;">Some content here with tight line height.</p>
|
|
</div>
|
|
<div class="card" style="padding: 30px; margin: 5px;">
|
|
<h3 style="color: green;">Another Card</h3>
|
|
<p style="color: #999; font-size: 16px;">Different spacing and colors for no reason.</p>
|
|
</div>
|
|
<button style="background: red; color: white; padding: 5px 10px; border: none;">Click Me</button>
|
|
<button style="background: #007bff; color: white; padding: 12px 24px; border: none; border-radius: 20px;">Also Click</button>
|
|
</main>
|
|
</body>
|
|
</html>`);
|
|
|
|
fs.writeFileSync(path.join(qaDesignDir, 'style.css'), `body {
|
|
font-family: Arial, sans-serif;
|
|
margin: 0;
|
|
padding: 20px;
|
|
}
|
|
.card {
|
|
border: 1px solid #ddd;
|
|
border-radius: 4px;
|
|
}
|
|
`);
|
|
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial design test page']);
|
|
|
|
// Start a simple file server for the design test page
|
|
qaDesignServer = Bun.serve({
|
|
port: 0,
|
|
fetch(req) {
|
|
const url = new URL(req.url);
|
|
const filePath = path.join(qaDesignDir, url.pathname === '/' ? 'index.html' : url.pathname.slice(1));
|
|
try {
|
|
const content = fs.readFileSync(filePath);
|
|
const ext = path.extname(filePath);
|
|
const contentType = ext === '.css' ? 'text/css' : ext === '.html' ? 'text/html' : 'text/plain';
|
|
return new Response(content, { headers: { 'Content-Type': contentType } });
|
|
} catch {
|
|
return new Response('Not Found', { status: 404 });
|
|
}
|
|
},
|
|
});
|
|
|
|
// Copy design-review skill
|
|
fs.mkdirSync(path.join(qaDesignDir, 'design-review'), { recursive: true });
|
|
fs.copyFileSync(
|
|
path.join(ROOT, 'design-review', 'SKILL.md'),
|
|
path.join(qaDesignDir, 'design-review', 'SKILL.md'),
|
|
);
|
|
});
|
|
|
|
afterAll(() => {
|
|
qaDesignServer?.stop();
|
|
try { fs.rmSync(qaDesignDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('design-review-fix', async () => {
|
|
const serverUrl = `http://localhost:${(qaDesignServer as any)?.port}`;
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `The Aside browser is installed and running. Read design-review/SKILL.md for the design review + fix workflow and follow its BROWSER SETUP section: drive the browser with \`aside repl\` scripts shaped exactly like its cookbook. Do not look for any other browser binary.
|
|
|
|
Review the site at ${serverUrl}. Use --quick mode. Skip any AskUserQuestion calls — this is non-interactive. Fix up to 3 issues max. Write your report to ./design-audit.md.`,
|
|
workingDirectory: qaDesignDir,
|
|
maxTurns: 30,
|
|
timeout: CAPTURE_LONG_MS,
|
|
testName: 'design-review-fix',
|
|
runId,
|
|
});
|
|
|
|
logCost('/design-review fix', result);
|
|
|
|
const reportPath = path.join(qaDesignDir, 'design-audit.md');
|
|
const reportExists = fs.existsSync(reportPath);
|
|
|
|
// Outcome: the report names a seeded defect, and the page changed (committed
|
|
// or not) relative to the initial fixture commit. The commit prefix is not required.
|
|
const report = reportExists ? fs.readFileSync(reportPath, 'utf-8') : '';
|
|
const namesSeededDefect = /4[78]px|line-height|border-radius|radius|padding|spacing|heading/i.test(report);
|
|
const initialCommit = spawnSync('git', ['rev-list', '--max-parents=0', 'HEAD'], {
|
|
cwd: qaDesignDir, stdio: 'pipe', timeout: 30_000,
|
|
}).stdout.toString().trim();
|
|
const pageChanged = spawnSync('git', ['diff', '--name-only', initialCommit, '--', 'index.html', 'style.css'], {
|
|
cwd: qaDesignDir, stdio: 'pipe', timeout: 30_000,
|
|
}).stdout.toString().trim() !== '';
|
|
|
|
// The agent must actually drive Aside: an `aside repl` Bash call, a printed sentinel
|
|
// (from a tool_result, never the input), and no reach for the retired browse binary.
|
|
const bashCommands = result.toolCalls
|
|
.filter(t => t.tool === 'Bash')
|
|
.map(t => String(t.input?.command ?? ''));
|
|
const droveAside = bashCommands.some(c => /aside repl/.test(c));
|
|
const sentinelPrinted = /GSTACK_STEP_OK/.test(toolOutput(result));
|
|
const usedBrowseBin = bashCommands.some(c => /browse\/dist\/browse|\$B /.test(c));
|
|
|
|
recordE2E(evalCollector, '/design-review fix', 'Design Review E2E', result, {
|
|
passed: ['success', 'error_max_turns'].includes(result.exitReason) && droveAside && sentinelPrinted && !usedBrowseBin
|
|
&& reportExists && namesSeededDefect && pageChanged,
|
|
});
|
|
|
|
// Accept error_max_turns — the fix loop is complex
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(droveAside).toBe(true);
|
|
expect(sentinelPrinted).toBe(true);
|
|
expect(usedBrowseBin).toBe(false);
|
|
expect(reportExists, 'design-audit.md must be written').toBe(true);
|
|
expect(namesSeededDefect, 'the report names a seeded design defect').toBe(true);
|
|
expect(pageChanged, 'at least one design issue is fixed in index.html or style.css').toBe(true);
|
|
}, CAPTURE_LONG_MS);
|
|
});
|
|
|
|
// Module-level afterAll — finalize eval collector after all tests complete
|
|
afterAll(async () => {
|
|
await finalizeEvalCollector(evalCollector);
|
|
});
|
|
|
|
// --- Design detector (impeccable engine shim) E2E ---
|
|
//
|
|
// The user-installed impeccable engine is stood in for by test/fixtures/
|
|
// fake-impeccable.ts (prints the captured detect --json sample, exit 2),
|
|
// reached through IMPECCABLE_BIN from OUTSIDE the temp repo (the wrapper
|
|
// ignores an in-repo IMPECCABLE_BIN by design). The skill text the agent reads
|
|
// is the extracted Setup detector block + Phase 0 (+ the Phase 3 DOM-dump
|
|
// section for the DOM case), never the 1,500-line SKILL.md, with the installed
|
|
// bin path pointed at THIS checkout so the test does not depend on ~/.claude.
|
|
|
|
|
|
/** design-review's detector prose with the installed bin/lib paths rewritten to this checkout. */
|
|
function detectorSkillText(sections: Array<[string, string]>): string {
|
|
const full = fs.readFileSync(path.join(ROOT, 'design-review', 'SKILL.md'), 'utf-8');
|
|
return sections.map(([a, b]) => sliceBetween(full, a, b)).join('\n\n---\n\n')
|
|
.replaceAll('$HOME/.claude/skills/gstack', ROOT)
|
|
.replaceAll('~/.claude/skills/gstack', ROOT);
|
|
}
|
|
|
|
function makeFakeEngine(): string {
|
|
return installFakeImpeccable('skill-e2e-fake-impeccable-').dir;
|
|
}
|
|
|
|
function detectorReportEntries(report: string): string[] {
|
|
return report.split(/(?=^[\t ]*(?:#{1,6}\s+|[-*|]\s*|\d+[.)]\s+)?(?:\*\*|`)?FINDING-\d+)/m);
|
|
}
|
|
|
|
const INSTALL_OR_OVERRIDE = /\bnpx\b|gstack-design-detect\.ts install|\b(?:curl|wget|npm install|bun add)\b|IMPECCABLE_BIN\s*=/;
|
|
|
|
/** A quoted-delimiter heredoc body is literal data, so a report that says
|
|
* "no npx" is not an npx run. Unquoted bodies still expand and stay checked;
|
|
* an unterminated body keeps the whole command checked. */
|
|
function executedShellText(command: string): string {
|
|
const kept: string[] = [];
|
|
let delimiter: string | undefined, tabs = false;
|
|
for (const line of command.split('\n')) {
|
|
if (delimiter !== undefined) {
|
|
if ((tabs ? line.replace(/^\t*/, '') : line) === delimiter) delimiter = undefined;
|
|
continue;
|
|
}
|
|
kept.push(line);
|
|
const quoted = /<<(-)?[ \t]*(?:'([^'\n]+)'|"([^"\n]+)"|\\(\w+))/.exec(line);
|
|
if (quoted) { delimiter = quoted[2] ?? quoted[3] ?? quoted[4]; tabs = !!quoted[1]; }
|
|
}
|
|
return delimiter === undefined ? kept.join('\n') : command;
|
|
}
|
|
|
|
function commandRunsInstallOrOverride(command: string): boolean {
|
|
return INSTALL_OR_OVERRIDE.test(executedShellText(command));
|
|
}
|
|
|
|
const SOURCE_SCAN = /gstack-design-detect\.ts scan --changed (?:main|'main'|"main"|"?\$\{?\w+\}?"?)(?=[\s;]|$)/;
|
|
|
|
/** CI 37111582659 (6fb8ea3d): the model resolved the base through the excerpt's gh fallback and
|
|
* scanned `--changed "$_BASE"`. The command proves the detector scan ran; the engine's single
|
|
* invocation over exactly the files changed vs main proves its base resolved to main. */
|
|
function sourceScanAgainstMain(commands: string[], invocations: string[], repoDir: string): boolean {
|
|
if (!commands.some(command => SOURCE_SCAN.test(executedShellText(command))) || invocations.length !== 1) return false;
|
|
const argv = (JSON.parse(invocations[0]!) as { argv?: unknown }).argv;
|
|
if (!Array.isArray(argv) || argv[0] !== 'detect' || argv[1] !== '--json') return false;
|
|
const root = fs.realpathSync(repoDir);
|
|
const targets = argv.slice(2).map(target => fs.existsSync(String(target)) ? path.relative(root, fs.realpathSync(String(target))) : '');
|
|
return targets.join() === 'index.html,styles.css';
|
|
}
|
|
|
|
/** CI 38980005793: the agent ran the generated DOM block, which scans `"$_REPORT/dom/$_RUN"` with `_RUN="run1"`
|
|
* assigned in the same command. A DOM scan counts when one of its own arguments, after that command's literal
|
|
* `_REPORT=`/`_RUN=` assignments are substituted, is the run directory; `scan --changed` (source mode), a scan of
|
|
* another directory, quoted heredoc text and a non-literal or conflicting assignment never count. */
|
|
function domScanOfRun(command: string, runDir: string): boolean {
|
|
const shell = executedShellText(command);
|
|
const assigned = new Map<string, Set<string>>();
|
|
for (const m of shell.matchAll(/(?<![\w$])(_REPORT|_RUN)=(?:"([^"$`\\]*)"|'([^']*)'|([^\s;&|<>()"'$`\\]*))(?=[\s;&|)]|$)/g)) {
|
|
assigned.set(m[1]!, (assigned.get(m[1]!) ?? new Set()).add(m[2] ?? m[3] ?? m[4] ?? ''));
|
|
}
|
|
const resolved = shell.replace(/\$(?:\{(_REPORT|_RUN)\}|(_REPORT|_RUN)(?!\w))/g, (all, braced?: string, bare?: string) => {
|
|
const values = assigned.get((braced ?? bare)!);
|
|
return values?.size === 1 ? [...values][0]! : all;
|
|
});
|
|
return [...resolved.matchAll(/gstack-design-detect\.ts scan ([^\n;&|<>]*)/g)].some(scan => {
|
|
const args = (scan[1]!.match(/"[^"]*"|'[^']*'|\S+/g) ?? []).map(arg => arg.replace(/^(["'])(.*)\1$/, '$2').replace(/\/+$/, ''));
|
|
return !args.includes('--changed') && args.includes(runDir);
|
|
});
|
|
}
|
|
|
|
if (!evalsEnabled) test('plugin handoff credits an executed source scan whose base resolved to main', () => {
|
|
const repoDir = fs.mkdtempSync(path.join(os.tmpdir(), 'plugin-scan-replay-'));
|
|
try {
|
|
for (const file of ['index.html', 'styles.css', 'other.css']) fs.writeFileSync(path.join(repoDir, file), '');
|
|
const engine = (...files: string[]) => JSON.stringify({ argv: ['detect', '--json', ...files.map(file => path.join(repoDir, file))], cwd: repoDir, stdinIsTTY: false });
|
|
const detect = '/__w/gstack/gstack/bin/gstack-design-detect.ts';
|
|
// Captured CI 37111582659 command (paths shortened): gh fallback resolved _BASE=main.
|
|
const captured = `_BASE=$(gh pr view --json baseRefName -q .baseRefName 2>/dev/null || true); if [ -z "$_BASE" ]; then _BASE=main; fi; git diff --name-only "$_BASE"...HEAD; _DJ=$(mktemp); bun --no-env-file run ${detect} scan --changed "$_BASE" --format gstack --host claude > "$_DJ" 2>/tmp/scan-stderr.txt; echo "DETECT_EXIT_CODE=$?"`;
|
|
const literal = `_DJ=$(mktemp); bun --no-env-file run ${detect} scan --changed main --format gstack --host claude > "$_DJ"; echo "DETECT_EXIT_CODE=$?"`;
|
|
const reportOnly = `cat > detector-output.md <<'EOF'\nScan: \`bun --no-env-file run ${detect} scan --changed main --format gstack --host claude\`\nEOF`;
|
|
expect(sourceScanAgainstMain([captured], [engine('index.html', 'styles.css')], repoDir)).toBe(true);
|
|
expect(sourceScanAgainstMain([literal, reportOnly], [engine('index.html', 'styles.css')], repoDir)).toBe(true);
|
|
expect(sourceScanAgainstMain([literal.replace('--changed main', "--changed 'main'")], [engine('index.html', 'styles.css')], repoDir)).toBe(true);
|
|
// Negative controls: quoted report text, another literal base, a base that resolved elsewhere, no or extra engine runs, a direct engine call.
|
|
expect(sourceScanAgainstMain([reportOnly], [engine('index.html', 'styles.css')], repoDir)).toBe(false);
|
|
expect(sourceScanAgainstMain([literal.replace('--changed main', '--changed HEAD~1')], [engine('index.html', 'styles.css')], repoDir)).toBe(false);
|
|
expect(sourceScanAgainstMain([captured], [engine('styles.css')], repoDir)).toBe(false);
|
|
expect(sourceScanAgainstMain([captured], [engine('index.html', 'other.css', 'styles.css')], repoDir)).toBe(false);
|
|
expect(sourceScanAgainstMain([captured], [], repoDir)).toBe(false);
|
|
expect(sourceScanAgainstMain([captured], [engine('index.html', 'styles.css'), engine('index.html', 'styles.css')], repoDir)).toBe(false);
|
|
expect(sourceScanAgainstMain(['impeccable detect --json index.html styles.css'], [engine('index.html', 'styles.css')], repoDir)).toBe(false);
|
|
expect(sourceScanAgainstMain([literal.replace('--changed main', '--changed mainline')], [engine('index.html', 'styles.css')], repoDir)).toBe(false);
|
|
} finally {
|
|
fs.rmSync(repoDir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
if (!evalsEnabled) test('plugin handoff counts executed install commands, not quoted report text', () => {
|
|
// PR lane 36794871032: the report heredoc said "no `npx impeccable`" and failed noInstallOrOverride.
|
|
const reportWrite = "cat > detector-output.md <<'EOF'\n# Detector output\n- No install, no launcher, no `npx impeccable`, no Impeccable skill files read.\nEOF\necho \"written: $(wc -l < detector-output.md) lines\"; git status --short";
|
|
expect(commandRunsInstallOrOverride(reportWrite)).toBe(false);
|
|
expect(commandRunsInstallOrOverride(reportWrite.replace("<<'EOF'", '<<"EOF"'))).toBe(false);
|
|
expect(commandRunsInstallOrOverride(reportWrite.replace("<<'EOF'", '<<\\EOF'))).toBe(false);
|
|
expect(commandRunsInstallOrOverride(reportWrite.replace("<<'EOF'", "<<-'EOF'").replace('\nEOF\n', '\n\t\tEOF\n'))).toBe(false);
|
|
expect(commandRunsInstallOrOverride('npx impeccable detect .')).toBe(true);
|
|
expect(commandRunsInstallOrOverride('IMPECCABLE_BIN=/tmp/x bun run detect.ts probe')).toBe(true);
|
|
expect(commandRunsInstallOrOverride(reportWrite.replace("<<'EOF'", '<<EOF').replace('`npx impeccable`', '$(npx impeccable)'))).toBe(true);
|
|
expect(commandRunsInstallOrOverride(reportWrite + '\nnpx impeccable detect .')).toBe(true);
|
|
expect(commandRunsInstallOrOverride(reportWrite.replace('\nEOF\n', '\nEO\n'))).toBe(true);
|
|
});
|
|
|
|
if (!evalsEnabled) test('detector report handoffs stay with their entry across inline cross-references', () => {
|
|
const report = `### FINDING-001 \`[low-contrast]\` — impact=high — DEFERRED
|
|
handoff=\`/impeccable colorize\`
|
|
|
|
### FINDING-002 \`[ai-color-palette]\` — impact=medium — DEFERRED
|
|
The colors also appear in FINDING-001. This is unconfirmed static evidence.
|
|
handoff=\`/impeccable colorize\`
|
|
|
|
### FINDING-003 \`[skipped-heading]\` — impact=medium — DEFERRED
|
|
handoff=\`/impeccable typeset\`
|
|
`;
|
|
const hasPaletteHandoff = (text: string) => detectorReportEntries(text).some(entry => entry.includes('[ai-color-palette]') && entry.includes('/impeccable colorize'));
|
|
expect(hasPaletteHandoff(report)).toBe(true);
|
|
expect(hasPaletteHandoff(report.replace('handoff=`/impeccable colorize`\n\n### FINDING-003', '### FINDING-003'))).toBe(false);
|
|
for (const marker of ['.', ')']) {
|
|
const numbered = report.replace(/^### FINDING-(\d+)/gm, (_, number) => `${Number(number)}${marker} **FINDING-${number}**`);
|
|
expect(hasPaletteHandoff(numbered)).toBe(true);
|
|
expect(hasPaletteHandoff(numbered.replace(`handoff=\`/impeccable colorize\`\n\n3${marker}`, `3${marker}`))).toBe(false);
|
|
}
|
|
});
|
|
|
|
function pluginDetectorFixture() {
|
|
const fixture = installFakeImpeccable('skill-e2e-plugin-');
|
|
const repoDir = path.join(fixture.dir, 'repo');
|
|
const home = path.join(fixture.dir, 'home');
|
|
const configDir = path.join(fixture.dir, 'custom-claude');
|
|
const gstackHome = path.join(fixture.dir, 'gstack');
|
|
const impeccableHome = path.join(fixture.dir, 'engine-cache');
|
|
for (const dir of [repoDir, home, configDir, gstackHome, impeccableHome]) fs.mkdirSync(dir);
|
|
const env = hermeticChildEnv({
|
|
HOME: home, USERPROFILE: home, CLAUDE_CONFIG_DIR: configDir,
|
|
GSTACK_HOME: gstackHome, IMPECCABLE_HOME: impeccableHome,
|
|
CLAUDE_PLUGIN_DATA: '', GSTACK_HEADLESS: '1',
|
|
});
|
|
for (const key of Object.keys(env)) if (key.startsWith('IMPECCABLE_') && key !== 'IMPECCABLE_HOME') delete env[key];
|
|
const engines: Record<string, string> = {};
|
|
for (const version of ['4.3.1', '4.10.0']) {
|
|
const skillDir = path.join(configDir, 'plugins/cache/fixture-market/impeccable', version, 'skills/impeccable');
|
|
const scripts = path.join(skillDir, 'scripts');
|
|
const engineDir = path.join(scripts, 'bin', `${process.platform}-${process.arch}`);
|
|
fs.mkdirSync(engineDir, { recursive: true });
|
|
fs.writeFileSync(path.join(skillDir, 'SKILL.md'), '# Impeccable fixture marker\n');
|
|
fs.writeFileSync(path.join(scripts, 'impeccable'), `#!/bin/sh\nprintf launcher > '${fixture.dir}/launcher-ran'\nexit 99\n`, { mode: 0o755 });
|
|
fs.writeFileSync(path.join(scripts, 'VERSION'), '1.6.0\n');
|
|
engines[version] = path.join(engineDir, 'impeccable');
|
|
fs.writeFileSync(engines[version], `#!/bin/sh\nIMPECCABLE_FAKE_LOG='${fixture.dir}/${version}.jsonl' exec '${process.execPath}' '${fixture.bin}' "$@"\n`, { mode: 0o755 });
|
|
engines[version] = fs.realpathSync(engines[version]);
|
|
}
|
|
const git = (...args: string[]) => {
|
|
const result = spawnSync('git', args, { cwd: repoDir, env, encoding: 'utf-8', timeout: 5000 });
|
|
if (result.status !== 0) throw new Error(`Plugin fixture git ${args[0]} failed: ${result.stderr}`);
|
|
};
|
|
git('init', '-b', 'main');
|
|
git('config', 'user.email', 'test@test.com');
|
|
git('config', 'user.name', 'Test');
|
|
fs.writeFileSync(path.join(repoDir, 'index.html'), '<h1>Clean</h1>\n');
|
|
git('add', '.');
|
|
git('commit', '-m', 'initial');
|
|
git('checkout', '-b', 'feature/landing');
|
|
fs.copyFileSync(path.join(ROOT, 'test/fixtures/review-eval-design-slop.html'), path.join(repoDir, 'index.html'));
|
|
fs.copyFileSync(path.join(ROOT, 'test/fixtures/review-eval-design-slop.css'), path.join(repoDir, 'styles.css'));
|
|
git('add', '.');
|
|
git('commit', '-m', 'landing page');
|
|
// The engine reports this repository's files and lines, so its evidence is checkable where the page links it.
|
|
const located = (file: string, needle: string) => ({ file, line: fs.readFileSync(path.join(repoDir, file), 'utf-8').split('\n').findIndex(text => text.includes(needle)) + 1 });
|
|
const sample = path.join(fixture.dir, 'impeccable-detect-sample.json');
|
|
fs.writeFileSync(sample, JSON.stringify((JSON.parse(fs.readFileSync(sample, 'utf-8')) as Array<Record<string, unknown>>).map(finding => {
|
|
const snippet = String(finding.snippet);
|
|
const where = finding.antipattern === 'skipped-heading' ? located('index.html', 'Feature One')
|
|
: finding.antipattern === 'marketing-buzzword' ? located('index.html', 'streamline')
|
|
: located('styles.css', /on (#[0-9a-f]{6})/.exec(snippet)?.[1] ?? '#8b5cf6');
|
|
return { ...finding, ...where };
|
|
}), null, 2));
|
|
fs.writeFileSync(path.join(repoDir, 'design-review-detector.md'), detectorSkillText([
|
|
['**Design detector (optional, deterministic):**', '**Create output directories:**'],
|
|
['**Phase 0: mechanical scan**', '## Phases 1-6'],
|
|
]));
|
|
return { dir: fixture.dir, repoDir, env, engines };
|
|
}
|
|
|
|
if (!evalsEnabled) test('plugin detector fixture discovers the selected engine without an override', () => {
|
|
const fixture = pluginDetectorFixture();
|
|
try {
|
|
expect(fixture.env.IMPECCABLE_BIN).toBeUndefined();
|
|
const probe = spawnSync(process.execPath, ['--no-env-file', 'run', path.join(ROOT, 'bin/gstack-design-detect.ts'), 'probe', '--host', 'claude'], {
|
|
cwd: fixture.repoDir, env: fixture.env, encoding: 'utf-8', timeout: 10000,
|
|
});
|
|
expect(probe.status).toBe(0);
|
|
expect(probe.stdout).toContain(`IMPECCABLE_READY: ${fixture.engines['4.10.0']}`);
|
|
expect(probe.stdout).toContain('IMPECCABLE_SKILL: present');
|
|
const scan = spawnSync(process.execPath, ['--no-env-file', 'run', path.join(ROOT, 'bin/gstack-design-detect.ts'), 'scan', '--changed', 'main', '--host', 'claude'], {
|
|
cwd: fixture.repoDir, env: fixture.env, encoding: 'utf-8', timeout: 10000,
|
|
});
|
|
expect(scan.status).toBe(2);
|
|
expect(scan.stderr).toContain('handoff=/impeccable colorize');
|
|
// Census 36709485593: rows naming test/fixtures/... paths absent from this repo cost four
|
|
// reconciliation turns and exceeded max turns. Every row must cite a repo file:line holding its evidence.
|
|
const rows = [...scan.stderr.matchAll(/^ {2}(\S+):(\d+) {2}(.*)$/gm)];
|
|
expect(rows).toHaveLength(6);
|
|
for (const [, file, line, snippet] of rows) {
|
|
const text = fs.readFileSync(path.join(fixture.repoDir, file!), 'utf-8').split('\n')[Number(line) - 1]!;
|
|
const evidence = /on (#[0-9a-f]{6})/.exec(snippet!)?.[1] ?? (/Purple/.test(snippet!) ? '#8b5cf6' : /buzzword/.test(snippet!) ? 'streamline' : 'Feature One');
|
|
expect(text, `${file}:${line}`).toContain(evidence);
|
|
}
|
|
const shipped = JSON.parse(fs.readFileSync(DETECT_SAMPLE, 'utf-8')) as Array<{ file: string }>;
|
|
expect(shipped.every(finding => !fs.existsSync(path.join(fixture.repoDir, finding.file)))).toBe(true);
|
|
expect(fs.existsSync(path.join(fixture.dir, '4.10.0.jsonl'))).toBe(true);
|
|
expect(fs.existsSync(path.join(fixture.dir, '4.3.1.jsonl'))).toBe(false);
|
|
expect(fs.existsSync(path.join(fixture.dir, 'launcher-ran'))).toBe(false);
|
|
} finally {
|
|
fs.rmSync(fixture.dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
describeIfSelected('Design review plugin discovery E2E', ['design-review-plugin-handoff'], () => {
|
|
testConcurrentIfSelected('design-review-plugin-handoff', async () => {
|
|
const fixture = pluginDetectorFixture();
|
|
try {
|
|
const result = await runSkillTest({
|
|
prompt: `Load gstack's /design-review workflow by reading design-review-detector.md, the actual Setup detector and Phase 0 excerpt.
|
|
Supported actor scope: read that excerpt, run its detector probe and one source-mode scan, then write detector-output.md. This isolated repository is on feature/landing; its base is main. There is no URL.
|
|
Do not install or download anything, execute a launcher, change environment variables, browse, ask questions, spawn agents, or edit anything except detector-output.md. Do not read Impeccable skill files. If discovery fails, report that failure and stop; do not repair the environment.
|
|
Write the probe's first line and skill-presence line, then one FINDING-NNN entry per DETECT_TOP rule with its [rule-id] and impact. All findings are deferred, unconfirmed static evidence because rendered-page confirmation is outside this actor's scope. Apply the excerpt's deferred-finding reporting requirements. Do not fix source files or claim visual verification.`,
|
|
workingDirectory: fixture.repoDir,
|
|
maxTurns: 10,
|
|
timeout: CAPTURE_MS,
|
|
testName: 'design-review-plugin-handoff',
|
|
runId,
|
|
tools: ['Bash', 'Read', 'Write'],
|
|
env: fixture.env,
|
|
});
|
|
logCost('/design-review plugin handoff', result);
|
|
const reportPath = path.join(fixture.repoDir, 'detector-output.md');
|
|
const report = fs.existsSync(reportPath) ? fs.readFileSync(reportPath, 'utf-8') : '';
|
|
const outputs = result.toolCalls.map(call => String(call.output ?? '')).join('\n');
|
|
const commands = result.toolCalls.filter(call => call.tool === 'Bash').map(call => String(call.input?.command ?? ''));
|
|
const handoffs = [...outputs.matchAll(/\[([\w-]+)\] impact=(\w+)[^\n]*handoff=(\/impeccable \w+)/g)];
|
|
const entries = detectorReportEntries(report);
|
|
const engineLog = path.join(fixture.dir, '4.10.0.jsonl');
|
|
const invocations = fs.existsSync(engineLog) ? fs.readFileSync(engineLog, 'utf-8').trim().split('\n') : [];
|
|
const sourceUnchanged = fs.readFileSync(path.join(fixture.repoDir, 'index.html'), 'utf-8') === fs.readFileSync(path.join(ROOT, 'test/fixtures/review-eval-design-slop.html'), 'utf-8');
|
|
if (process.env.GSTACK_EVAL_DIR) {
|
|
const evidenceDir = path.join(process.env.GSTACK_EVAL_DIR, 'plugin-handoff', `${Date.now()}`);
|
|
fs.mkdirSync(evidenceDir, { recursive: true, mode: 0o700 });
|
|
for (const [name, text] of Object.entries({
|
|
'report.md': report,
|
|
'skill-excerpt.md': fs.readFileSync(path.join(fixture.repoDir, 'design-review-detector.md'), 'utf-8'),
|
|
'engine-invocations.jsonl': invocations.join('\n'),
|
|
})) fs.writeFileSync(path.join(evidenceDir, name), text, { mode: 0o600 });
|
|
}
|
|
const checks = {
|
|
success: result.exitReason === 'success',
|
|
selectedEngine: outputs.includes(`IMPECCABLE_READY: ${fixture.engines['4.10.0']}`),
|
|
skillPresent: outputs.includes('IMPECCABLE_SKILL: present'),
|
|
probeReported: report.includes(`IMPECCABLE_READY: ${fixture.engines['4.10.0']}`) && report.includes('IMPECCABLE_SKILL: present'),
|
|
probeExecuted: commands.some(command => /gstack-design-detect\.ts probe/.test(command)),
|
|
scanExecuted: sourceScanAgainstMain(commands, invocations, fixture.repoDir),
|
|
noInstallOrOverride: !commands.some(commandRunsInstallOrOverride),
|
|
oneNewEngineInvocation: invocations.length === 1,
|
|
oldEngineNotExecuted: !fs.existsSync(path.join(fixture.dir, '4.3.1.jsonl')),
|
|
launcherNotExecuted: !fs.existsSync(path.join(fixture.dir, 'launcher-ran')),
|
|
sourceUnchanged,
|
|
findingsReported: report.includes('FINDING-001') && report.includes('[ai-color-palette]') && report.includes('[low-contrast]'),
|
|
deferred: /deferred/i.test(report),
|
|
nonemptyHandoffs: handoffs.length > 0,
|
|
perFindingHandoff: handoffs.every(([, rule, impact, command]) => entries.some(entry => entry.includes(`[${rule}]`) && entry.toLowerCase().includes(impact) && entry.includes(command))),
|
|
};
|
|
recordE2E(evalCollector, '/design-review plugin handoff', 'Design review plugin discovery E2E', result, { passed: Object.values(checks).every(Boolean) });
|
|
expect(Object.entries(checks).filter(([, passed]) => !passed).map(([name]) => name)).toEqual([]);
|
|
} finally {
|
|
fs.rmSync(fixture.dir, { recursive: true, force: true });
|
|
}
|
|
}, CAPTURE_MS);
|
|
});
|
|
|
|
describeIfSelected('Design review detector shim E2E', ['design-review-detector-shim', 'design-review-detector-shim-dom'], () => {
|
|
let repoDir: string;
|
|
let engineDir: string;
|
|
let server: ReturnType<typeof Bun.serve> | null = null;
|
|
|
|
beforeAll(() => {
|
|
repoDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-detector-shim-'));
|
|
const run = (cmd: string, args: string[]) => spawnSync(cmd, args, { cwd: repoDir, stdio: 'pipe', timeout: 5000 });
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
fs.writeFileSync(path.join(repoDir, 'index.html'), '<h1>Clean</h1>\n');
|
|
fs.writeFileSync(path.join(repoDir, 'styles.css'), 'body { font-size: 16px; }\n');
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial']);
|
|
run('git', ['checkout', '-b', 'feature/landing']);
|
|
fs.writeFileSync(path.join(repoDir, 'index.html'), fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-design-slop.html'), 'utf-8'));
|
|
fs.writeFileSync(path.join(repoDir, 'styles.css'), fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-design-slop.css'), 'utf-8'));
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'add landing page']);
|
|
engineDir = makeFakeEngine();
|
|
fs.writeFileSync(
|
|
path.join(repoDir, 'design-review-detector.md'),
|
|
detectorSkillText([
|
|
['**Design detector (optional, deterministic):**', '**Create output directories:**'],
|
|
['**Phase 0: mechanical scan**', '## Phases 1-6'],
|
|
]),
|
|
);
|
|
fs.writeFileSync(
|
|
path.join(repoDir, 'design-review-dom-dump.md'),
|
|
detectorSkillText([['### DOM dump (DOM mode only', '### Auth Detection']]),
|
|
);
|
|
});
|
|
|
|
afterAll(() => {
|
|
server?.stop(true);
|
|
try { fs.rmSync(repoDir, { recursive: true, force: true }); } catch {}
|
|
try { fs.rmSync(engineDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('design-review-detector-shim', async () => {
|
|
const started = Date.now();
|
|
let evidenceDir: string | undefined;
|
|
let result: SkillTestResult | undefined;
|
|
let passed = false;
|
|
let failure: unknown;
|
|
try {
|
|
evidenceDir = fs.mkdtempSync(path.join(os.tmpdir(), 'detector-source-evidence-'));
|
|
const receiptPath = path.join(evidenceDir, 'calls.jsonl');
|
|
const outPath = path.join(repoDir, 'detector-output.md');
|
|
fs.rmSync(outPath, { force: true });
|
|
fs.writeFileSync(path.join(evidenceDir, 'bun'), `#!${process.execPath}
|
|
import * as fs from 'node:fs';
|
|
import * as path from 'node:path';
|
|
import { spawnSync } from 'node:child_process';
|
|
const argv = process.argv.slice(2);
|
|
const wrapper = argv.findIndex(arg => {
|
|
try { return fs.realpathSync(path.resolve(arg)) === ${JSON.stringify(fs.realpathSync(path.join(ROOT, 'bin', 'gstack-design-detect.ts')))}; }
|
|
catch { return false; }
|
|
});
|
|
const dir = wrapper < 0 ? null : fs.mkdtempSync(${JSON.stringify(path.join(evidenceDir, 'call-'))});
|
|
const log = dir && path.join(dir, 'engine.jsonl');
|
|
const run = spawnSync(process.execPath, argv, {
|
|
cwd: process.cwd(), stdio: 'inherit', timeout: ${CAPTURE_MS},
|
|
env: { ...process.env, ...(log ? { IMPECCABLE_FAKE_LOG: log } : {}) },
|
|
});
|
|
if (dir) fs.appendFileSync(${JSON.stringify(receiptPath)}, JSON.stringify({
|
|
argv: argv.slice(wrapper + 1), cwd: process.cwd(), exit: run.status,
|
|
engine: log && fs.existsSync(log) ? fs.readFileSync(log, 'utf8').trim().split('\\n').map(line => JSON.parse(line)) : [],
|
|
}) + '\\n', { mode: 0o600 });
|
|
process.exitCode = run.status ?? 1;
|
|
`, { mode: 0o755 });
|
|
result = await runSkillTest({
|
|
prompt: `You are in a git repo on branch feature/landing with changes against main (the base branch).
|
|
Read design-review-detector.md: it is the Setup "Design detector" block and "Phase 0: mechanical scan" from /design-review.
|
|
This is a diff-aware run with no URL, so it is SOURCE mode. Run the probe, then the Phase 0 source-mode scan with base main, exactly as written (use --host claude).
|
|
Do not run any browser step, do not fix anything, do not run npx.
|
|
Then write ${repoDir}/detector-output.md: one FINDING-NNN row per rule in the DETECT_TOP block, each tagged with its [rule-id] and the printed impact, plus the first line the probe printed.`,
|
|
workingDirectory: repoDir,
|
|
maxTurns: 15,
|
|
timeout: CAPTURE_MS,
|
|
testName: 'design-review-detector-shim',
|
|
runId,
|
|
env: {
|
|
IMPECCABLE_BIN: path.join(engineDir, 'impeccable'), IMPECCABLE_FAKE_OUTPUT: DETECT_SAMPLE,
|
|
PATH: `${evidenceDir}${path.delimiter}${process.env.PATH ?? ''}`,
|
|
},
|
|
});
|
|
|
|
logCost('/design-review detector shim (source)', result);
|
|
expect(result.exitReason).toBe('success');
|
|
expect(result.browseErrors).toEqual([]);
|
|
|
|
const bash = result.toolCalls.filter(c => c.tool === 'Bash').map(c => String(c.input?.command ?? ''));
|
|
expect(bash.some(c => c.includes('npx impeccable'))).toBe(false);
|
|
// `$B` runs the browse binary only in command position; a shell variable
|
|
// named B (a base branch, say) is not a browser step.
|
|
expect(bash.some(c => /(?:^|[;&|(\n]|\b(?:then|do|else)\b)\s*"?\$B"?\s+\S|\bbrowse\s|\baside\s+repl\b|\bplaywright\b|\bpuppeteer\b/.test(c))).toBe(false);
|
|
expect(result.toolCalls.some(c => /browser|browse|playwright|puppeteer/i.test(c.tool))).toBe(false);
|
|
const calls: Array<{ argv: string[]; cwd: string; exit: number; engine: Array<{ argv: string[]; cwd: string }> }> =
|
|
fs.readFileSync(receiptPath, 'utf8').trim().split('\n').map(line => JSON.parse(line));
|
|
const probes = calls.filter(c => c.argv[0] === 'probe');
|
|
expect(probes.length).toBeGreaterThan(0);
|
|
for (const probe of probes) {
|
|
expect(probe.argv).toEqual(['probe', '--host', 'claude']);
|
|
expect(probe.cwd).toBe(fs.realpathSync(repoDir));
|
|
expect(probe.exit).toBe(0);
|
|
}
|
|
const scans = calls.filter(c => c.argv[0] === 'scan');
|
|
expect(scans.length).toBeGreaterThan(0);
|
|
for (const scan of scans) {
|
|
expect(scan.cwd).toBe(fs.realpathSync(repoDir));
|
|
expect(scan.exit).toBe(2);
|
|
expect(scan.argv[scan.argv.indexOf('--changed') + 1]).toBe('main');
|
|
expect(scan.argv[scan.argv.indexOf('--host') + 1]).toBe('claude');
|
|
expect(scan.argv[scan.argv.indexOf('--format') + 1]).toBe('gstack');
|
|
expect(scan.argv.slice(1).sort()).toEqual(['--changed', 'main', '--format', 'gstack', '--host', 'claude'].sort());
|
|
expect(scan.engine).toHaveLength(1);
|
|
expect(scan.engine[0].cwd).toBe(fs.realpathSync(repoDir));
|
|
expect(scan.engine[0].argv).toEqual(['detect', '--json',
|
|
fs.realpathSync(path.join(repoDir, 'index.html')), fs.realpathSync(path.join(repoDir, 'styles.css'))]);
|
|
}
|
|
const toolOutputs = result.toolCalls.map(c => String(c.output ?? '')).join('\n');
|
|
expect(fs.existsSync(outPath)).toBe(true);
|
|
const out = fs.readFileSync(outPath, 'utf-8');
|
|
expect(toolOutputs.includes('IMPECCABLE_READY') || out.includes('IMPECCABLE_READY')).toBe(true);
|
|
expect(out).toContain('FINDING-001');
|
|
expect(out).toContain('[ai-color-palette]');
|
|
expect(out).toContain('[low-contrast]');
|
|
passed = true;
|
|
} catch (error) {
|
|
failure = error;
|
|
throw error;
|
|
} finally {
|
|
let artifactNote = '';
|
|
if (evidenceDir && (process.env.EVALS_RUN_ID || process.env.GSTACK_EVAL_DIR)) {
|
|
try {
|
|
const segment = (process.env.EVALS_RUN_ID || 'local').replace(/[^a-zA-Z0-9_-]/g, '_').slice(0, 120);
|
|
const root = path.resolve(process.env.GSTACK_EVAL_DIR || getProjectEvalDir(), 'design-detector', segment);
|
|
fs.mkdirSync(root, { recursive: true, mode: 0o700 });
|
|
const retained = fs.mkdtempSync(path.join(root, `design-review-detector-shim-${started}-`));
|
|
fs.chmodSync(retained, 0o700);
|
|
const receipt = path.join(evidenceDir, 'calls.jsonl');
|
|
fs.writeFileSync(path.join(retained, 'calls.jsonl'), fs.existsSync(receipt) ? fs.readFileSync(receipt, 'utf8') : '', { mode: 0o600 });
|
|
artifactNote = `Detector artifacts: ${retained}`;
|
|
console.log(artifactNote);
|
|
} catch (error) {
|
|
artifactNote = `Detector artifact write failed: ${String(error)}`;
|
|
console.error(artifactNote);
|
|
}
|
|
}
|
|
if (evidenceDir) { try { fs.rmSync(evidenceDir, { recursive: true, force: true }); } catch {} }
|
|
const error = (failure instanceof Error ? failure.message : String(failure)) + (artifactNote ? `\n${artifactNote}` : '');
|
|
if (result) {
|
|
recordE2E(evalCollector, '/design-review detector shim', 'Design review detector shim E2E (source mode)', result, {
|
|
passed, ...(passed ? {} : { error }),
|
|
});
|
|
} else {
|
|
evalCollector?.addTest({
|
|
name: '/design-review detector shim', suite: 'Design review detector shim E2E (source mode)', tier: 'e2e',
|
|
passed: false, duration_ms: Date.now() - started, cost_usd: 0,
|
|
model: process.env.EVALS_MODEL ?? resolveEvalModel('capture'), exit_reason: 'harness_error',
|
|
error: `${error}\nRunner returned no result; cost and usage unavailable.`,
|
|
});
|
|
}
|
|
}
|
|
}, CAPTURE_MS);
|
|
|
|
// DOM mode needs a browser engine for the dump: gstack's own browse binary
|
|
// (CI builds it with build:gates). Self-skips when it is absent, like the
|
|
// other render gates.
|
|
testConcurrentIfSelected(
|
|
'design-review-detector-shim-dom',
|
|
async () => {
|
|
if (!fs.existsSync(browseBin)) {
|
|
console.log('design-review-detector-shim (dom mode): browse binary absent, skipping (build it with bun run build:gates)');
|
|
return;
|
|
}
|
|
const site = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-detector-site-'));
|
|
fs.copyFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-design-slop.html'), path.join(site, 'index.html'));
|
|
fs.copyFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-design-slop.css'), path.join(site, 'styles.css'));
|
|
server = Bun.serve({
|
|
hostname: '127.0.0.1', port: 0,
|
|
fetch(req) {
|
|
const p = new URL(req.url).pathname.replace(/^\//, '') || 'index.html';
|
|
const f = path.join(site, p);
|
|
return fs.existsSync(f) ? new Response(Bun.file(f)) : new Response('not found', { status: 404 });
|
|
},
|
|
});
|
|
const url = `http://127.0.0.1:${server.port}/index.html`;
|
|
const reportDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-detector-report-'));
|
|
const gstackHome = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-detector-home-'));
|
|
// REPORT_DIR must sit under <gstack home>/projects/<slug>/designs/ for the wrapper's allow-list.
|
|
const allowed = path.join(gstackHome, 'projects', 'shim', 'designs', 'design-audit-20260908');
|
|
fs.mkdirSync(path.join(allowed, 'dom', 'run1'), { recursive: true });
|
|
// The agent's $B commands and this test's cleanup share ONE daemon, scoped to this run.
|
|
const browseState = path.join(gstackHome, 'browse.json');
|
|
try {
|
|
const result = await runSkillTest({
|
|
prompt: `Read design-review-detector.md (the /design-review detector block + Phase 0) and design-review-dom-dump.md (the Phase 3 DOM dump section).
|
|
The target is the URL ${url}, so this is DOM mode: never scan source files.
|
|
Aside is NOT available; use the fallback browser engine: $B is ${browseBin}. Run "$B goto ${url}" first, then follow the fallback-engine DOM dump steps exactly as written, with {page} = home, REPORT_DIR=${allowed}, RUN_ID=run1, and --host claude. Then run the single scan over ${allowed}/dom/run1 and write ${allowed}/detector-output.md with one FINDING-NNN row per rule in the DETECT_TOP block, each tagged [rule-id], and the line "static scan of the rendered DOM; cross-origin CSS not resolved".
|
|
Do not run npx. Do not fix anything.`,
|
|
workingDirectory: repoDir,
|
|
maxTurns: 25,
|
|
timeout: CAPTURE_LONG_MS,
|
|
testName: 'design-review-detector-shim-dom',
|
|
runId,
|
|
env: { IMPECCABLE_BIN: path.join(engineDir, 'impeccable'), IMPECCABLE_FAKE_OUTPUT: DETECT_SAMPLE, GSTACK_HOME: gstackHome, BROWSE_STATE_FILE: browseState },
|
|
});
|
|
logCost('/design-review detector shim (dom)', result);
|
|
recordE2E(evalCollector, '/design-review detector shim (dom)', 'Design review detector shim E2E (DOM mode)', result);
|
|
expect(result.exitReason).toBe('success');
|
|
const bash = result.toolCalls.filter(c => c.tool === 'Bash').map(c => String(c.input?.command ?? ''));
|
|
expect(bash.some(c => c.includes('dom-dump.js') && c.includes('--out') && c.includes('--raw'))).toBe(true); // $B js '('"$_DUMP"')()' with the file spliced in
|
|
expect(bash.some(c => domScanOfRun(c, path.join(allowed, 'dom', 'run1')))).toBe(true);
|
|
expect(bash.some(c => /gstack-design-detect\.ts scan --changed/.test(c))).toBe(false);
|
|
const dumps = fs.readdirSync(path.join(allowed, 'dom', 'run1')).filter(f => f.endsWith('.dom.html'));
|
|
expect(dumps.length).toBeGreaterThan(0);
|
|
expect(fs.readFileSync(path.join(allowed, 'dom', 'run1', dumps[0]), 'utf-8')).toContain('data-gstack-dom-css');
|
|
const out = fs.readFileSync(path.join(allowed, 'detector-output.md'), 'utf-8');
|
|
expect(out).toContain('[ai-color-palette]');
|
|
expect(out).toContain('static scan of the rendered DOM');
|
|
} finally {
|
|
server?.stop(true); server = null;
|
|
try { spawnSync(browseBin, ['stop'], { stdio: 'pipe', timeout: 10_000, env: { ...process.env, BROWSE_STATE_FILE: browseState } }); } catch {}
|
|
for (const d of [site, reportDir, gstackHome]) { try { fs.rmSync(d, { recursive: true, force: true }); } catch {} }
|
|
}
|
|
},
|
|
CAPTURE_LONG_MS,
|
|
);
|
|
});
|
|
|
|
describeIfSelected('Design HTML slop gate E2E', ['design-html-slop-gate'], () => {
|
|
let workDir: string;
|
|
let engineDir: string;
|
|
|
|
beforeAll(() => {
|
|
workDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-html-gate-'));
|
|
const run = (cmd: string, args: string[]) => spawnSync(cmd, args, { cwd: workDir, stdio: 'pipe', timeout: 5000 });
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
const css = fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-design-slop.css'), 'utf-8');
|
|
const html = fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-design-slop.html'), 'utf-8')
|
|
.replace('<link rel="stylesheet" href="styles.css">', `<style>\n${css}\n</style>`);
|
|
fs.writeFileSync(path.join(workDir, 'finalized.html'), html);
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'finalized html']);
|
|
engineDir = makeFakeEngine();
|
|
const full = fs.readFileSync(path.join(ROOT, 'design-html', 'SKILL.md'), 'utf-8');
|
|
const text = [
|
|
sliceBetween(full, '**Design detector (optional, deterministic):**', '## Step 0: Input Detection'),
|
|
sliceBetween(full, '### Slop Gate (bounded, never a loop)', '### Verification Screenshots'),
|
|
].join('\n\n---\n\n').replaceAll('$HOME/.claude/skills/gstack', ROOT).replaceAll('~/.claude/skills/gstack', ROOT);
|
|
fs.writeFileSync(path.join(workDir, 'design-html-gate.md'), text);
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(workDir, { recursive: true, force: true }); } catch {}
|
|
try { fs.rmSync(engineDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('design-html-slop-gate', async () => {
|
|
const result = await runSkillTest({
|
|
prompt: `Read design-html-gate.md: the /design-html detector probe block and its "Slop Gate (bounded, never a loop)" step.
|
|
finalized.html in this directory is the finished page. Run the probe (--host claude), then the slop gate on finalized.html exactly as written: one surgical fix pass over the non-advisory findings, one rescan, then stop.
|
|
Write ${workDir}/gate-output.md listing what you fixed and every remaining finding as accepted-with-reason, each tagged with its [rule-id]. Do not take screenshots, do not run npx, do not scan more than twice.`,
|
|
workingDirectory: workDir,
|
|
maxTurns: 20,
|
|
timeout: CAPTURE_MS,
|
|
testName: 'design-html-slop-gate',
|
|
runId,
|
|
env: { IMPECCABLE_BIN: path.join(engineDir, 'impeccable'), IMPECCABLE_FAKE_OUTPUT: DETECT_SAMPLE },
|
|
});
|
|
|
|
logCost('/design-html slop gate', result);
|
|
recordE2E(evalCollector, '/design-html slop gate', 'Design HTML slop gate E2E', result);
|
|
expect(result.exitReason).toBe('success');
|
|
const scans = result.toolCalls.filter(c => c.tool === 'Bash' && /gstack-design-detect\.ts scan /.test(String(c.input?.command ?? '')));
|
|
expect(scans.length).toBeGreaterThanOrEqual(1);
|
|
expect(scans.length).toBeLessThanOrEqual(2);
|
|
const outPath = path.join(workDir, 'gate-output.md');
|
|
expect(fs.existsSync(outPath)).toBe(true);
|
|
const out = fs.readFileSync(outPath, 'utf-8').toLowerCase();
|
|
expect(out).toContain('ai-color-palette');
|
|
expect(out).toContain('accepted');
|
|
}, CAPTURE_MS);
|
|
});
|