mirror of
https://github.com/garrytan/gstack.git
synced 2026-06-26 19:49:57 +02:00
v1.58.5.0 feat: first-run activation scaffold + gstack router front door (#2078)
* feat: first-run activation — project-aware scaffold, router front door, onboarding nudges Adds the activation system that drives a new install toward a concrete first move: - bin/gstack-first-task-detect: local-git+filesystem repo classifier emitting one validated enum bucket (greenfield/code_<lang>/branch_ahead/dirty_default/clean_default), portable timeouts, fail-safe empty output. - generate-first-run-guidance.ts: unified preamble section — first-run project-aware scaffold + returning-session plan->review->ship tip, gated on a persistent .activated marker and never run in headless. Detection wired lazily in generate-preamble-bash.ts. - SKILL.md.tmpl: top-level gstack skill is now a pure router (browse body removed; it lives in /browse), routing any request and sending browser/QA work to /browse. - setup: first-move nudge on first install. office-hours: closing handoff that launches the next review via the Skill tool. - telemetry-ingest: accept onboarding/first_task_scaffold_shown/handoff/route event types. * test: cover first-run detection + repoint browse-content assertions to /browse - New unit tests for every detection bucket, the eval-safe enum contract, and the first-run gating (test/preamble-first-task-scaffold.test.ts); periodic E2E that runs the detector through the real harness (test/skill-e2e-first-task-scaffold.test.ts). - Repoint browse-content assertions (gen-skill-docs, audit-compliance, skill-validation, LLM-judge eval) from the root skill to browse/SKILL.md following the router split; add a regression pinning that the router carries no browse body. - Register first-task-scaffold touchfiles + periodic tier; bump parity/carve size caps ~1-2KB per skill for the shared first-run-guidance preamble section. - Refresh ship golden fixtures for the preamble addition. * chore: regenerate SKILL.md + llms.txt for first-run activation * chore: bump version and changelog (v1.58.5.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(test): repoint bws skillmd-* setup-block assertions to browse/SKILL.md The skillmd-setup-discovery / -no-local-binary / -outside-git E2E tests extracted the `## SETUP`→`## IMPORTANT` browse binary-discovery block from the root SKILL.md. P2 moved that block to browse/SKILL.md (end anchor is now `## Core QA Patterns`), so the slice came back empty and the `browse/dist/browse` guard failed. Repoint to browse/SKILL.md. Verified: 7/7 e2e-browse pass locally. * fix(test): tolerate skill-discovery race in PTY plan-mode smoke The e2e-pty-plan-smoke suite (office-hours / plan-mode-no-op) failed in CI with `Unknown command: /office-hours` (claude exited ~10s) while passing locally. Root cause: a cold CI container's overlay-FS scan of the symlinked ~/.claude/skills registry finishes AFTER the runner's 8s boot grace, so the first `/skill` send reaches claude before the skill is indexed and is rejected as unknown. The runner gave up on the first "Unknown command:" line. runPlanSkillObservation now re-sends the skill command up to 3x (6s apart), re-marking the buffer each time so stale scrollback can't re-trip the check, before concluding the skill is genuinely unregistered. A real dangling-symlink / missing-skill still surfaces as 'exited' (after retries), preserving the original diagnostic. Pure-helper contract unchanged: 95/95 unit tests pass. This is a pre-existing harness bug (fails identically on #2077's own branch, which introduced the suite) surfaced while shipping the activation feature. * debug(ci): temporarily instrument pty-smoke skill discovery Capture claude version, env, registry tree, and a claude -p discovery probe to pin why /office-hours isn't discovered in CI (retries proved it's not a race). Temporary — revert once the registry fix is identified. * chore: revert pty-smoke harness experiments (race-retry + CI debug step) Diagnosis is conclusive and the experiments aren't the fix, so restore the harness to its original state (net-zero diff vs main for both files). What the CI debug step proved: `claude -p` returns READY — claude v2.1.187 fully DISCOVERS /office-hours from the symlinked registry. Only the interactive PTY TUI rejects it as "Unknown command" (and it received the full command text). So the e2e-pty-plan-smoke failure is a claude 2.1.187 interactive-TUI regression (skills discovered by `claude -p` aren't exposed as TUI slash commands), pre-existing in the #2077 harness and failing identically on its own origin branch — unrelated to this activation PR. The race-retry can't help (the TUI genuinely lacks the command); the debug step also tripped actionlint (shellcheck SC2012). Both reverted. * fix(ci): copy SKILL.md as real files in pty-smoke registry (cross-mount symlink) The e2e-pty-plan-smoke suite failed with "Unknown command: /office-hours" in CI while passing locally. Root cause (proven, not guessed): claude 2.1.187's interactive-TUI skill scanner does not follow the /github/home -> /__w cross-mount symlink the registry used for per-skill SKILL.md. Evidence: a CI debug step showed `claude -p` discovered the skill (printed READY), and a local macOS repro with the identical symlinked registry recognized /office-hours — isolating the failure to the container's cross-mount symlink, not registration content, claude version, duplicate names, or a race. Fix: register the per-skill SKILL.md + sections as REAL copies (same mount as $HOME) so the TUI reads them directly. The gstack root stays a symlink — the preamble's runtime bash resolves bin/* and sections/* through it and bash follows cross-mount symlinks fine. * fix(ci): guard rm expansion in pty-smoke registry (shellcheck SC2115) * fix(ci): also register pty-smoke skills project-scoped (cwd/.claude/skills) The real-file user-dir registration still left the TUI rejecting /office-hours in the container. claude's interactive TUI surfaces /slash commands from the PROJECT dir (<cwd>/.claude/skills); the smokes run with cwd=$REPO whose .claude/skills is gitignored (absent on a fresh CI checkout), so the user-dir registry feeds `claude -p` (READY) but not the TUI. Populate $REPO/.claude/skills with real SKILL.md + sections copies (no gstack symlink there — it would point at its own parent; runtime paths use the user-dir gstack symlink). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -23,12 +23,14 @@ function getAllSkillMds(): Array<{ name: string; content: string }> {
|
||||
describe('Audit compliance', () => {
|
||||
// Fix 1: W007 — No hardcoded credentials in documentation
|
||||
test('no hardcoded credential patterns in SKILL.md.tmpl', () => {
|
||||
const tmpl = readFileSync(join(ROOT, 'SKILL.md.tmpl'), 'utf-8');
|
||||
// P2 (v1.2.0): the browse QA examples moved from the root router to
|
||||
// browse/SKILL.md.tmpl. The security intent is unchanged — the QA form
|
||||
// examples must not ship real-looking credentials; generic placeholders
|
||||
// ("user@test.com", "password") are fine.
|
||||
const tmpl = readFileSync(join(ROOT, 'browse', 'SKILL.md.tmpl'), 'utf-8');
|
||||
expect(tmpl).not.toContain('"password123"');
|
||||
expect(tmpl).not.toContain('"test@example.com"');
|
||||
expect(tmpl).not.toContain('"test@test.com"');
|
||||
expect(tmpl).toContain('$TEST_EMAIL');
|
||||
expect(tmpl).toContain('$TEST_PASSWORD');
|
||||
});
|
||||
|
||||
// Fix 2: Conditional telemetry — binary calls wrapped with existence check
|
||||
@@ -71,7 +73,8 @@ describe('Audit compliance', () => {
|
||||
|
||||
// Fix 4: W011 — Untrusted content warning in command reference
|
||||
test('command reference includes untrusted content warning after Navigation', () => {
|
||||
const rootSkill = readFileSync(join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
// P2 (v1.2.0): the command reference moved from the root router to browse/SKILL.md.
|
||||
const rootSkill = readFileSync(join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const navIdx = rootSkill.indexOf('### Navigation');
|
||||
const readingIdx = rootSkill.indexOf('### Reading');
|
||||
expect(navIdx).toBeGreaterThan(-1);
|
||||
|
||||
+29
@@ -60,6 +60,17 @@ echo "SESSION_KIND: $_SESSION_KIND"
|
||||
if [ "$_SESSION_KIND" != "headless" ] && { [ -n "${CONDUCTOR_WORKSPACE_PATH:-}" ] || [ -n "${CONDUCTOR_PORT:-}" ]; }; then
|
||||
echo "CONDUCTOR_SESSION: true"
|
||||
fi
|
||||
_ACTIVATED=$([ -f ~/.gstack/.activated ] && echo "yes" || echo "no")
|
||||
_FIRST_LOOP_SHOWN=$([ -f ~/.gstack/.first-loop-tip-shown ] && echo "yes" || echo "no")
|
||||
echo "ACTIVATED: $_ACTIVATED"
|
||||
echo "FIRST_LOOP_SHOWN: $_FIRST_LOOP_SHOWN"
|
||||
# First-run project detection: run the detector ONLY on the first-ever skill run
|
||||
# (ACTIVATED=no, interactive) so it stays off the hot path for every run after.
|
||||
_FIRST_TASK=""
|
||||
if [ "$_ACTIVATED" = "no" ] && [ "$_SESSION_KIND" != "headless" ]; then
|
||||
_FIRST_TASK=$(~/.claude/skills/gstack/bin/gstack-first-task-detect 2>/dev/null || true)
|
||||
fi
|
||||
echo "FIRST_TASK: $_FIRST_TASK"
|
||||
_LAKE_SEEN=$([ -f ~/.gstack/.completeness-intro-seen ] && echo "yes" || echo "no")
|
||||
echo "LAKE_INTRO: $_LAKE_SEEN"
|
||||
_TEL=$(~/.claude/skills/gstack/bin/gstack-config get telemetry 2>/dev/null || true)
|
||||
@@ -229,6 +240,24 @@ touch ~/.gstack/.proactive-prompted
|
||||
|
||||
Skip if `PROACTIVE_PROMPTED` is `yes`.
|
||||
|
||||
## First-run guidance (one-time)
|
||||
|
||||
If `ACTIVATED` is `no` (first skill run on this machine) AND the preamble printed a non-empty `FIRST_TASK:` value that is NOT `nongit`: show ONE short, project-specific line mapped from the token, as a heads-up, then CONTINUE with whatever the user actually asked — do NOT halt their task. Map the token: `greenfield` → "Fresh repo — shape it first with `/spec` or `/office-hours`." `code_node`/`code_python`/`code_rust`/`code_go`/`code_ruby`/`code_ios` → "There's code here — `/qa` to see it work, or `/investigate` if something's off." `branch_ahead` → "Unshipped work on this branch — `/review` then `/ship`." `dirty_default` → "Uncommitted changes — `/review` before committing." `clean_default` → "Pick one: `/spec`, `/investigate`, or `/qa`." Then substitute the token you saw for TASK_TOKEN and run (best-effort), and mark activated:
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-telemetry-log --event-type first_task_scaffold_shown --skill "TASK_TOKEN" --outcome shown 2>/dev/null || true
|
||||
touch ~/.gstack/.activated 2>/dev/null || true
|
||||
```
|
||||
|
||||
If `ACTIVATED` is `no` but `FIRST_TASK:` is empty or `nongit` (headless, non-git, or nothing actionable): show nothing, just run `touch ~/.gstack/.activated 2>/dev/null || true`.
|
||||
|
||||
Else if `ACTIVATED` is `yes` AND `FIRST_LOOP_SHOWN` is `no`: say once as a heads-up (then continue):
|
||||
|
||||
> Tip: gstack pays off when you complete one loop — **plan → review → ship**. A common first loop: `/office-hours` or `/spec` to shape it, `/plan-eng-review` to lock it, then `/ship`.
|
||||
|
||||
Then run `touch ~/.gstack/.first-loop-tip-shown 2>/dev/null || true`.
|
||||
|
||||
Skip this section if `ACTIVATED` and `FIRST_LOOP_SHOWN` are both `yes`.
|
||||
|
||||
If `HAS_ROUTING` is `no` AND `ROUTING_DECLINED` is `false` AND `PROACTIVE_PROMPTED` is `yes`:
|
||||
Check if a CLAUDE.md file exists in the project root. If it does not exist, create it.
|
||||
|
||||
|
||||
+29
@@ -46,6 +46,17 @@ echo "SESSION_KIND: $_SESSION_KIND"
|
||||
if [ "$_SESSION_KIND" != "headless" ] && { [ -n "${CONDUCTOR_WORKSPACE_PATH:-}" ] || [ -n "${CONDUCTOR_PORT:-}" ]; }; then
|
||||
echo "CONDUCTOR_SESSION: true"
|
||||
fi
|
||||
_ACTIVATED=$([ -f ~/.gstack/.activated ] && echo "yes" || echo "no")
|
||||
_FIRST_LOOP_SHOWN=$([ -f ~/.gstack/.first-loop-tip-shown ] && echo "yes" || echo "no")
|
||||
echo "ACTIVATED: $_ACTIVATED"
|
||||
echo "FIRST_LOOP_SHOWN: $_FIRST_LOOP_SHOWN"
|
||||
# First-run project detection: run the detector ONLY on the first-ever skill run
|
||||
# (ACTIVATED=no, interactive) so it stays off the hot path for every run after.
|
||||
_FIRST_TASK=""
|
||||
if [ "$_ACTIVATED" = "no" ] && [ "$_SESSION_KIND" != "headless" ]; then
|
||||
_FIRST_TASK=$($GSTACK_BIN/gstack-first-task-detect 2>/dev/null || true)
|
||||
fi
|
||||
echo "FIRST_TASK: $_FIRST_TASK"
|
||||
_LAKE_SEEN=$([ -f ~/.gstack/.completeness-intro-seen ] && echo "yes" || echo "no")
|
||||
echo "LAKE_INTRO: $_LAKE_SEEN"
|
||||
_TEL=$($GSTACK_BIN/gstack-config get telemetry 2>/dev/null || true)
|
||||
@@ -215,6 +226,24 @@ touch ~/.gstack/.proactive-prompted
|
||||
|
||||
Skip if `PROACTIVE_PROMPTED` is `yes`.
|
||||
|
||||
## First-run guidance (one-time)
|
||||
|
||||
If `ACTIVATED` is `no` (first skill run on this machine) AND the preamble printed a non-empty `FIRST_TASK:` value that is NOT `nongit`: show ONE short, project-specific line mapped from the token, as a heads-up, then CONTINUE with whatever the user actually asked — do NOT halt their task. Map the token: `greenfield` → "Fresh repo — shape it first with `/spec` or `/office-hours`." `code_node`/`code_python`/`code_rust`/`code_go`/`code_ruby`/`code_ios` → "There's code here — `/qa` to see it work, or `/investigate` if something's off." `branch_ahead` → "Unshipped work on this branch — `/review` then `/ship`." `dirty_default` → "Uncommitted changes — `/review` before committing." `clean_default` → "Pick one: `/spec`, `/investigate`, or `/qa`." Then substitute the token you saw for TASK_TOKEN and run (best-effort), and mark activated:
|
||||
```bash
|
||||
$GSTACK_BIN/gstack-telemetry-log --event-type first_task_scaffold_shown --skill "TASK_TOKEN" --outcome shown 2>/dev/null || true
|
||||
touch ~/.gstack/.activated 2>/dev/null || true
|
||||
```
|
||||
|
||||
If `ACTIVATED` is `no` but `FIRST_TASK:` is empty or `nongit` (headless, non-git, or nothing actionable): show nothing, just run `touch ~/.gstack/.activated 2>/dev/null || true`.
|
||||
|
||||
Else if `ACTIVATED` is `yes` AND `FIRST_LOOP_SHOWN` is `no`: say once as a heads-up (then continue):
|
||||
|
||||
> Tip: gstack pays off when you complete one loop — **plan → review → ship**. A common first loop: `/office-hours` or `/spec` to shape it, `/plan-eng-review` to lock it, then `/ship`.
|
||||
|
||||
Then run `touch ~/.gstack/.first-loop-tip-shown 2>/dev/null || true`.
|
||||
|
||||
Skip this section if `ACTIVATED` and `FIRST_LOOP_SHOWN` are both `yes`.
|
||||
|
||||
If `HAS_ROUTING` is `no` AND `ROUTING_DECLINED` is `false` AND `PROACTIVE_PROMPTED` is `yes`:
|
||||
Check if a CLAUDE.md file exists in the project root. If it does not exist, create it.
|
||||
|
||||
|
||||
+29
@@ -48,6 +48,17 @@ echo "SESSION_KIND: $_SESSION_KIND"
|
||||
if [ "$_SESSION_KIND" != "headless" ] && { [ -n "${CONDUCTOR_WORKSPACE_PATH:-}" ] || [ -n "${CONDUCTOR_PORT:-}" ]; }; then
|
||||
echo "CONDUCTOR_SESSION: true"
|
||||
fi
|
||||
_ACTIVATED=$([ -f ~/.gstack/.activated ] && echo "yes" || echo "no")
|
||||
_FIRST_LOOP_SHOWN=$([ -f ~/.gstack/.first-loop-tip-shown ] && echo "yes" || echo "no")
|
||||
echo "ACTIVATED: $_ACTIVATED"
|
||||
echo "FIRST_LOOP_SHOWN: $_FIRST_LOOP_SHOWN"
|
||||
# First-run project detection: run the detector ONLY on the first-ever skill run
|
||||
# (ACTIVATED=no, interactive) so it stays off the hot path for every run after.
|
||||
_FIRST_TASK=""
|
||||
if [ "$_ACTIVATED" = "no" ] && [ "$_SESSION_KIND" != "headless" ]; then
|
||||
_FIRST_TASK=$($GSTACK_BIN/gstack-first-task-detect 2>/dev/null || true)
|
||||
fi
|
||||
echo "FIRST_TASK: $_FIRST_TASK"
|
||||
_LAKE_SEEN=$([ -f ~/.gstack/.completeness-intro-seen ] && echo "yes" || echo "no")
|
||||
echo "LAKE_INTRO: $_LAKE_SEEN"
|
||||
_TEL=$($GSTACK_BIN/gstack-config get telemetry 2>/dev/null || true)
|
||||
@@ -217,6 +228,24 @@ touch ~/.gstack/.proactive-prompted
|
||||
|
||||
Skip if `PROACTIVE_PROMPTED` is `yes`.
|
||||
|
||||
## First-run guidance (one-time)
|
||||
|
||||
If `ACTIVATED` is `no` (first skill run on this machine) AND the preamble printed a non-empty `FIRST_TASK:` value that is NOT `nongit`: show ONE short, project-specific line mapped from the token, as a heads-up, then CONTINUE with whatever the user actually asked — do NOT halt their task. Map the token: `greenfield` → "Fresh repo — shape it first with `/spec` or `/office-hours`." `code_node`/`code_python`/`code_rust`/`code_go`/`code_ruby`/`code_ios` → "There's code here — `/qa` to see it work, or `/investigate` if something's off." `branch_ahead` → "Unshipped work on this branch — `/review` then `/ship`." `dirty_default` → "Uncommitted changes — `/review` before committing." `clean_default` → "Pick one: `/spec`, `/investigate`, or `/qa`." Then substitute the token you saw for TASK_TOKEN and run (best-effort), and mark activated:
|
||||
```bash
|
||||
$GSTACK_BIN/gstack-telemetry-log --event-type first_task_scaffold_shown --skill "TASK_TOKEN" --outcome shown 2>/dev/null || true
|
||||
touch ~/.gstack/.activated 2>/dev/null || true
|
||||
```
|
||||
|
||||
If `ACTIVATED` is `no` but `FIRST_TASK:` is empty or `nongit` (headless, non-git, or nothing actionable): show nothing, just run `touch ~/.gstack/.activated 2>/dev/null || true`.
|
||||
|
||||
Else if `ACTIVATED` is `yes` AND `FIRST_LOOP_SHOWN` is `no`: say once as a heads-up (then continue):
|
||||
|
||||
> Tip: gstack pays off when you complete one loop — **plan → review → ship**. A common first loop: `/office-hours` or `/spec` to shape it, `/plan-eng-review` to lock it, then `/ship`.
|
||||
|
||||
Then run `touch ~/.gstack/.first-loop-tip-shown 2>/dev/null || true`.
|
||||
|
||||
Skip this section if `ACTIVATED` and `FIRST_LOOP_SHOWN` are both `yes`.
|
||||
|
||||
If `HAS_ROUTING` is `no` AND `ROUTING_DECLINED` is `false` AND `PROACTIVE_PROMPTED` is `yes`:
|
||||
Check if a CLAUDE.md file exists in the project root. If it does not exist, create it.
|
||||
|
||||
|
||||
+16
-12
@@ -108,7 +108,7 @@ const CLAUDE_GENERATED_SKILLS = ALL_SKILLS.filter(skill => !CLAUDE_SKIPPED_SKILL
|
||||
|
||||
describe('gen-skill-docs', () => {
|
||||
test('generated SKILL.md contains all command categories', () => {
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const categories = new Set(Object.values(COMMAND_DESCRIPTIONS).map(d => d.category));
|
||||
for (const cat of categories) {
|
||||
expect(content).toContain(`### ${cat}`);
|
||||
@@ -116,7 +116,7 @@ describe('gen-skill-docs', () => {
|
||||
});
|
||||
|
||||
test('generated SKILL.md contains all commands', () => {
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
for (const [cmd, meta] of Object.entries(COMMAND_DESCRIPTIONS)) {
|
||||
const display = meta.usage || cmd;
|
||||
expect(content).toContain(display);
|
||||
@@ -124,7 +124,7 @@ describe('gen-skill-docs', () => {
|
||||
});
|
||||
|
||||
test('command table is sorted alphabetically within categories', () => {
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
// Extract command names from the Navigation section as a test
|
||||
const navSection = content.match(/### Navigation\n\|.*\n\|.*\n([\s\S]*?)(?=\n###|\n## )/);
|
||||
expect(navSection).not.toBeNull();
|
||||
@@ -149,7 +149,7 @@ describe('gen-skill-docs', () => {
|
||||
});
|
||||
|
||||
test('snapshot flags section contains all flags', () => {
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
for (const flag of SNAPSHOT_FLAGS) {
|
||||
expect(content).toContain(flag.short);
|
||||
expect(content).toContain(flag.description);
|
||||
@@ -284,10 +284,12 @@ describe('gen-skill-docs', () => {
|
||||
});
|
||||
|
||||
test('templates contain placeholders', () => {
|
||||
// P2 (v1.2.0): the root template is a pure router — only {{PREAMBLE}}.
|
||||
// The browse command/snapshot placeholders live in browse/SKILL.md.tmpl now.
|
||||
const rootTmpl = fs.readFileSync(path.join(ROOT, 'SKILL.md.tmpl'), 'utf-8');
|
||||
expect(rootTmpl).toContain('{{COMMAND_REFERENCE}}');
|
||||
expect(rootTmpl).toContain('{{SNAPSHOT_FLAGS}}');
|
||||
expect(rootTmpl).toContain('{{PREAMBLE}}');
|
||||
expect(rootTmpl).not.toContain('{{COMMAND_REFERENCE}}');
|
||||
expect(rootTmpl).not.toContain('{{SNAPSHOT_FLAGS}}');
|
||||
|
||||
const browseTmpl = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md.tmpl'), 'utf-8');
|
||||
expect(browseTmpl).toContain('{{COMMAND_REFERENCE}}');
|
||||
@@ -592,7 +594,7 @@ describe('GitLab support in generated skills', () => {
|
||||
describe('description quality evals', () => {
|
||||
// Regression: snapshot flags lost value hints (-d <N>, -s <sel>, -o <path>)
|
||||
test('snapshot flags with values include value hints in output', () => {
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
for (const flag of SNAPSHOT_FLAGS) {
|
||||
if (flag.takesValue) {
|
||||
expect(flag.valueHint).toBeDefined();
|
||||
@@ -659,11 +661,13 @@ describe('description quality evals', () => {
|
||||
|
||||
// Guard: generated output uses → not ->
|
||||
test('generated SKILL.md uses unicode arrows', () => {
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
// Check the Tips section specifically (where we regressed -> from →)
|
||||
const tipsSection = content.slice(content.indexOf('## Tips'));
|
||||
expect(tipsSection).toContain('→');
|
||||
expect(tipsSection).not.toContain('->');
|
||||
// P2 (v1.2.0): the browse body moved out of the top-level router into
|
||||
// browse/SKILL.md. Guard arrow style on the browse body (sliced from its
|
||||
// H1 so the auto-generated `-->` header comments are excluded).
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const body = content.slice(content.indexOf('# browse: QA Testing'));
|
||||
expect(body).toContain('→');
|
||||
expect(body).not.toContain('->');
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -128,6 +128,8 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
maxSkeletonBytes: 90_000,
|
||||
minUnionBytes: 120_000,
|
||||
mustContain: ['VERSION', 'CHANGELOG', 'review', 'merge', 'PR'],
|
||||
// v1.58.5.0: pre-push-guard install (#2077) stacks on the shared first-run-guidance preamble.
|
||||
maxSizeRatio: 1.08,
|
||||
},
|
||||
'plan-ceo-review': {
|
||||
skill: 'plan-ceo-review',
|
||||
@@ -161,7 +163,8 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
gateAfterStop: 'EXIT PLAN MODE GATE',
|
||||
},
|
||||
behavioral: 'plan',
|
||||
maxSkeletonBytes: 62_000,
|
||||
// v1.2.0 activation lift (shared first-run-guidance preamble) + #2077 ask-first scope gate.
|
||||
maxSkeletonBytes: 67_000,
|
||||
minUnionBytes: 70_000,
|
||||
mustContain: ['Architecture', 'Code Quality', 'Test', 'Performance'],
|
||||
// Cross-cutting preamble growth (v1.57.2.0 AUQ-failure prose fallback + the
|
||||
@@ -185,9 +188,11 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
behavioral: 'plan',
|
||||
// +Conductor AUQ-default-prose rule + one-way/continuation safety in the
|
||||
// always-loaded AskUserQuestion Format section.
|
||||
maxSkeletonBytes: 84_000,
|
||||
// v1.2.0 activation lift (shared first-run-guidance preamble) + #2077 ask-first scope gate.
|
||||
maxSkeletonBytes: 88_000,
|
||||
minUnionBytes: 70_000,
|
||||
mustContain: ['design', 'visual'],
|
||||
maxSizeRatio: 1.07,
|
||||
},
|
||||
'plan-devex-review': {
|
||||
skill: 'plan-devex-review',
|
||||
@@ -203,7 +208,8 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
behavioral: 'plan',
|
||||
// +Conductor AUQ-default-prose rule + one-way/destructive prose safety +
|
||||
// continuation protocol in the always-loaded AskUserQuestion Format section.
|
||||
maxSkeletonBytes: 78_000,
|
||||
// v1.2.0 activation lift: first-run-guidance section in the shared preamble.
|
||||
maxSkeletonBytes: 80_000,
|
||||
minUnionBytes: 70_000,
|
||||
mustContain: ['developer experience', 'Getting Started'],
|
||||
// Default-on Codex outside-voice (codexPreflight block + CODEX_MODE branch
|
||||
@@ -224,9 +230,12 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
gateAfterStop: undefined,
|
||||
},
|
||||
behavioral: 'prompt',
|
||||
maxSkeletonBytes: 96_000,
|
||||
// v1.2.0 activation lift: first-run-guidance section in the shared preamble,
|
||||
// plus the P1 office-hours closing handoff (AUQ that launches the next skill).
|
||||
maxSkeletonBytes: 98_000,
|
||||
minUnionBytes: 70_000,
|
||||
mustContain: ['design doc', 'problem statement'],
|
||||
maxSizeRatio: 1.07,
|
||||
},
|
||||
'document-release': {
|
||||
skill: 'document-release',
|
||||
@@ -243,7 +252,8 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
behavioral: 'prompt',
|
||||
// +Conductor AUQ-default-prose rule + one-way/continuation safety in the
|
||||
// always-loaded AskUserQuestion Format section.
|
||||
maxSkeletonBytes: 53_000,
|
||||
// v1.2.0 activation lift: first-run-guidance section in the shared preamble.
|
||||
maxSkeletonBytes: 56_000,
|
||||
minUnionBytes: 55_000,
|
||||
mustContain: ['CHANGELOG', 'Diataxis', 'coverage'],
|
||||
// Two intentional additions stack on this small skill: the AUQ-failure prose
|
||||
@@ -270,7 +280,8 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
behavioral: 'prompt',
|
||||
// +Conductor AUQ-default-prose rule + one-way/continuation safety in the
|
||||
// always-loaded AskUserQuestion Format section.
|
||||
maxSkeletonBytes: 67_000,
|
||||
// v1.2.0 activation lift: first-run-guidance section in the shared preamble.
|
||||
maxSkeletonBytes: 69_000,
|
||||
minUnionBytes: 72_000,
|
||||
mustContain: ['Typography', 'Color', 'Aesthetic Direction'],
|
||||
// Cross-cutting preamble growth (v1.57.2.0 AUQ-failure prose fallback ~2KB +
|
||||
@@ -308,7 +319,8 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
behavioral: 'prompt',
|
||||
// +Conductor AUQ-default-prose rule + one-way/continuation safety in the
|
||||
// always-loaded AskUserQuestion Format section.
|
||||
maxSkeletonBytes: 73_000,
|
||||
// v1.2.0 activation lift: first-run-guidance section in the shared preamble.
|
||||
maxSkeletonBytes: 75_000,
|
||||
minUnionBytes: 72_000,
|
||||
mustContain: ['OWASP', 'STRIDE', 'daily', 'comprehensive', 'verif'],
|
||||
// cso keeps its mode-dispatch + FP-filtering phases always-loaded, so the
|
||||
|
||||
@@ -221,7 +221,9 @@ const MONOLITH_INVARIANTS: ParityInvariant[] = [
|
||||
skill: 'qa',
|
||||
mustContain: ['bug', 'browse', 'fix'],
|
||||
mustHaveHeadings: ['## Preamble', '## When to invoke'],
|
||||
maxSizeRatio: 1.05,
|
||||
// v1.2.0 activation lift: the unified first-run-guidance section (P4 scaffold +
|
||||
// P3 loop tip) is added to every skill's shared preamble — intentional, ~1KB.
|
||||
maxSizeRatio: 1.07,
|
||||
minBytes: 50_000,
|
||||
},
|
||||
{
|
||||
@@ -231,14 +233,16 @@ const MONOLITH_INVARIANTS: ParityInvariant[] = [
|
||||
// Cross-cutting preamble growth (v1.57.2.0 AUQ-failure prose fallback ~2KB + the
|
||||
// cross-session decision-memory nudge) lands this skill just over the strict 1.05;
|
||||
// headroom for the shared preamble additions (matches the carved-skill overrides).
|
||||
maxSizeRatio: 1.07,
|
||||
// v1.2.0 activation lift adds the first-run-guidance section on top.
|
||||
maxSizeRatio: 1.09,
|
||||
minBytes: 30_000,
|
||||
},
|
||||
{
|
||||
skill: 'autoplan',
|
||||
mustContain: ['ceo', 'eng', 'design'],
|
||||
mustHaveHeadings: ['## Preamble', '## When to invoke'],
|
||||
maxSizeRatio: 1.05,
|
||||
// v1.2.0 activation lift: shared first-run-guidance preamble section.
|
||||
maxSizeRatio: 1.07,
|
||||
minBytes: 70_000,
|
||||
},
|
||||
];
|
||||
|
||||
@@ -41,6 +41,10 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
|
||||
'hermetic-canary': ['test/helpers/hermetic-env.ts', 'test/helpers/session-runner.ts', 'test/skill-e2e-hermetic-canary.test.ts', 'lib/conductor-env-shim.ts'],
|
||||
'hermetic-sentinel': ['test/helpers/hermetic-env.ts', 'test/helpers/session-runner.ts', 'test/skill-e2e-hermetic-canary.test.ts', 'lib/conductor-env-shim.ts'],
|
||||
|
||||
// P4 first-run scaffold (activation lift) — the detection binary end-to-end
|
||||
// through the real runner, plus the preamble wiring that gates + maps it.
|
||||
'first-task-scaffold': ['bin/gstack-first-task-detect', 'scripts/resolvers/preamble/generate-first-run-guidance.ts', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'test/skill-e2e-first-task-scaffold.test.ts', 'test/helpers/session-runner.ts'],
|
||||
|
||||
// SKILL.md setup + preamble (depend on ROOT SKILL.md + gen-skill-docs)
|
||||
'skillmd-setup-discovery': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
|
||||
'skillmd-no-local-binary': ['SKILL.md', 'SKILL.md.tmpl', 'scripts/gen-skill-docs.ts'],
|
||||
@@ -459,6 +463,9 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic'> = {
|
||||
'session-awareness': 'gate',
|
||||
'operational-learning': 'gate',
|
||||
|
||||
// P4 first-run scaffold — periodic (onboarding, non-safety, model-touched marker)
|
||||
'first-task-scaffold': 'periodic',
|
||||
|
||||
// QA — gate for functional, periodic for quality/benchmarks
|
||||
'qa-quick': 'gate',
|
||||
'qa-b6-static': 'periodic',
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { execFileSync, execSync } from 'node:child_process';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
|
||||
// P4 first-run scaffold (activation lift). Two surfaces under test:
|
||||
// 1. bin/gstack-first-task-detect — classifies a repo into ONE enum bucket.
|
||||
// 2. The unified first-run-guidance preamble wiring (generated into SKILL.md).
|
||||
|
||||
const ROOT = path.join(import.meta.dir, '..');
|
||||
const DETECT = path.join(ROOT, 'bin', 'gstack-first-task-detect');
|
||||
|
||||
// The complete, closed set the detector is ever allowed to emit. The eval-safety
|
||||
// guarantee is that nothing outside this set ever reaches the preamble.
|
||||
const ENUM = new Set([
|
||||
'greenfield', 'code_node', 'code_python', 'code_rust', 'code_go',
|
||||
'code_ruby', 'code_ios', 'branch_ahead', 'dirty_default', 'clean_default', 'nongit',
|
||||
]);
|
||||
|
||||
const GIT_ENV = {
|
||||
...process.env,
|
||||
GIT_AUTHOR_NAME: 'T', GIT_AUTHOR_EMAIL: 't@e.x',
|
||||
GIT_COMMITTER_NAME: 'T', GIT_COMMITTER_EMAIL: 't@e.x',
|
||||
};
|
||||
|
||||
function detect(cwd: string): string {
|
||||
return execFileSync(DETECT, [], { cwd, encoding: 'utf-8', env: GIT_ENV }).trim();
|
||||
}
|
||||
function git(cwd: string, args: string) {
|
||||
execSync(`git ${args}`, { cwd, env: GIT_ENV, stdio: 'ignore' });
|
||||
}
|
||||
|
||||
let tmp: string;
|
||||
beforeAll(() => { tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ftd-')); });
|
||||
afterAll(() => { fs.rmSync(tmp, { recursive: true, force: true }); });
|
||||
|
||||
function freshRepo(name: string): string {
|
||||
const d = path.join(tmp, name);
|
||||
fs.mkdirSync(d, { recursive: true });
|
||||
git(d, 'init -q -b main');
|
||||
return d;
|
||||
}
|
||||
|
||||
describe('gstack-first-task-detect — bucket classification', () => {
|
||||
test('non-git directory → nongit', () => {
|
||||
const d = path.join(tmp, 'plain'); fs.mkdirSync(d, { recursive: true });
|
||||
expect(detect(d)).toBe('nongit');
|
||||
});
|
||||
|
||||
test('git repo, no commits → greenfield', () => {
|
||||
expect(detect(freshRepo('green'))).toBe('greenfield');
|
||||
});
|
||||
|
||||
test('Node project with a commit → code_node', () => {
|
||||
const d = freshRepo('node');
|
||||
fs.writeFileSync(path.join(d, 'package.json'), '{"name":"x"}');
|
||||
git(d, 'add -A'); git(d, 'commit -qm init');
|
||||
expect(detect(d)).toBe('code_node');
|
||||
});
|
||||
|
||||
test('Python project with a commit → code_python', () => {
|
||||
const d = freshRepo('py');
|
||||
fs.writeFileSync(path.join(d, 'pyproject.toml'), '[project]\nname="x"');
|
||||
git(d, 'add -A'); git(d, 'commit -qm init');
|
||||
expect(detect(d)).toBe('code_python');
|
||||
});
|
||||
|
||||
// The remaining language markers (a typo in any would ship undetected).
|
||||
for (const [name, file, token] of [
|
||||
['Rust', 'Cargo.toml', 'code_rust'],
|
||||
['Go', 'go.mod', 'code_go'],
|
||||
['Ruby', 'Gemfile', 'code_ruby'],
|
||||
] as const) {
|
||||
test(`${name} project with a commit → ${token}`, () => {
|
||||
const d = freshRepo(`lang-${token}`);
|
||||
fs.writeFileSync(path.join(d, file), 'x');
|
||||
git(d, 'add -A'); git(d, 'commit -qm init');
|
||||
expect(detect(d)).toBe(token);
|
||||
});
|
||||
}
|
||||
|
||||
test('iOS project (.xcodeproj) with a commit → code_ios', () => {
|
||||
const d = freshRepo('ios');
|
||||
fs.mkdirSync(path.join(d, 'App.xcodeproj'));
|
||||
fs.writeFileSync(path.join(d, 'App.xcodeproj', 'project.pbxproj'), '// x');
|
||||
git(d, 'add -A'); git(d, 'commit -qm init');
|
||||
expect(detect(d)).toBe('code_ios');
|
||||
});
|
||||
|
||||
// Precedence (the detector's most fragile logic): branch-state buckets must
|
||||
// win over language markers, so a real repo isn't mislabeled "verify tests".
|
||||
test('feature branch ahead + package.json → branch_ahead (not code_node)', () => {
|
||||
const origin = freshRepo('prec-origin');
|
||||
git(origin, 'commit -qm base --allow-empty');
|
||||
const clone = path.join(tmp, 'prec-clone');
|
||||
git(tmp, `clone -q ${origin} prec-clone`);
|
||||
fs.writeFileSync(path.join(clone, 'package.json'), '{"name":"x"}');
|
||||
git(clone, 'checkout -q -b feature');
|
||||
git(clone, 'add -A'); git(clone, 'commit -qm work');
|
||||
expect(detect(clone)).toBe('branch_ahead');
|
||||
});
|
||||
|
||||
test('dirty default branch + package.json → dirty_default (not code_node)', () => {
|
||||
const d = freshRepo('prec-dirty');
|
||||
fs.writeFileSync(path.join(d, 'package.json'), '{"name":"x"}');
|
||||
git(d, 'add -A'); git(d, 'commit -qm init');
|
||||
fs.writeFileSync(path.join(d, 'package.json'), '{"name":"x","v":2}');
|
||||
expect(detect(d)).toBe('dirty_default');
|
||||
});
|
||||
|
||||
test('feature branch ahead of origin → branch_ahead', () => {
|
||||
const origin = freshRepo('origin');
|
||||
git(origin, 'commit -qm base --allow-empty');
|
||||
const clone = path.join(tmp, 'clone');
|
||||
git(tmp, `clone -q ${origin} clone`);
|
||||
git(clone, 'checkout -q -b feature');
|
||||
fs.writeFileSync(path.join(clone, 'f.txt'), 'x');
|
||||
git(clone, 'add -A'); git(clone, 'commit -qm work');
|
||||
expect(detect(clone)).toBe('branch_ahead');
|
||||
});
|
||||
|
||||
test('uncommitted changes on default branch → dirty_default', () => {
|
||||
const d = freshRepo('dirty');
|
||||
fs.writeFileSync(path.join(d, 'a.txt'), 'x');
|
||||
git(d, 'add -A'); git(d, 'commit -qm init');
|
||||
fs.writeFileSync(path.join(d, 'a.txt'), 'changed');
|
||||
// No recognized language marker, so the dirty-default branch must win.
|
||||
expect(detect(d)).toBe('dirty_default');
|
||||
});
|
||||
|
||||
test('clean default branch, 5+ commits, no language marker → clean_default', () => {
|
||||
const d = freshRepo('clean');
|
||||
for (let i = 0; i < 6; i++) git(d, `commit -qm c${i} --allow-empty`);
|
||||
expect(detect(d)).toBe('clean_default');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gstack-first-task-detect — contract', () => {
|
||||
test('output is always a whitelisted enum token or empty (eval-safe)', () => {
|
||||
for (const name of ['plain', 'green', 'node', 'py', 'clone', 'dirty', 'clean']) {
|
||||
const out = detect(path.join(tmp, name));
|
||||
if (out !== '') expect(ENUM.has(out)).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('detector is executable', () => {
|
||||
expect(fs.statSync(DETECT).mode & 0o111).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('first-run-guidance preamble wiring (generated)', () => {
|
||||
const md = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
|
||||
|
||||
test('detection is gated to the first-ever run only (ACTIVATED=no, not headless)', () => {
|
||||
expect(md).toContain('if [ "$_ACTIVATED" = "no" ] && [ "$_SESSION_KIND" != "headless" ]');
|
||||
expect(md).toContain('gstack-first-task-detect');
|
||||
});
|
||||
|
||||
test('emits the unified first-run guidance section branching on ACTIVATED', () => {
|
||||
expect(md).toContain('## First-run guidance (one-time)');
|
||||
expect(md).toContain('`ACTIVATED` is `no`'); // P4 scaffold branch
|
||||
expect(md).toContain('`ACTIVATED` is `yes` AND `FIRST_LOOP_SHOWN` is `no`'); // P3 tip branch
|
||||
});
|
||||
|
||||
test('marks activated + logs the scaffold telemetry only on the shown path', () => {
|
||||
expect(md).toContain('first_task_scaffold_shown');
|
||||
expect(md).toContain('touch ~/.gstack/.activated');
|
||||
expect(md).toContain('touch ~/.gstack/.first-loop-tip-shown');
|
||||
});
|
||||
});
|
||||
@@ -84,9 +84,11 @@ Report what each command returned.`,
|
||||
}, 90_000);
|
||||
|
||||
testConcurrentIfSelected('skillmd-setup-discovery', async () => {
|
||||
const skillMd = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
// P2 (v1.2.0): the browse SETUP/binary-discovery block moved from the root
|
||||
// router to browse/SKILL.md (end anchor is now ## Core QA Patterns).
|
||||
const skillMd = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const setupStart = skillMd.indexOf('## SETUP');
|
||||
const setupEnd = skillMd.indexOf('## IMPORTANT');
|
||||
const setupEnd = skillMd.indexOf('## Core QA Patterns');
|
||||
const setupBlock = skillMd.slice(setupStart, setupEnd);
|
||||
|
||||
// Guard: verify we extracted a valid setup block
|
||||
@@ -116,9 +118,11 @@ Report whether it worked.`,
|
||||
// Create a tmpdir with no browse binary — no local .claude/skills/gstack/browse/dist/browse
|
||||
const emptyDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-empty-'));
|
||||
|
||||
const skillMd = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
// P2 (v1.2.0): the browse SETUP/binary-discovery block moved from the root
|
||||
// router to browse/SKILL.md (end anchor is now ## Core QA Patterns).
|
||||
const skillMd = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const setupStart = skillMd.indexOf('## SETUP');
|
||||
const setupEnd = skillMd.indexOf('## IMPORTANT');
|
||||
const setupEnd = skillMd.indexOf('## Core QA Patterns');
|
||||
const setupBlock = skillMd.slice(setupStart, setupEnd);
|
||||
|
||||
const result = await runSkillTest({
|
||||
@@ -151,9 +155,11 @@ Report the exact output. Do NOT try to fix or install anything — just report w
|
||||
// Create a tmpdir outside any git repo
|
||||
const nonGitDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-nogit-'));
|
||||
|
||||
const skillMd = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
// P2 (v1.2.0): the browse SETUP/binary-discovery block moved from the root
|
||||
// router to browse/SKILL.md (end anchor is now ## Core QA Patterns).
|
||||
const skillMd = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const setupStart = skillMd.indexOf('## SETUP');
|
||||
const setupEnd = skillMd.indexOf('## IMPORTANT');
|
||||
const setupEnd = skillMd.indexOf('## Core QA Patterns');
|
||||
const setupBlock = skillMd.slice(setupStart, setupEnd);
|
||||
|
||||
const result = await runSkillTest({
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* P4 first-run scaffold — E2E (periodic tier, ~$0.02 each, deterministic).
|
||||
*
|
||||
* Exercises bin/gstack-first-task-detect END-TO-END through the real runner +
|
||||
* hermetic env (path resolution, execution, git-in-cwd), not just the unit
|
||||
* harness. Deterministic by construction: it asserts the binary's enum token
|
||||
* from the Bash tool_result in the stream-json transcript (never the model's
|
||||
* prose), so it pins the detector's integration contract without depending on
|
||||
* non-deterministic model phrasing.
|
||||
*
|
||||
* Periodic (not gate): onboarding behavior is non-safety, and the scaffold
|
||||
* marker is model-touched (best-effort). The deterministic bucket logic itself
|
||||
* is fully covered by the unit test (test/preamble-first-task-scaffold.test.ts).
|
||||
*/
|
||||
|
||||
import { expect, afterAll } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import { execSync } from 'node:child_process';
|
||||
import { runSkillTest } from './helpers/session-runner';
|
||||
import {
|
||||
describeIfSelected, testIfSelected, createEvalCollector, finalizeEvalCollector,
|
||||
recordE2E, runId, logCost,
|
||||
} from './helpers/e2e-helpers';
|
||||
|
||||
const ROOT = path.join(import.meta.dir, '..');
|
||||
const DETECT = path.join(ROOT, 'bin', 'gstack-first-task-detect');
|
||||
const evalCollector = createEvalCollector('e2e-first-task-scaffold');
|
||||
const MODEL = 'claude-haiku-4-5-20251001';
|
||||
|
||||
const GIT_ENV = {
|
||||
...process.env,
|
||||
GIT_AUTHOR_NAME: 'T', GIT_AUTHOR_EMAIL: 't@e.x',
|
||||
GIT_COMMITTER_NAME: 'T', GIT_COMMITTER_EMAIL: 't@e.x',
|
||||
};
|
||||
|
||||
/** Concatenated Bash tool_result text from the stream-json transcript. */
|
||||
function toolResultText(transcript: any[]): string {
|
||||
const chunks: string[] = [];
|
||||
for (const event of transcript) {
|
||||
if (event.type !== 'user') continue;
|
||||
for (const item of event.message?.content ?? []) {
|
||||
if (item.type !== 'tool_result') continue;
|
||||
if (typeof item.content === 'string') chunks.push(item.content);
|
||||
else for (const c of item.content ?? []) if (c.type === 'text') chunks.push(c.text);
|
||||
}
|
||||
}
|
||||
return chunks.join('\n');
|
||||
}
|
||||
|
||||
async function detectVia(workDir: string, testName: string): Promise<string> {
|
||||
const result = await runSkillTest({
|
||||
prompt: `Run exactly this one bash command and then stop, printing its output verbatim: ${DETECT}`,
|
||||
workingDirectory: workDir,
|
||||
maxTurns: 3,
|
||||
allowedTools: ['Bash'],
|
||||
timeout: 120_000,
|
||||
testName,
|
||||
runId,
|
||||
model: MODEL,
|
||||
});
|
||||
logCost(testName, result);
|
||||
recordE2E(evalCollector, testName, 'e2e-first-task-scaffold', result);
|
||||
expect(result.exitReason).toBe('success');
|
||||
return toolResultText(result.transcript);
|
||||
}
|
||||
|
||||
describeIfSelected('first-run scaffold detection (E2E)', ['first-task-scaffold'], () => {
|
||||
testIfSelected('first-task-scaffold', async () => {
|
||||
if (!process.env.ANTHROPIC_API_KEY) {
|
||||
throw new Error('first-task-scaffold requires ANTHROPIC_API_KEY (source ~/.zshrc); refusing to skip');
|
||||
}
|
||||
|
||||
// code_node bucket: package.json + a commit, on the default branch.
|
||||
const nodeDir = fs.mkdtempSync(path.join(os.tmpdir(), 'fts-node-'));
|
||||
// greenfield bucket: git repo, zero commits.
|
||||
const greenDir = fs.mkdtempSync(path.join(os.tmpdir(), 'fts-green-'));
|
||||
try {
|
||||
execSync('git init -q -b main', { cwd: nodeDir, env: GIT_ENV });
|
||||
fs.writeFileSync(path.join(nodeDir, 'package.json'), '{"name":"x"}');
|
||||
execSync('git add -A && git commit -qm init', { cwd: nodeDir, env: GIT_ENV });
|
||||
execSync('git init -q -b main', { cwd: greenDir, env: GIT_ENV });
|
||||
|
||||
const nodeOut = await detectVia(nodeDir, 'first-task-scaffold');
|
||||
expect(nodeOut).toContain('code_node');
|
||||
|
||||
const greenOut = await detectVia(greenDir, 'first-task-scaffold-greenfield');
|
||||
expect(greenOut).toContain('greenfield');
|
||||
} finally {
|
||||
fs.rmSync(nodeDir, { recursive: true, force: true });
|
||||
fs.rmSync(greenDir, { recursive: true, force: true });
|
||||
}
|
||||
}, 300_000);
|
||||
});
|
||||
|
||||
afterAll(() => finalizeEvalCollector(evalCollector));
|
||||
+19
-17
@@ -65,10 +65,10 @@ describeIfSelected('LLM-as-judge quality evals', [
|
||||
], () => {
|
||||
testIfSelected('command reference table', async () => {
|
||||
const t0 = Date.now();
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const start = content.indexOf('## Command Reference');
|
||||
const end = content.indexOf('## Tips');
|
||||
const section = content.slice(start, end);
|
||||
// P2 (v1.2.0): the command reference moved from the root router to browse/SKILL.md.
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const start = content.indexOf('## Full Command List');
|
||||
const section = content.slice(start);
|
||||
|
||||
const scores = await judge('command reference table', section);
|
||||
console.log('Command reference scores:', JSON.stringify(scores, null, 2));
|
||||
@@ -94,9 +94,10 @@ describeIfSelected('LLM-as-judge quality evals', [
|
||||
|
||||
testIfSelected('snapshot flags reference', async () => {
|
||||
const t0 = Date.now();
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const start = content.indexOf('## Snapshot System');
|
||||
const end = content.indexOf('## Command Reference');
|
||||
// P2 (v1.2.0): snapshot flags moved from the root router to browse/SKILL.md.
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const start = content.indexOf('## Snapshot Flags');
|
||||
const end = content.indexOf('## CSS Inspector');
|
||||
const section = content.slice(start, end);
|
||||
|
||||
const scores = await judge('snapshot flags reference', section);
|
||||
@@ -145,9 +146,10 @@ describeIfSelected('LLM-as-judge quality evals', [
|
||||
|
||||
testIfSelected('setup block', async () => {
|
||||
const t0 = Date.now();
|
||||
const content = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
// P2 (v1.2.0): the browse setup block moved from the root router to browse/SKILL.md.
|
||||
const content = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const setupStart = content.indexOf('## SETUP');
|
||||
const setupEnd = content.indexOf('## IMPORTANT');
|
||||
const setupEnd = content.indexOf('## Core QA Patterns');
|
||||
const section = content.slice(setupStart, setupEnd);
|
||||
|
||||
const scores = await judge('setup/binary discovery instructions', section);
|
||||
@@ -172,10 +174,10 @@ describeIfSelected('LLM-as-judge quality evals', [
|
||||
|
||||
testIfSelected('regression vs baseline', async () => {
|
||||
const t0 = Date.now();
|
||||
const generated = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const genStart = generated.indexOf('## Command Reference');
|
||||
const genEnd = generated.indexOf('## Tips');
|
||||
const genSection = generated.slice(genStart, genEnd);
|
||||
// P2 (v1.2.0): the command reference moved from the root router to browse/SKILL.md.
|
||||
const generated = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const genStart = generated.indexOf('## Full Command List');
|
||||
const genSection = generated.slice(genStart);
|
||||
|
||||
const baseline = `## Command Reference
|
||||
|
||||
@@ -480,10 +482,10 @@ describeIfSelected('Baseline score pinning', ['baseline score pinning'], () => {
|
||||
const baselines = JSON.parse(fs.readFileSync(baselinesPath, 'utf-8'));
|
||||
const regressions: string[] = [];
|
||||
|
||||
const skillContent = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
const cmdStart = skillContent.indexOf('## Command Reference');
|
||||
const cmdEnd = skillContent.indexOf('## Tips');
|
||||
const cmdSection = skillContent.slice(cmdStart, cmdEnd);
|
||||
// P2 (v1.2.0): the command reference moved from the root router to browse/SKILL.md.
|
||||
const skillContent = fs.readFileSync(path.join(ROOT, 'browse', 'SKILL.md'), 'utf-8');
|
||||
const cmdStart = skillContent.indexOf('## Full Command List');
|
||||
const cmdSection = skillContent.slice(cmdStart);
|
||||
const cmdScores = await judge('command reference table', cmdSection);
|
||||
|
||||
for (const dim of ['clarity', 'completeness', 'actionability'] as const) {
|
||||
|
||||
@@ -26,15 +26,18 @@ function readShipUnion(): string {
|
||||
}
|
||||
|
||||
describe('SKILL.md command validation', () => {
|
||||
test('all $B commands in SKILL.md are valid browse commands', () => {
|
||||
// P2 (v1.2.0): the top-level gstack skill is a pure ROUTER, not the browse
|
||||
// skill. The browse body lives only in browse/SKILL.md now. This regression
|
||||
// pins the split: the router carries routing rules and zero browse commands,
|
||||
// while browse/SKILL.md still advertises the full QA surface (asserted below).
|
||||
test('top-level SKILL.md is a router with no browse body (P2)', () => {
|
||||
const md = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
|
||||
expect(md).not.toContain('gstack browse: QA Testing'); // browse body removed
|
||||
expect(md).toContain('## Route first'); // router head present
|
||||
expect(md).toContain('invoke `/investigate`'); // routing rules present
|
||||
const result = validateSkill(path.join(ROOT, 'SKILL.md'));
|
||||
expect(result.invalid).toHaveLength(0);
|
||||
expect(result.valid.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('all snapshot flags in SKILL.md are valid', () => {
|
||||
const result = validateSkill(path.join(ROOT, 'SKILL.md'));
|
||||
expect(result.snapshotFlagErrors).toHaveLength(0);
|
||||
expect(result.invalid).toHaveLength(0); // no INVALID browse commands
|
||||
expect(result.valid.length).toBe(0); // and no browse commands at all — it routes, not browses
|
||||
});
|
||||
|
||||
test('all $B commands in browse/SKILL.md are valid browse commands', () => {
|
||||
|
||||
Reference in New Issue
Block a user