Files
gstack/test/preamble-first-task-scaffold.test.ts
T
Garry Tan 11de390be1 v1.58.5.0 feat: first-run activation scaffold + gstack router front door (#2078)
* feat: first-run activation — project-aware scaffold, router front door, onboarding nudges

Adds the activation system that drives a new install toward a concrete first move:
- bin/gstack-first-task-detect: local-git+filesystem repo classifier emitting one
  validated enum bucket (greenfield/code_<lang>/branch_ahead/dirty_default/clean_default),
  portable timeouts, fail-safe empty output.
- generate-first-run-guidance.ts: unified preamble section — first-run project-aware
  scaffold + returning-session plan->review->ship tip, gated on a persistent .activated
  marker and never run in headless. Detection wired lazily in generate-preamble-bash.ts.
- SKILL.md.tmpl: top-level gstack skill is now a pure router (browse body removed; it
  lives in /browse), routing any request and sending browser/QA work to /browse.
- setup: first-move nudge on first install. office-hours: closing handoff that launches
  the next review via the Skill tool.
- telemetry-ingest: accept onboarding/first_task_scaffold_shown/handoff/route event types.

* test: cover first-run detection + repoint browse-content assertions to /browse

- New unit tests for every detection bucket, the eval-safe enum contract, and the
  first-run gating (test/preamble-first-task-scaffold.test.ts); periodic E2E that runs
  the detector through the real harness (test/skill-e2e-first-task-scaffold.test.ts).
- Repoint browse-content assertions (gen-skill-docs, audit-compliance, skill-validation,
  LLM-judge eval) from the root skill to browse/SKILL.md following the router split;
  add a regression pinning that the router carries no browse body.
- Register first-task-scaffold touchfiles + periodic tier; bump parity/carve size caps
  ~1-2KB per skill for the shared first-run-guidance preamble section.
- Refresh ship golden fixtures for the preamble addition.

* chore: regenerate SKILL.md + llms.txt for first-run activation

* chore: bump version and changelog (v1.58.5.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(test): repoint bws skillmd-* setup-block assertions to browse/SKILL.md

The skillmd-setup-discovery / -no-local-binary / -outside-git E2E tests extracted
the `## SETUP`→`## IMPORTANT` browse binary-discovery block from the root SKILL.md.
P2 moved that block to browse/SKILL.md (end anchor is now `## Core QA Patterns`),
so the slice came back empty and the `browse/dist/browse` guard failed. Repoint to
browse/SKILL.md. Verified: 7/7 e2e-browse pass locally.

* fix(test): tolerate skill-discovery race in PTY plan-mode smoke

The e2e-pty-plan-smoke suite (office-hours / plan-mode-no-op) failed in CI with
`Unknown command: /office-hours` (claude exited ~10s) while passing locally. Root
cause: a cold CI container's overlay-FS scan of the symlinked ~/.claude/skills
registry finishes AFTER the runner's 8s boot grace, so the first `/skill` send
reaches claude before the skill is indexed and is rejected as unknown. The runner
gave up on the first "Unknown command:" line.

runPlanSkillObservation now re-sends the skill command up to 3x (6s apart),
re-marking the buffer each time so stale scrollback can't re-trip the check,
before concluding the skill is genuinely unregistered. A real dangling-symlink /
missing-skill still surfaces as 'exited' (after retries), preserving the original
diagnostic. Pure-helper contract unchanged: 95/95 unit tests pass.

This is a pre-existing harness bug (fails identically on #2077's own branch, which
introduced the suite) surfaced while shipping the activation feature.

* debug(ci): temporarily instrument pty-smoke skill discovery

Capture claude version, env, registry tree, and a claude -p discovery probe to
pin why /office-hours isn't discovered in CI (retries proved it's not a race).
Temporary — revert once the registry fix is identified.

* chore: revert pty-smoke harness experiments (race-retry + CI debug step)

Diagnosis is conclusive and the experiments aren't the fix, so restore the
harness to its original state (net-zero diff vs main for both files).

What the CI debug step proved: `claude -p` returns READY — claude v2.1.187 fully
DISCOVERS /office-hours from the symlinked registry. Only the interactive PTY TUI
rejects it as "Unknown command" (and it received the full command text). So the
e2e-pty-plan-smoke failure is a claude 2.1.187 interactive-TUI regression (skills
discovered by `claude -p` aren't exposed as TUI slash commands), pre-existing in
the #2077 harness and failing identically on its own origin branch — unrelated to
this activation PR. The race-retry can't help (the TUI genuinely lacks the
command); the debug step also tripped actionlint (shellcheck SC2012). Both reverted.

* fix(ci): copy SKILL.md as real files in pty-smoke registry (cross-mount symlink)

The e2e-pty-plan-smoke suite failed with "Unknown command: /office-hours" in CI
while passing locally. Root cause (proven, not guessed): claude 2.1.187's
interactive-TUI skill scanner does not follow the /github/home -> /__w cross-mount
symlink the registry used for per-skill SKILL.md. Evidence: a CI debug step showed
`claude -p` discovered the skill (printed READY), and a local macOS repro with the
identical symlinked registry recognized /office-hours — isolating the failure to
the container's cross-mount symlink, not registration content, claude version,
duplicate names, or a race.

Fix: register the per-skill SKILL.md + sections as REAL copies (same mount as
$HOME) so the TUI reads them directly. The gstack root stays a symlink — the
preamble's runtime bash resolves bin/* and sections/* through it and bash follows
cross-mount symlinks fine.

* fix(ci): guard rm expansion in pty-smoke registry (shellcheck SC2115)

* fix(ci): also register pty-smoke skills project-scoped (cwd/.claude/skills)

The real-file user-dir registration still left the TUI rejecting /office-hours in
the container. claude's interactive TUI surfaces /slash commands from the PROJECT
dir (<cwd>/.claude/skills); the smokes run with cwd=$REPO whose .claude/skills is
gitignored (absent on a fresh CI checkout), so the user-dir registry feeds
`claude -p` (READY) but not the TUI. Populate $REPO/.claude/skills with real
SKILL.md + sections copies (no gstack symlink there — it would point at its own
parent; runtime paths use the user-dir gstack symlink).

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 09:42:45 -07:00

172 lines
6.7 KiB
TypeScript

import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { execFileSync, execSync } from 'node:child_process';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
// P4 first-run scaffold (activation lift). Two surfaces under test:
// 1. bin/gstack-first-task-detect — classifies a repo into ONE enum bucket.
// 2. The unified first-run-guidance preamble wiring (generated into SKILL.md).
const ROOT = path.join(import.meta.dir, '..');
const DETECT = path.join(ROOT, 'bin', 'gstack-first-task-detect');
// The complete, closed set the detector is ever allowed to emit. The eval-safety
// guarantee is that nothing outside this set ever reaches the preamble.
const ENUM = new Set([
'greenfield', 'code_node', 'code_python', 'code_rust', 'code_go',
'code_ruby', 'code_ios', 'branch_ahead', 'dirty_default', 'clean_default', 'nongit',
]);
const GIT_ENV = {
...process.env,
GIT_AUTHOR_NAME: 'T', GIT_AUTHOR_EMAIL: 't@e.x',
GIT_COMMITTER_NAME: 'T', GIT_COMMITTER_EMAIL: 't@e.x',
};
function detect(cwd: string): string {
return execFileSync(DETECT, [], { cwd, encoding: 'utf-8', env: GIT_ENV }).trim();
}
function git(cwd: string, args: string) {
execSync(`git ${args}`, { cwd, env: GIT_ENV, stdio: 'ignore' });
}
let tmp: string;
beforeAll(() => { tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ftd-')); });
afterAll(() => { fs.rmSync(tmp, { recursive: true, force: true }); });
function freshRepo(name: string): string {
const d = path.join(tmp, name);
fs.mkdirSync(d, { recursive: true });
git(d, 'init -q -b main');
return d;
}
describe('gstack-first-task-detect — bucket classification', () => {
test('non-git directory → nongit', () => {
const d = path.join(tmp, 'plain'); fs.mkdirSync(d, { recursive: true });
expect(detect(d)).toBe('nongit');
});
test('git repo, no commits → greenfield', () => {
expect(detect(freshRepo('green'))).toBe('greenfield');
});
test('Node project with a commit → code_node', () => {
const d = freshRepo('node');
fs.writeFileSync(path.join(d, 'package.json'), '{"name":"x"}');
git(d, 'add -A'); git(d, 'commit -qm init');
expect(detect(d)).toBe('code_node');
});
test('Python project with a commit → code_python', () => {
const d = freshRepo('py');
fs.writeFileSync(path.join(d, 'pyproject.toml'), '[project]\nname="x"');
git(d, 'add -A'); git(d, 'commit -qm init');
expect(detect(d)).toBe('code_python');
});
// The remaining language markers (a typo in any would ship undetected).
for (const [name, file, token] of [
['Rust', 'Cargo.toml', 'code_rust'],
['Go', 'go.mod', 'code_go'],
['Ruby', 'Gemfile', 'code_ruby'],
] as const) {
test(`${name} project with a commit → ${token}`, () => {
const d = freshRepo(`lang-${token}`);
fs.writeFileSync(path.join(d, file), 'x');
git(d, 'add -A'); git(d, 'commit -qm init');
expect(detect(d)).toBe(token);
});
}
test('iOS project (.xcodeproj) with a commit → code_ios', () => {
const d = freshRepo('ios');
fs.mkdirSync(path.join(d, 'App.xcodeproj'));
fs.writeFileSync(path.join(d, 'App.xcodeproj', 'project.pbxproj'), '// x');
git(d, 'add -A'); git(d, 'commit -qm init');
expect(detect(d)).toBe('code_ios');
});
// Precedence (the detector's most fragile logic): branch-state buckets must
// win over language markers, so a real repo isn't mislabeled "verify tests".
test('feature branch ahead + package.json → branch_ahead (not code_node)', () => {
const origin = freshRepo('prec-origin');
git(origin, 'commit -qm base --allow-empty');
const clone = path.join(tmp, 'prec-clone');
git(tmp, `clone -q ${origin} prec-clone`);
fs.writeFileSync(path.join(clone, 'package.json'), '{"name":"x"}');
git(clone, 'checkout -q -b feature');
git(clone, 'add -A'); git(clone, 'commit -qm work');
expect(detect(clone)).toBe('branch_ahead');
});
test('dirty default branch + package.json → dirty_default (not code_node)', () => {
const d = freshRepo('prec-dirty');
fs.writeFileSync(path.join(d, 'package.json'), '{"name":"x"}');
git(d, 'add -A'); git(d, 'commit -qm init');
fs.writeFileSync(path.join(d, 'package.json'), '{"name":"x","v":2}');
expect(detect(d)).toBe('dirty_default');
});
test('feature branch ahead of origin → branch_ahead', () => {
const origin = freshRepo('origin');
git(origin, 'commit -qm base --allow-empty');
const clone = path.join(tmp, 'clone');
git(tmp, `clone -q ${origin} clone`);
git(clone, 'checkout -q -b feature');
fs.writeFileSync(path.join(clone, 'f.txt'), 'x');
git(clone, 'add -A'); git(clone, 'commit -qm work');
expect(detect(clone)).toBe('branch_ahead');
});
test('uncommitted changes on default branch → dirty_default', () => {
const d = freshRepo('dirty');
fs.writeFileSync(path.join(d, 'a.txt'), 'x');
git(d, 'add -A'); git(d, 'commit -qm init');
fs.writeFileSync(path.join(d, 'a.txt'), 'changed');
// No recognized language marker, so the dirty-default branch must win.
expect(detect(d)).toBe('dirty_default');
});
test('clean default branch, 5+ commits, no language marker → clean_default', () => {
const d = freshRepo('clean');
for (let i = 0; i < 6; i++) git(d, `commit -qm c${i} --allow-empty`);
expect(detect(d)).toBe('clean_default');
});
});
describe('gstack-first-task-detect — contract', () => {
test('output is always a whitelisted enum token or empty (eval-safe)', () => {
for (const name of ['plain', 'green', 'node', 'py', 'clone', 'dirty', 'clean']) {
const out = detect(path.join(tmp, name));
if (out !== '') expect(ENUM.has(out)).toBe(true);
}
});
test('detector is executable', () => {
expect(fs.statSync(DETECT).mode & 0o111).toBeGreaterThan(0);
});
});
describe('first-run-guidance preamble wiring (generated)', () => {
const md = fs.readFileSync(path.join(ROOT, 'ship', 'SKILL.md'), 'utf-8');
test('detection is gated to the first-ever run only (ACTIVATED=no, not headless)', () => {
expect(md).toContain('if [ "$_ACTIVATED" = "no" ] && [ "$_SESSION_KIND" != "headless" ]');
expect(md).toContain('gstack-first-task-detect');
});
test('emits the unified first-run guidance section branching on ACTIVATED', () => {
expect(md).toContain('## First-run guidance (one-time)');
expect(md).toContain('`ACTIVATED` is `no`'); // P4 scaffold branch
expect(md).toContain('`ACTIVATED` is `yes` AND `FIRST_LOOP_SHOWN` is `no`'); // P3 tip branch
});
test('marks activated + logs the scaffold telemetry only on the shown path', () => {
expect(md).toContain('first_task_scaffold_shown');
expect(md).toContain('touch ~/.gstack/.activated');
expect(md).toContain('touch ~/.gstack/.first-loop-tip-shown');
});
});