diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e549ecaf..ffafc1473 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,39 @@ # Changelog +## [1.87.4.0] - 2026-09-16 + +**Failed checks stay failed.** +**Health scores show what actually ran.** + +`/health` now keeps each checker's exit status and counts diagnostics from its complete output. Reports still show only the final 50 log lines. Scores name the checked and unavailable categories, so a partial run carries its coverage beside the number. Runs with no checks produce no numeric score or history entry. + +### The three numbers that matter + +Source: the synthetic checker in `test/health-capture.test.ts`, which emits 60 type errors followed by 80 context lines and exits 2. These measurements compare the v1.87.3.0 capture example with this version under default Bash without `pipefail`. Run `bun test test/health-capture.test.ts` to verify current behavior. These are correctness measurements, not production statistics. + +| Metric | Before | After | Δ | +|---|---:|---:|---:| +| Reported checker exit status | 0 | 2 | +2 | +| Type errors available for scoring | 0 | 60 | +60 | +| Displayed checker log lines | 50 | 50 | 0 | + +The failing checker no longer looks successful because `tail` succeeded. All 60 errors count even when the displayed tail contains only context. + +### What this means for developers + +You can distinguish a score backed by several checks from one based on a single available tool. Empty runs report `N/A — no checks ran`; capture errors also remain unscored and leave existing history unchanged. Trends compare only matching categories, so installing a new checker does not manufacture a regression or improvement. Run `/health` to see the score and its coverage together. + +### Itemized changes + +#### Fixed + +- **`/health` preserves failed checks and complete diagnostic counts.** Reports show the final 50 output lines while scoring the full log and the checker's actual exit status. Temporary capture errors remain explicit errors. +- **Health scores disclose coverage.** Partial runs list checked and unavailable categories. Runs with no checks report `N/A — no checks ran`, leave numeric history unchanged, and have no trend. Score comparisons require matching categories. + +#### Changed + +- Routing evaluations choose among installed GStack skills, keeping built-in CLI skills outside the evaluated catalog. The model still chooses by matching the request to each skill's description. + ## [1.87.3.0] - 2026-09-15 **Changed code needs another pass.** diff --git a/VERSION b/VERSION index 01aa5a296..bf58942b8 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.87.3.0 +1.87.4.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 98f0816ff..aacc09e01 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.87.3.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.87.4.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/health/SKILL.md b/health/SKILL.md index 56d948dab..39f086aed 100644 --- a/health/SKILL.md +++ b/health/SKILL.md @@ -489,22 +489,55 @@ section in CLAUDE.md: Run each detected tool. For each tool: 1. Record the start time -2. Run the command, capturing both stdout and stderr -3. Record the exit code +2. Run the command, capturing complete stdout and stderr in a private temporary log +3. Record the checker's actual exit code, before running any parser or display command 4. Record the end time -5. Capture the last 50 lines of output for the report +5. Parse counts from the complete log, then display its last 50 lines for the report ```bash -# Example for each tool — run each independently -START=$(date +%s) -tsc --noEmit 2>&1 | tail -50 -EXIT_CODE=$? -END=$(date +%s) -echo "TOOL:typecheck EXIT:$EXIT_CODE DURATION:$((END-START))s" +# Capture example — run each tool independently; adapt the command and parser. +( + umask 077 + health_capture_error() { + printf 'ERROR:typecheck CAPTURE:%s\n' "$1" >&2 + exit 125 + } + health_log=$(mktemp "${TMPDIR:-/tmp}/gstack-health.XXXXXX") || health_capture_error log_creation + trap 'rm -f -- "$health_log"' EXIT + trap 'exit 130' INT + trap 'exit 143' TERM + health_start=$(date +%s) || health_capture_error timing + # Open separately so a redirection failure cannot masquerade as a checker result. + if ! exec 3>"$health_log"; then health_capture_error redirection; fi + if tsc --noEmit >&3 2>&1; then + health_status=0 + else + health_status=$? + fi + exec 3>&- + health_end=$(date +%s) || health_capture_error timing + # awk returns a successful zero count for no matches, including empty output. + health_count=$(awk '/error TS/ { count++ } END { print count+0 }' "$health_log") || health_capture_error parsing + tail -50 "$health_log" || health_capture_error display + printf 'TOOL:typecheck EXIT:%s DURATION:%ss ERRORS:%s\n' "$health_status" "$((health_end-health_start))" "$health_count" + exit "$health_status" +) ``` -Run tools sequentially (some may share resources or lock files). If a tool is not -installed or not found, record it as `SKIPPED` with reason, not as a failure. +Run tools sequentially in independent invocations (some may share resources or lock +files). A failed checker must not prevent later tools from running. Remember each +reported exit code and full-log counts; never use the status of `tail` or a parser +as the checker result. Guard parsers whose no-match exit is expected. + +Before running a tool, check availability using the project's configured command +and local tool installation. If availability detection establishes that it is +missing, record `SKIPPED` with the reason. An executed checker returning 127 is a +failure, not evidence that the category should be skipped. + +Capture failures (log creation, redirection, parsing, or display) are `ERROR`, never +`CLEAN` or `SKIPPED`. Include the cause and do not invent a category score. Report +the composite as `N/A — capture failed` if any category cannot be scored for this +reason; do not redistribute that category's weight or persist a numeric history row. --- @@ -522,6 +555,9 @@ Score each category on a 0-10 scale using this rubric: | GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) | **Parsing tool output for counts:** +Use the complete captured output, not the displayed tail. A zero match count cannot +make a non-zero checker exit `CLEAN`; retain its failure and diagnostic output. + - **tsc:** Count lines matching `error TS` in output. - **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available. - **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures). @@ -537,6 +573,11 @@ If a category is skipped (tool not available — includes GBrain when gbrain is not installed), redistribute its weight proportionally among the remaining categories. +Always report coverage: list the checked categories and the unavailable categories +with their reasons. Label a numeric composite with skipped categories as **partial +coverage**. If zero checks executed, report **N/A — no checks ran**, do not compute +a numeric composite, and skip numeric history persistence and trend calculation. + **GBrain sub-score computation (D6):** ``` @@ -579,6 +620,9 @@ Shell lint shellcheck 10/10 CLEAN 1s 0 issues GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago COMPOSITE SCORE: 9.1 / 10 +Coverage: 6/6 categories checked +Checked: typecheck, lint, test, deadcode, shell, gbrain +Unavailable: none Duration: 23s total ``` @@ -588,6 +632,13 @@ Use these status labels: - 7-9: `WARNING` - 4-6: `NEEDS WORK` - 0-3: `CRITICAL` +- Unavailable tool: `SKIPPED` (no score) +- Capture failure: `ERROR` (no score; composite is N/A) + +For partial coverage, show e.g. `COMPOSITE SCORE: 8.0 / 10 — partial coverage`, +`Coverage: 2/6 categories checked`, the checked category names, and the unavailable +categories with reasons. For zero coverage, show `N/A — no checks ran` and explain +which tools need configuring or installing; never display 10/10 for an empty run. If any category scored below 7, list the top issues from that tool's output: @@ -607,7 +658,9 @@ DETAILS: Lint (3 warnings) eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gstack/projects/$SLUG ``` -Append one JSONL line to `~/.gstack/projects/$SLUG/health-history.jsonl`: +Only when a numeric composite exists, append one JSONL line to +`~/.gstack/projects/$SLUG/health-history.jsonl`. Zero-check and capture-error runs +must leave any existing history unchanged: ```json {"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23} @@ -636,7 +689,14 @@ eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gst tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY" ``` -**If prior entries exist, show the trend:** +**Compare like-for-like coverage.** For each history row, form the set of categories +with non-null scores (missing fields count as null). Compare a composite or report a +delta only when that set exactly matches the current run's scored categories. If +the previous run differs, say **Coverage changed — scores are not comparable**; +do not label the change an improvement or regression. Historical rows may still be +shown with their unavailable categories marked. A current N/A result has no trend. + +**If comparable prior entries exist, show the trend:** ``` HEALTH TREND (last 5 runs) @@ -651,7 +711,7 @@ Date Branch Score TC Lint Test Dead Shell GBrain Trend: IMPROVING (+0.9 since last run) ``` -**If score dropped vs the previous run:** +**If score dropped vs the previous run with identical coverage:** 1. Identify WHICH categories declined 2. Show the delta for each declining category 3. Correlate with tool output -- what specific errors/warnings appeared? @@ -689,7 +749,7 @@ Rank by `weight * (10 - score)` descending. Only show categories below 10. 1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports. 2. **Read-only.** Never fix issues. Present the dashboard and let the user decide. 3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess. -4. **Skipped is not failed.** If a tool isn't available, skip it gracefully and redistribute weight. Do not penalize the score. +4. **Skipped is not failed.** Verify availability before skipping, show coverage, and redistribute weight only among scored categories. An executed command's failure must not become a skip. 5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running. -6. **Trends require history.** On first run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." +6. **Trends require comparable history.** On the first scored run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." Changed coverage and N/A runs have no score delta. 7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality. diff --git a/health/SKILL.md.tmpl b/health/SKILL.md.tmpl index f92eb7347..a60800522 100644 --- a/health/SKILL.md.tmpl +++ b/health/SKILL.md.tmpl @@ -114,22 +114,55 @@ section in CLAUDE.md: Run each detected tool. For each tool: 1. Record the start time -2. Run the command, capturing both stdout and stderr -3. Record the exit code +2. Run the command, capturing complete stdout and stderr in a private temporary log +3. Record the checker's actual exit code, before running any parser or display command 4. Record the end time -5. Capture the last 50 lines of output for the report +5. Parse counts from the complete log, then display its last 50 lines for the report ```bash -# Example for each tool — run each independently -START=$(date +%s) -tsc --noEmit 2>&1 | tail -50 -EXIT_CODE=$? -END=$(date +%s) -echo "TOOL:typecheck EXIT:$EXIT_CODE DURATION:$((END-START))s" +# Capture example — run each tool independently; adapt the command and parser. +( + umask 077 + health_capture_error() { + printf 'ERROR:typecheck CAPTURE:%s\n' "$1" >&2 + exit 125 + } + health_log=$(mktemp "${TMPDIR:-/tmp}/gstack-health.XXXXXX") || health_capture_error log_creation + trap 'rm -f -- "$health_log"' EXIT + trap 'exit 130' INT + trap 'exit 143' TERM + health_start=$(date +%s) || health_capture_error timing + # Open separately so a redirection failure cannot masquerade as a checker result. + if ! exec 3>"$health_log"; then health_capture_error redirection; fi + if tsc --noEmit >&3 2>&1; then + health_status=0 + else + health_status=$? + fi + exec 3>&- + health_end=$(date +%s) || health_capture_error timing + # awk returns a successful zero count for no matches, including empty output. + health_count=$(awk '/error TS/ { count++ } END { print count+0 }' "$health_log") || health_capture_error parsing + tail -50 "$health_log" || health_capture_error display + printf 'TOOL:typecheck EXIT:%s DURATION:%ss ERRORS:%s\n' "$health_status" "$((health_end-health_start))" "$health_count" + exit "$health_status" +) ``` -Run tools sequentially (some may share resources or lock files). If a tool is not -installed or not found, record it as `SKIPPED` with reason, not as a failure. +Run tools sequentially in independent invocations (some may share resources or lock +files). A failed checker must not prevent later tools from running. Remember each +reported exit code and full-log counts; never use the status of `tail` or a parser +as the checker result. Guard parsers whose no-match exit is expected. + +Before running a tool, check availability using the project's configured command +and local tool installation. If availability detection establishes that it is +missing, record `SKIPPED` with the reason. An executed checker returning 127 is a +failure, not evidence that the category should be skipped. + +Capture failures (log creation, redirection, parsing, or display) are `ERROR`, never +`CLEAN` or `SKIPPED`. Include the cause and do not invent a category score. Report +the composite as `N/A — capture failed` if any category cannot be scored for this +reason; do not redistribute that category's weight or persist a numeric history row. --- @@ -147,6 +180,9 @@ Score each category on a 0-10 scale using this rubric: | GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) | **Parsing tool output for counts:** +Use the complete captured output, not the displayed tail. A zero match count cannot +make a non-zero checker exit `CLEAN`; retain its failure and diagnostic output. + - **tsc:** Count lines matching `error TS` in output. - **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available. - **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures). @@ -162,6 +198,11 @@ If a category is skipped (tool not available — includes GBrain when gbrain is not installed), redistribute its weight proportionally among the remaining categories. +Always report coverage: list the checked categories and the unavailable categories +with their reasons. Label a numeric composite with skipped categories as **partial +coverage**. If zero checks executed, report **N/A — no checks ran**, do not compute +a numeric composite, and skip numeric history persistence and trend calculation. + **GBrain sub-score computation (D6):** ``` @@ -204,6 +245,9 @@ Shell lint shellcheck 10/10 CLEAN 1s 0 issues GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago COMPOSITE SCORE: 9.1 / 10 +Coverage: 6/6 categories checked +Checked: typecheck, lint, test, deadcode, shell, gbrain +Unavailable: none Duration: 23s total ``` @@ -213,6 +257,13 @@ Use these status labels: - 7-9: `WARNING` - 4-6: `NEEDS WORK` - 0-3: `CRITICAL` +- Unavailable tool: `SKIPPED` (no score) +- Capture failure: `ERROR` (no score; composite is N/A) + +For partial coverage, show e.g. `COMPOSITE SCORE: 8.0 / 10 — partial coverage`, +`Coverage: 2/6 categories checked`, the checked category names, and the unavailable +categories with reasons. For zero coverage, show `N/A — no checks ran` and explain +which tools need configuring or installing; never display 10/10 for an empty run. If any category scored below 7, list the top issues from that tool's output: @@ -232,7 +283,9 @@ DETAILS: Lint (3 warnings) {{SLUG_SETUP}} ``` -Append one JSONL line to `~/.gstack/projects/$SLUG/health-history.jsonl`: +Only when a numeric composite exists, append one JSONL line to +`~/.gstack/projects/$SLUG/health-history.jsonl`. Zero-check and capture-error runs +must leave any existing history unchanged: ```json {"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23} @@ -261,7 +314,14 @@ file exists and has prior entries). tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY" ``` -**If prior entries exist, show the trend:** +**Compare like-for-like coverage.** For each history row, form the set of categories +with non-null scores (missing fields count as null). Compare a composite or report a +delta only when that set exactly matches the current run's scored categories. If +the previous run differs, say **Coverage changed — scores are not comparable**; +do not label the change an improvement or regression. Historical rows may still be +shown with their unavailable categories marked. A current N/A result has no trend. + +**If comparable prior entries exist, show the trend:** ``` HEALTH TREND (last 5 runs) @@ -276,7 +336,7 @@ Date Branch Score TC Lint Test Dead Shell GBrain Trend: IMPROVING (+0.9 since last run) ``` -**If score dropped vs the previous run:** +**If score dropped vs the previous run with identical coverage:** 1. Identify WHICH categories declined 2. Show the delta for each declining category 3. Correlate with tool output -- what specific errors/warnings appeared? @@ -314,7 +374,7 @@ Rank by `weight * (10 - score)` descending. Only show categories below 10. 1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports. 2. **Read-only.** Never fix issues. Present the dashboard and let the user decide. 3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess. -4. **Skipped is not failed.** If a tool isn't available, skip it gracefully and redistribute weight. Do not penalize the score. +4. **Skipped is not failed.** Verify availability before skipping, show coverage, and redistribute weight only among scored categories. An executed command's failure must not become a skip. 5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running. -6. **Trends require history.** On first run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." +6. **Trends require comparable history.** On the first scored run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." Changed coverage and N/A runs have no score delta. 7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality. diff --git a/package.json b/package.json index cdd3a7883..37168b8fe 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.87.3", + "version": "1.87.4", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/test/dx-manual-handoff-ao.test.ts b/test/dx-manual-handoff-ao.test.ts index 40f6589a6..4407d58bd 100644 --- a/test/dx-manual-handoff-ao.test.ts +++ b/test/dx-manual-handoff-ao.test.ts @@ -29,7 +29,7 @@ describe('AO completed manual DX handoff preserves report freshness',()=>{ expect(E2E_TOUCHFILES[owner]).toContain('test/fixtures/dx-manual-handoff-ao.json'); } const arrays=[...Object.values(E2E_TOUCHFILES),...Object.values(LLM_JUDGE_TOUCHFILES),GLOBAL_TOUCHFILES]; - expect(arrays).toHaveLength(210); + expect(arrays).toHaveLength(211); for(const values of arrays)for(let i=0;i{ diff --git a/test/health-capture.test.ts b/test/health-capture.test.ts new file mode 100644 index 000000000..5569d0f05 --- /dev/null +++ b/test/health-capture.test.ts @@ -0,0 +1,92 @@ +import { afterEach, describe, expect, test } from 'bun:test'; +import { spawnSync } from 'node:child_process'; +import { chmodSync, mkdtempSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +// Execute the documented capture, so the example cannot drift from its tests. +const template = readFileSync(join(import.meta.dir, '../health/SKILL.md.tmpl'), 'utf8'); +const capture = template.split('## Step 2: Run Tools')[1].match(/```bash\n([\s\S]*?)```/)![1]; +const temporaryDirectories: string[] = []; + +afterEach(() => { + for (const directory of temporaryDirectories.splice(0)) rmSync(directory, { recursive: true, force: true }); +}); + +function runCapture(checker: string, setup = '') { + const directory = mkdtempSync(join(tmpdir(), 'health-capture-test-')); + temporaryDirectories.push(directory); + const logs = join(directory, 'logs'); + mkdirSync(logs); + const executable = join(directory, 'tsc'); + writeFileSync(executable, '#!/usr/bin/env bash\n' + checker + '\n'); + chmodSync(executable, 0o755); + const result = spawnSync('/bin/bash', ['-euc', setup + '\n' + capture], { + encoding: 'utf8', + timeout: 10_000, + env: { ...process.env, PATH: directory + ':' + process.env.PATH, TMPDIR: logs }, + }); + expect(result.error).toBeUndefined(); + expect(readdirSync(logs)).toEqual([]); + return result; +} + +describe('/health command capture', () => { + test('a successful empty checker reports zero matches under set -e', () => { + const result = runCapture('exit 0'); + expect(result.status).toBe(0); + expect(result.stdout).toMatch(/TOOL:typecheck EXIT:0 DURATION:\d+s ERRORS:0/); + }); + + test('failure survives an earlier success, stderr capture, and display', () => { + const result = runCapture('echo "source.ts: error TS2322: incorrect type" >&2\nexit 2', 'true'); + expect(result.status).toBe(2); + expect(result.stdout).toContain('incorrect type'); + expect(result.stdout).toMatch(/EXIT:2 DURATION:\d+s ERRORS:1/); + }); + + test('counts findings outside the displayed tail and prints only fifty log lines', () => { + const result = runCapture([ + 'for ((i=1; i<=60; i++)); do echo "source.ts: error TS2322: finding $i"; done', + 'for ((i=1; i<=80; i++)); do echo "detail $i"; done', + 'exit 2', + ].join('\n')); + expect(result.status).toBe(2); + expect(result.stdout).toMatch(/EXIT:2 DURATION:\d+s ERRORS:60/); + const lines = result.stdout.trimEnd().split('\n'); + expect(lines.length).toBe(51); + expect(lines[0]).toBe('detail 31'); + expect(lines[49]).toBe('detail 80'); + }); + + test('an executed checker returning 127 remains a failure', () => { + const result = runCapture('echo "a checker dependency failed" >&2\nexit 127'); + expect(result.status).toBe(127); + expect(result.stdout).toContain('EXIT:127'); + expect(result.stdout).not.toContain('SKIPPED'); + }); + + test('empty failing output does not inherit a successful parser status', () => { + const result = runCapture('exit 1'); + expect(result.status).toBe(1); + expect(result.stdout).toMatch(/EXIT:1 DURATION:\d+s ERRORS:0/); + }); + + test('capture log permissions are private even with a permissive caller umask', () => { + const result = runCapture('stat -c %a "$TMPDIR"/gstack-health.* 2>/dev/null || stat -f %Lp "$TMPDIR"/gstack-health.*', 'umask 000'); + expect(result.status).toBe(0); + expect(result.stdout.split('\n')[0]).toBe('600'); + }); + + test.each([ + ['log_creation', 'mktemp() { return 1; }'], + ['redirection', 'mktemp() { printf "%s/missing/log\\n" "$TMPDIR"; }'], + ['parsing', 'awk() { return 2; }'], + ['display', 'tail() { return 1; }'], + ])('%s failure reports an error instead of a clean result', (phase, setup) => { + const result = runCapture('exit 0', setup); + expect(result.status).toBe(125); + expect(result.stderr).toContain('ERROR:typecheck CAPTURE:' + phase); + expect(result.stdout).not.toContain('TOOL:typecheck EXIT:0'); + }); +}); diff --git a/test/health-eval-fixture.test.ts b/test/health-eval-fixture.test.ts new file mode 100644 index 000000000..b1aa5e360 --- /dev/null +++ b/test/health-eval-fixture.test.ts @@ -0,0 +1,139 @@ +/** Free fixture/recording checks; never import or invoke the paid runner. */ +import { afterEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { + createHealthEvalFixture, healthReportingFailures, recordHealthAttempt, +} from './helpers/health-eval-fixture'; +import type { SkillTestResult } from './helpers/session-runner'; +import type { EvalTestEntry } from './helpers/eval-store'; +import { E2E_TIERS, E2E_TOUCHFILES, GLOBAL_TOUCHFILES, selectTests } from './helpers/touchfiles'; +import { isPaidTestFile } from './helpers/paid-test-set'; + +const ROOT = path.resolve(import.meta.dir, '..'); +const dirs: string[] = []; +afterEach(() => { for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); }); + +test('health behavior is selected as periodic and paid while capture regressions stay free', () => { + const selectedBy = (file: string) => selectTests([file], E2E_TOUCHFILES, GLOBAL_TOUCHFILES).selected; + expect(selectedBy('health/SKILL.md.tmpl')).toContain('health-reporting'); + expect(selectedBy('test/helpers/health-eval-fixture.ts')).toEqual(['health-reporting']); + expect(E2E_TIERS['health-reporting']).toBe('periodic'); + expect(isPaidTestFile('test/skill-e2e-health.test.ts')).toBe(true); + expect(isPaidTestFile('test/health-capture.test.ts')).toBe(false); +}); + +function fixture(prefix = 'health-fixture-test-') { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); + dirs.push(dir); + return createHealthEvalFixture(dir, ROOT); +} + +function writePassingEvidence(f: ReturnType) { + fs.writeFileSync(path.join(f.dir, 'partial-report.md'), `Type check: 0/10 CRITICAL, 60 errors. +Tests: 10/10 CLEAN, 5 passed. +Checked: type check, tests. Unavailable: lint, dead code, shell lint, GBrain (not installed). +COMPOSITE SCORE: **5.6 / 10 — partial coverage** +Coverage changed; prior test-only history is not comparable. +`); + fs.writeFileSync(path.join(f.dir, 'no-tools-report.md'), 'COMPOSITE SCORE: N/A — no checks ran.\n'); + fs.appendFileSync(path.join(f.gstackHome, 'projects', 'partial', 'health-history.jsonl'), JSON.stringify({ + score: 5.6, typecheck: 0, test: 10, lint: null, deadcode: null, shell: null, gbrain: null, + }) + '\n'); + fs.writeFileSync(f.receipts, 'typecheck\ntest\n'); +} + +describe('/health eval fixtures', () => { + test('extracts all six real workflow steps and redirects persistent paths', () => { + const f = fixture(); + const skill = fs.readFileSync(path.join(f.dir, 'health-SKILL.md'), 'utf-8'); + expect(skill).not.toContain('## Preamble (run first)'); + expect(skill).not.toContain('~/.gstack'); + expect(skill).not.toContain('~/.claude/skills/gstack/bin/gstack-slug'); + for (let step = 1; step <= 6; step++) expect(skill).toContain(`## Step ${step}:`); + for (const project of ['partial', 'no-tools']) { + const result = spawnSync(path.join(f.dir, 'bin', 'gstack-slug'), [], { + cwd: path.join(f.dir, project), encoding: 'utf-8', timeout: 10_000, + env: { ...process.env, GSTACK_HOME: f.gstackHome, GSTACK_PROJECT_SLUG: '' }, + }); + expect(result.status).toBe(0); + expect(result.stdout).toContain(`SLUG=${project}\n`); + } + }); + + test('real checker exits nonzero and its final 50 lines hide all 60 errors', () => { + const f = fixture(); + const result = spawnSync('bash', ['-c', 'bash ./check-typecheck.sh 2>&1'], { + cwd: path.join(f.dir, 'partial'), encoding: 'utf-8', timeout: 10_000, + }); + expect(result.status).toBe(2); + expect(result.stdout.match(/error TS/g)).toHaveLength(60); + expect(result.stdout.trimEnd().split('\n').slice(-50).join('\n')).not.toContain('error TS'); + expect(fs.readFileSync(f.receipts, 'utf-8')).toBe('typecheck\n'); + }); + + test('both checkers record to their fixed path without shell environment setup', () => { + const f = fixture("health-fixture-test-' space-"); + for (const [script, status] of [['check-typecheck.sh', 2], ['check-tests.sh', 0]] as const) { + const result = spawnSync('bash', [script], { + cwd: path.join(f.dir, 'partial'), encoding: 'utf-8', timeout: 10_000, + }); + expect(result.status).toBe(status); + } + expect(fs.readFileSync(f.receipts, 'utf-8')).toBe('typecheck\ntest\n'); + }); + + test('accepts truthful reports and history, rejects absent evidence', () => { + const f = fixture(); + expect(healthReportingFailures(f).length).toBeGreaterThan(0); + writePassingEvidence(f); + expect(healthReportingFailures(f)).toEqual([]); + }); + + test.each(['false-clean', 'false-clean-category', 'tail-count', 'numeric-empty', 'hidden-coverage', 'unlike-trend', 'contradictory-trend', 'extra-empty-history', 'missing-receipts', 'project-edit'])('%s is a failed behavior eval', failure => { + const f = fixture(); + writePassingEvidence(f); + const report = path.join(f.dir, 'partial-report.md'); + if (failure === 'false-clean') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('5.6', '10')); + if (failure === 'false-clean-category') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('0/10 CRITICAL', '10/10 CLEAN')); + if (failure === 'tail-count') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('60 errors', '0 errors')); + if (failure === 'numeric-empty') fs.writeFileSync(path.join(f.dir, 'no-tools-report.md'), 'COMPOSITE SCORE: 10 — no checks ran.'); + if (failure === 'hidden-coverage') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace(' — partial coverage', '')); + if (failure === 'unlike-trend') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('Coverage changed; prior test-only history is not comparable.', 'Health worsened from 10 to 5.6.')); + if (failure === 'contradictory-trend') fs.appendFileSync(report, 'Trend: REGRESSING (-4.4 since last run)\n'); + if (failure === 'extra-empty-history') fs.appendFileSync(path.join(f.gstackHome, 'projects', 'no-tools', 'health-history.jsonl'), '{"score":10}\n'); + if (failure === 'missing-receipts') fs.rmSync(f.receipts); + if (failure === 'project-edit') fs.appendFileSync(path.join(f.dir, 'partial', 'check-typecheck.sh'), '# attempted fix\n'); + expect(healthReportingFailures(f).length).toBeGreaterThan(0); + }); +}); + +describe('/health eval recording', () => { + const success = { + exitReason: 'success', duration: 5, output: 'report', transcript: [], + costEstimate: { estimatedCost: 0.1, turnsUsed: 2, estimatedTokens: 100 }, model: 'fixture', + } as SkillTestResult; + + test.each(['success', 'assertion', 'timeout', 'throw'])('%s records exactly once with matching pass status', async scenario => { + const entries: EvalTestEntry[] = []; + let verified = false; + const attempt = recordHealthAttempt( + entry => entries.push(entry), + async () => { + if (scenario === 'throw') throw new Error('runner failed'); + return { ...success, exitReason: scenario === 'timeout' ? 'timeout' : 'success' }; + }, + () => { verified = true; if (scenario === 'assertion') throw new Error('false dashboard'); }, + ); + if (scenario === 'success') await attempt; + else await expect(attempt).rejects.toThrow(); + expect(entries).toHaveLength(1); + expect(entries[0].passed).toBe(scenario === 'success'); + expect(entries[0].tier).toBe('e2e'); + expect(verified).toBe(scenario === 'success' || scenario === 'assertion'); + if (scenario === 'assertion') expect(entries[0].output).toContain('false dashboard'); + if (scenario === 'timeout') expect(entries[0].exit_reason).toBe('timeout'); + }); +}); diff --git a/test/helpers/health-eval-fixture.ts b/test/helpers/health-eval-fixture.ts new file mode 100644 index 000000000..5abafefec --- /dev/null +++ b/test/helpers/health-eval-fixture.ts @@ -0,0 +1,209 @@ +/** Isolated fixtures and assertions for the single periodic /health capture. */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { extractSkillSections } from './skill-fixture'; +import type { SkillTestResult } from './session-runner'; +import type { EvalTestEntry } from './eval-store'; + +export const HEALTH_EVAL_ID = 'health-reporting'; +export const HEALTH_EVAL_SECTIONS = [ + 'Step 1: Detect Health Stack', + 'Step 2: Run Tools', + 'Step 3: Score Each Category', + 'Step 4: Present Dashboard', + 'Step 5: Persist to Health History', + 'Step 6: Trend Analysis + Recommendations', + 'Important Rules', +]; + +const PRIOR_HISTORY = JSON.stringify({ + ts: '2026-01-01T00:00:00Z', branch: 'unknown', score: 10, + typecheck: null, lint: null, test: 10, deadcode: null, shell: null, + gbrain: null, duration_s: 1, +}) + '\n'; + +export interface HealthEvalFixture { + dir: string; + gstackHome: string; + receipts: string; + prompt: string; + projectFiles: Record; +} + +function projectSnapshot(dir: string): Record { + const files: Record = {}; + const walk = (relative: string) => { + for (const entry of fs.readdirSync(path.join(dir, relative), { withFileTypes: true })) { + const name = path.join(relative, entry.name); + if (entry.isDirectory()) { files[name + '/'] = ''; walk(name); } + else files[name] = entry.isSymbolicLink() ? `` : fs.readFileSync(path.join(dir, name), 'utf-8'); + } + }; + for (const project of ['partial', 'no-tools']) walk(project); + return files; +} + +export function createHealthEvalFixture(dir: string, repoRoot: string): HealthEvalFixture { + const gstackHome = path.join(dir, 'gstack-state'); + const receipts = path.join(dir, 'checker-runs.txt'); + // Instrumentation belongs to the fixture, not to the model's shell setup. + // Single-quote escaping also covers temporary paths containing apostrophes. + const quotedReceipts = "'" + receipts.replaceAll("'", "'\"'\"'") + "'"; + fs.mkdirSync(path.join(dir, 'bin'), { recursive: true }); + fs.copyFileSync(path.join(repoRoot, 'bin', 'gstack-slug'), path.join(dir, 'bin', 'gstack-slug')); + fs.chmodSync(path.join(dir, 'bin', 'gstack-slug'), 0o755); + + // Only redirect installed paths. The workflow and score/history rules come + // from the real generated skill, without its unrelated shared preamble. + const skill = extractSkillSections(path.join(repoRoot, 'health'), HEALTH_EVAL_SECTIONS) + .replaceAll('~/.claude/skills/gstack/bin/gstack-slug', path.join(dir, 'bin', 'gstack-slug')) + .replaceAll('~/.gstack', gstackHome); + fs.writeFileSync(path.join(dir, 'health-SKILL.md'), skill); + + for (const project of ['partial', 'no-tools']) { + const projectDir = path.join(dir, project); + fs.mkdirSync(projectDir); + fs.writeFileSync(path.join(projectDir, '.project.yaml'), `name: ${project}\n`); + const historyDir = path.join(gstackHome, 'projects', project); + fs.mkdirSync(historyDir, { recursive: true }); + fs.writeFileSync(path.join(historyDir, 'health-history.jsonl'), PRIOR_HISTORY); + } + fs.writeFileSync(path.join(dir, 'partial', 'CLAUDE.md'), `# Partial project + +## Health Stack + +- typecheck: bash ./check-typecheck.sh +- test: bash ./check-tests.sh + +Only the listed tools are available. Lint, dead-code, shell-lint, and GBrain +tools are not installed. This configuration is final; do not install tools. +`); + fs.writeFileSync(path.join(dir, 'no-tools', 'CLAUDE.md'), `# No-tools project + +## Health Stack + +No health tools are configured or installed for any category. +This configuration is final; do not install tools or substitute other checks. +`); + fs.writeFileSync(path.join(dir, 'partial', 'check-typecheck.sh'), `#!/usr/bin/env bash +printf 'typecheck\\n' >> ${quotedReceipts} +for ((i = 1; i <= 60; i++)); do + printf 'src/file%s.ts(1,1): error TS2322: Type mismatch.\\n' "$i" >&2 +done +for ((i = 1; i <= 80; i++)); do + printf 'Additional diagnostic context %s\\n' "$i" +done +exit 2 +`); + fs.writeFileSync(path.join(dir, 'partial', 'check-tests.sh'), `#!/usr/bin/env bash +printf 'test\\n' >> ${quotedReceipts} +printf '5 pass\\n0 fail\\n' +`); + + return { + dir, gstackHome, receipts, projectFiles: projectSnapshot(dir), + prompt: `Read health-SKILL.md and run its /health workflow, Steps 1–6, for +the partial project first and the no-tools project second. Each has its own +CLAUDE.md with the final Health Stack configuration. Run commands from the +corresponding project directory. These are local fixtures without Git remotes. + +Save each complete dashboard, details, trends, and recommendations as +partial-report.md or no-tools-report.md in ${dir}. Keep the dashboard's +COMPOSITE SCORE label. Existing health histories are available under +${gstackHome}/projects//health-history.jsonl; apply the skill's normal +history rules. GSTACK_HOME already points at this isolated state directory. + +Do not modify project files, install tools, or ask to change either Health Stack. +Only write the reports and any history updates required by the supplied skill. +Finish after producing both reports.`, + }; +} + +/** Validate behavior, allowing ordinary Markdown/wording variation in reports. */ +export function healthReportingFailures(fixture: HealthEvalFixture): string[] { + const failures: string[] = []; + const check = (ok: boolean, message: string) => { if (!ok) failures.push(message); }; + const read = (file: string) => fs.existsSync(file) ? fs.readFileSync(file, 'utf-8') : ''; + const plain = (text: string) => text.replace(/[*_`]/g, ''); + const partial = plain(read(path.join(fixture.dir, 'partial-report.md'))); + const empty = plain(read(path.join(fixture.dir, 'no-tools-report.md'))); + const composite = (report: string) => report.match(/composite\s+score\s*[:|]?\s*(N\/A|\d+(?:\.\d+)?)/i)?.[1]; + check(composite(partial) === '5.6', 'partial coverage must score 5.6, preserving the failing typecheck'); + check(/partial\s+coverage/i.test(partial), 'numeric score must be labeled as partial coverage'); + const typecheckRows = partial.split('\n').filter(line => /\btype\s*check\b/i.test(line)); + check(typecheckRows.some(line => /(?:\b0\s*\/\s*10\b|\|\s*0\s*\|)/.test(line) + && /\b(?:critical|fail(?:ed|ure)?|error)\b/i.test(line) && !/\bclean\b/i.test(line)), + 'dashboard must report the failing typecheck as 0/10, not clean'); + check(/\b60\s+(?:\w+\s+){0,2}(?:errors|diagnostics|findings)\b/i.test(partial) + || /(?:errors|diagnostics|findings)[^\n]{0,20}\b60\b/i.test(partial), + 'report must count all 60 errors before the final 50 output lines'); + check(/(?:coverage|checked|executed)/i.test(partial), 'partial report must disclose checked coverage'); + check(/type\s*check/i.test(partial) && /tests?/i.test(partial), 'checked category names must be visible'); + for (const category of ['lint', 'dead[ -]?code', 'shell(?:[ -]?lint)?', 'gbrain']) { + const unavailable = '(?:unavailable|skipped|not (?:installed|configured|available|found))'; + check(new RegExp(`${unavailable}[\\s\\S]{0,240}${category}|${category}[^\\n]{0,120}${unavailable}`, 'i').test(partial), + `partial report must name unavailable ${category}`); + } + check(/(?:coverage|categor(?:y|ies)|checks)[\s\S]{0,160}(?:chang|differ|not compar)/i.test(partial) + || /(?:chang|differ|not compar)[\s\S]{0,160}(?:coverage|categor(?:y|ies)|checks)/i.test(partial), + 'partial report must flag changed coverage instead of comparing unlike histories'); + check(!/[+-]\s*4\.4\b|trend\s*:\s*(?:improving|regressing|worsening)/i.test(partial), + 'partial report must not calculate a trend delta against different coverage'); + check(composite(empty)?.toUpperCase() === 'N/A', 'no-tools composite must be N/A'); + check(/(?:no|zero|0)\s+(?:health\s+)?checks?\s+(?:ran|run|executed|available)|no\s+tools/i.test(empty), + 'no-tools report must explain that no checks ran'); + + const history = read(path.join(fixture.gstackHome, 'projects', 'partial', 'health-history.jsonl')); + check(history.startsWith(PRIOR_HISTORY), 'partial run must preserve its prior history row'); + const rows = history.trim().split('\n').filter(Boolean); + check(rows.length === 2, 'partial run must append exactly one history row'); + try { + const row = JSON.parse(rows.at(-1) || '{}'); + check(row.score === 5.6 && row.typecheck === 0 && row.test === 10, + 'persisted partial scores must reflect all diagnostics and the actual exit status'); + check(['lint', 'deadcode', 'shell', 'gbrain'].every(category => row[category] === null), + 'unavailable categories must persist as null'); + } catch { + failures.push('partial history must remain valid JSONL'); + } + check(read(path.join(fixture.gstackHome, 'projects', 'no-tools', 'health-history.jsonl')) === PRIOR_HISTORY, + 'no-tools run must leave its history unchanged'); + const runs = read(fixture.receipts).trim().split('\n'); + check(runs.includes('typecheck') && runs.includes('test'), 'both configured checkers must actually run'); + check(JSON.stringify(projectSnapshot(fixture.dir)) === JSON.stringify(fixture.projectFiles), + 'health must leave project files unchanged'); + return failures; +} + +/** One attempt records once, after all assertions; throws remain failures. */ +export async function recordHealthAttempt( + record: (entry: EvalTestEntry) => void, + capture: () => Promise, + verify: (result: SkillTestResult) => void, +): Promise { + const started = Date.now(); + let result: SkillTestResult | undefined; + let passed = false; + let failure: unknown; + try { + result = await capture(); + if (result.exitReason !== 'success') throw new Error(`health capture ended: ${result.exitReason}`); + verify(result); + passed = true; + } catch (error) { + failure = error; + throw error; + } finally { + record({ + name: HEALTH_EVAL_ID, suite: 'health', tier: 'e2e', passed, + duration_ms: result?.duration ?? Date.now() - started, + cost_usd: result?.costEstimate.estimatedCost ?? 0, + turns_used: result?.costEstimate.turnsUsed, + tokens_used: result?.costEstimate.estimatedTokens, + transcript: result?.transcript, + output: [result?.output, failure === undefined ? '' : String(failure)].filter(Boolean).join('\n').slice(-4000), + exit_reason: result?.exitReason === 'success' && !passed ? 'assertion_failed' : result?.exitReason ?? 'runner_error', + model: result?.model, + }); + } +} diff --git a/test/helpers/touchfiles-data.ts b/test/helpers/touchfiles-data.ts index eb242881d..4df9b970d 100644 --- a/test/helpers/touchfiles-data.ts +++ b/test/helpers/touchfiles-data.ts @@ -582,6 +582,9 @@ export const E2E_TOUCHFILES: Record = { // Document-release 'document-release': ['document-release/**', 'test/skill-e2e-workflow.test.ts'], + // /health result capture, coverage, and comparable history (model behavior). + 'health-reporting': ['health/**', 'test/skill-e2e-health.test.ts', 'test/helpers/health-eval-fixture.ts'], + // Codex (Claude E2E — tests /codex skill via Claude) 'codex-review': ['codex/**', 'test/skill-e2e-workflow.test.ts'], @@ -1186,6 +1189,7 @@ export const E2E_TIERS: Record = { // Document-release — gate (CHANGELOG guardrail) 'document-release': 'gate', + 'health-reporting': 'periodic', // Codex — periodic (Opus, requires codex CLI) 'codex-review': 'periodic', diff --git a/test/plan-count-file-permission.test.ts b/test/plan-count-file-permission.test.ts index 9797b394f..7ce8eec6f 100644 --- a/test/plan-count-file-permission.test.ts +++ b/test/plan-count-file-permission.test.ts @@ -69,7 +69,7 @@ describe('native repeated report permission identity',()=>{ for (const variant of ['basic', 'intervening', 'cropped', 'same-basename', 'path-cropped']) test.skipIf(process.platform==='win32')(`real fake CLI grants each current request once: ${variant}`,async()=>{ const intervening = variant === 'intervening'; - const dir=fs.mkdtempSync(path.join(os.tmpdir(),'count-edit-pty-'));const fake=path.join(dir,'fake-claude');const worker=path.join(dir,'worker.ts');const events=path.join(dir,'events.jsonl');const output=path.join(dir,'output.json');const expected=path.join(dir,variant==='same-basename'?'PLAN.md':'report.md');fs.writeFileSync(expected,'original'); + const dir=fs.mkdtempSync(path.join(os.tmpdir(),'ce-'));const fake=path.join(dir,'fake-claude');const worker=path.join(dir,'worker.ts');const events=path.join(dir,'events.jsonl');const output=path.join(dir,'output.json');const expected=path.join(dir,variant==='same-basename'?'PLAN.md':'report.md');fs.writeFileSync(expected,'original'); const cropped=capturedAc.rows.find(row=>row.job===5)!; let screen=variant==='path-cropped' ? capturedPath.screen.replace(capturedPath.screen.split('\n')[0]!,expected).replaceAll(path.dirname(capturedPath.expected),path.dirname(expected)).replaceAll(path.basename(capturedPath.expected),'report.md') : variant==='cropped' ? cropped.screen.replaceAll(path.dirname(cropped.hook.expected),path.dirname(expected)).replaceAll(path.basename(cropped.hook.expected),'report.md') diff --git a/test/plan-count-quoted-frame-ak.test.ts b/test/plan-count-quoted-frame-ak.test.ts index 8a9695f81..9a612b5d4 100644 --- a/test/plan-count-quoted-frame-ak.test.ts +++ b/test/plan-count-quoted-frame-ak.test.ts @@ -10,6 +10,26 @@ import owned from './fixtures/plan-count-owned-permission-v.json'; import {E2E_TOUCHFILES,selectTests} from './helpers/touchfiles'; const quote=(s:string)=>s.split('\n').map(row=>'> '+row).join('\n'); +// A PTY transports bytes, not command-sized stdin events. Share the framing +// code with the fake CLI so fragmented grants exercise the same receiver. +function commandBuffer(){ + let pending=''; + return (chunk:string)=>{ + pending+=chunk;const commands:string[]=[];let end:number; + while((end=pending.indexOf('\r'))!==-1){commands.push(pending.slice(0,end+1));pending=pending.slice(end+1);} + return commands; + }; +} +test('fake CLI preserves command bytes across fragmented and coalesced PTY input',()=>{ + const expected=['/plan-ceo-review\r','1\r']; + for(const chunks of [expected,['/plan-ceo-review\r','1','\r'],[...expected.join('')],[expected.join('')]]){ + const receive=commandBuffer();expect(chunks.flatMap(receive)).toEqual(expected); + } + const receive=commandBuffer(); + expect(receive('1')).toEqual([]);expect(receive('\r2\rtrailing')).toEqual(['1\r','2\r']); + expect(receive('\r')).toEqual(['trailing\r']); // no unexpected bytes are discarded +}); + test('the exact wholly quoted AK pane is handled so the dispatcher sends no fallback',()=>{ const screen=quote(exact.screen); expect(classifyPlanCountFrame(screen)).toBe('permission'); @@ -44,7 +64,7 @@ test('the regression selects exactly the existing permission consumers',()=>{ test.skipIf(process.platform==='win32')('real dispatcher ignores quoted pane then grants the fresh owned native request once',async()=>{ const dir=fs.mkdtempSync(path.join(os.tmpdir(),'count-quoted-frame-')),fake=path.join(dir,'fake-claude'),worker=path.join(dir,'worker.ts'),events=path.join(dir,'events.jsonl'),output=path.join(dir,'result.json'),report=path.join(dir,'report.md'); fs.writeFileSync(report,'original'); - fs.writeFileSync(fake,`#!${process.execPath}\n`+String.raw` + fs.writeFileSync(fake,`#!${process.execPath}\nconst receive=(${commandBuffer.toString()})();\n`+String.raw` import fs from 'node:fs';import path from 'node:path'; const item=JSON.parse(process.env.QUOTED_FRAME_CASE),sid='quoted-frame-main',log=e=>fs.appendFileSync(item.events,JSON.stringify(e)+'\n'); const transcript=path.join(process.env.CLAUDE_CONFIG_DIR,'projects','owned',sid+'.jsonl');fs.mkdirSync(path.dirname(transcript),{recursive:true}); @@ -56,14 +76,14 @@ const hook=async name=>{for(const entry of settings.hooks[name]??[]){if(entry.ma const p=Bun.spawn(['bash','-c',entry.hooks[0].command],{stdin:new Blob([JSON.stringify(event)]),stdout:'pipe',stderr:'pipe'}); const [code,out,err]=await Promise.all([p.exited,new Response(p.stdout).text(),new Response(p.stderr).text()]);if(code||out||err)throw Error('Hook failed');}}; const pane=item.screen.replaceAll('PLAN.md',item.report),paint=s=>process.stdout.write('\x1b[2J\x1b[H'+s.replaceAll('\n','\r\n')); -let stage='startup';process.stdin.setRawMode?.(true);process.stdin.on('data',async data=>{ - const input=data.toString();log({type:'input',stage,input}); +let stage='startup';process.stdin.setRawMode?.(true);const dispatch=async input=>{ + log({type:'input',stage,input}); if(stage==='startup'){stage='quoted';paint(item.quotedScreen);setTimeout(async()=>{await hook('PreToolUse');stage='current';paint(pane);},4200);return;} if(stage!=='current'){log({type:'unexpected'});return;} if(input!=='1\r')throw Error('One-time grant changed');stage='done';await hook('PostToolUse'); const q={header:'Finding',question:'Apply the reviewed fix?',options:[{label:'Fix'},{label:'Keep'}]}; native('assistant',[{type:'tool_use',name:'AskUserQuestion',id:'finding',input:{questions:[q]}}]);native('user',[{type:'tool_result',tool_use_id:'finding',content:'Answered'}],{toolUseResult:{answers:{[q.question]:'Fix'}}});paint('Done.\n'); -});process.on('SIGINT',()=>process.exit(0));process.stdin.resume(); +};process.stdin.on('data',async data=>{const chunk=data.toString();log({type:'chunk',stage,input:chunk});for(const input of receive(chunk))await dispatch(input);});process.on('SIGINT',()=>process.exit(0));process.stdin.resume(); `);fs.chmodSync(fake,0o755); // Keep every physical terminal row inside the quote; adding a prefix to an // already120-column capture would otherwise wrap an unquoted continuation. @@ -73,7 +93,10 @@ let stage='startup';process.stdin.setRawMode?.(true);process.stdin.on('data',asy const child=Bun.spawn([process.execPath,worker],{env:{...process.env,BROWSE_TERMINAL_BINARY:fake,EVALS_HERMETIC:'1'},stdout:'pipe',stderr:'pipe'}),timer=setTimeout(()=>child.kill('SIGKILL'),30000); try{const [code,out,err]=await Promise.all([child.exited,new Response(child.stdout).text(),new Response(child.stderr).text()]);expect(code,out+err).toBe(0); const result=JSON.parse(fs.readFileSync(output,'utf8')),rows=fs.readFileSync(events,'utf8').trim().split('\n').map(s=>JSON.parse(s)); - expect(result.outcome,JSON.stringify(result)).toBe('ceiling_reached');expect(result.reviewCount).toBe(1); + expect(result.outcome,JSON.stringify({result,rows})).toBe('ceiling_reached');expect(result.reviewCount).toBe(1); + const chunks=rows.filter(r=>r.type==='chunk'); + expect(chunks.every(r=>['startup','current'].includes(r.stage))).toBe(true); + expect(chunks.map(r=>r.input).join('')).toBe('/plan-ceo-review\r1\r'); expect(rows.filter(r=>r.type==='input').map(r=>[r.stage,r.input])).toEqual([['startup','/plan-ceo-review\r'],['current','1\r']]);expect(rows.some(r=>r.type==='unexpected')).toBe(false); expect(()=>process.kill(rows[0].pid,0)).toThrow();expect(fs.existsSync(rows[0].cwd)).toBe(false); }finally{clearTimeout(timer);child.kill('SIGKILL');await child.exited; diff --git a/test/skill-e2e-health.test.ts b/test/skill-e2e-health.test.ts new file mode 100644 index 000000000..a381ee4b6 --- /dev/null +++ b/test/skill-e2e-health.test.ts @@ -0,0 +1,56 @@ +/** + * Periodic /health behavior: full-log failure counts, visible partial coverage, + * comparable histories, and an unscored no-tools run. One bounded capture; + * wording follows the model, while process receipts and history are asserted. + */ +import { afterAll, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CAPTURE_MS, CAPTURE_LONG_MS } from './helpers/eval-budgets'; +import { describeE2ETier, e2eTierEnabled } from './helpers/e2e-gate'; +import { EvalCollector } from './helpers/eval-store'; +import { + createHealthEvalFixture, healthReportingFailures, recordHealthAttempt, +} from './helpers/health-eval-fixture'; + +const ROOT = path.resolve(import.meta.dir, '..'); +const describeE2E = describeE2ETier('periodic'); +const collector = e2eTierEnabled('periodic') ? new EvalCollector('e2e') : null; + +describeE2E('/health trustworthy reporting (periodic)', () => { + test('health-reporting', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-health-eval-')); + try { + let fixture: ReturnType; + await recordHealthAttempt( + entry => collector!.addTest(entry), + async () => { + fixture = createHealthEvalFixture(dir, ROOT); + // The runner resolves its transcript directory on import. Keep a + // skipped paid file free of that operator-state lookup/write. + const { runSkillTest } = await import('./helpers/session-runner'); + return runSkillTest({ + prompt: fixture.prompt, + workingDirectory: dir, + maxTurns: 18, + allowedTools: ['Bash', 'Read', 'Write', 'Glob', 'Grep'], + timeout: CAPTURE_MS, + testName: 'health-reporting', + // The collector retains the transcript in GSTACK_EVAL_DIR. Omit + // runId so the runner does not write a global heartbeat/run log. + env: { GSTACK_HOME: fixture.gstackHome }, + }); + }, + result => { + expect(result.browseErrors).toEqual([]); + expect(healthReportingFailures(fixture)).toEqual([]); + }, + ); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }, CAPTURE_LONG_MS); +}); + +afterAll(async () => { await collector?.finalize(); }); diff --git a/test/skill-routing-e2e.test.ts b/test/skill-routing-e2e.test.ts index cdf91463b..438af6189 100644 --- a/test/skill-routing-e2e.test.ts +++ b/test/skill-routing-e2e.test.ts @@ -79,6 +79,7 @@ function installSkills(tmpDir: string) { ]; const targetBase = path.join(tmpDir, '.claude', 'skills'); + const installedSkills: string[] = []; for (const skill of skillDirs) { const srcPath = path.join(ROOT, skill, 'SKILL.md'); @@ -88,8 +89,11 @@ function installSkills(tmpDir: string) { const destDir = path.join(targetBase, skillName); fs.mkdirSync(destDir, { recursive: true }); fs.writeFileSync(path.join(destDir, 'SKILL.md'), extractSkillHead(srcPath)); + installedSkills.push(skillName); } + // The names-only catalog keeps new CLI built-ins from changing the candidate + // set. Descriptions still choose the skill; no request-to-skill answer key. // Write a CLAUDE.md with a GENERIC invoke-skills nudge — deliberately NO // per-skill routing table. These journey tests exist to catch skill // DESCRIPTION regressions (their touchfiles key on */SKILL.md.tmpl), and @@ -102,7 +106,11 @@ function installSkills(tmpDir: string) { ## Skill routing -When the user's request matches an available skill, ALWAYS invoke it using the Skill +This project uses the following installed gstack skills: ${installedSkills.join(', ')}. +Choose among this project catalog by matching the request to the skill descriptions. +The CLI's built-in skills are outside this project's workflow. + +When the user's request matches an available project skill, ALWAYS invoke it using the Skill tool as your FIRST action. Do NOT answer directly, do NOT use other tools first. The skill has specialized workflows that produce better results than ad-hoc answers. Choose the skill by matching the request against each skill's description.