v1.87.4.0 fix: preserve health failures and disclose coverage (#2882)

* fix: report health failures and coverage accurately

* chore: prepare health reporting release 1.87.4.0

* test: restrict routing evaluations to installed project skills

* test: stabilize health selection and terminal fixtures
This commit is contained in:
Garry Tan
2026-09-15 21:35:11 -07:00
committed by GitHub
parent 85b8c038fc
commit a6b3a57512
15 changed files with 728 additions and 43 deletions
+34
View File
@@ -1,5 +1,39 @@
# Changelog # Changelog
## [1.87.4.0] - 2026-09-16
**Failed checks stay failed.**
**Health scores show what actually ran.**
`/health` now keeps each checker's exit status and counts diagnostics from its complete output. Reports still show only the final 50 log lines. Scores name the checked and unavailable categories, so a partial run carries its coverage beside the number. Runs with no checks produce no numeric score or history entry.
### The three numbers that matter
Source: the synthetic checker in `test/health-capture.test.ts`, which emits 60 type errors followed by 80 context lines and exits 2. These measurements compare the v1.87.3.0 capture example with this version under default Bash without `pipefail`. Run `bun test test/health-capture.test.ts` to verify current behavior. These are correctness measurements, not production statistics.
| Metric | Before | After | Δ |
|---|---:|---:|---:|
| Reported checker exit status | 0 | 2 | +2 |
| Type errors available for scoring | 0 | 60 | +60 |
| Displayed checker log lines | 50 | 50 | 0 |
The failing checker no longer looks successful because `tail` succeeded. All 60 errors count even when the displayed tail contains only context.
### What this means for developers
You can distinguish a score backed by several checks from one based on a single available tool. Empty runs report `N/A — no checks ran`; capture errors also remain unscored and leave existing history unchanged. Trends compare only matching categories, so installing a new checker does not manufacture a regression or improvement. Run `/health` to see the score and its coverage together.
### Itemized changes
#### Fixed
- **`/health` preserves failed checks and complete diagnostic counts.** Reports show the final 50 output lines while scoring the full log and the checker's actual exit status. Temporary capture errors remain explicit errors.
- **Health scores disclose coverage.** Partial runs list checked and unavailable categories. Runs with no checks report `N/A — no checks ran`, leave numeric history unchanged, and have no trend. Score comparisons require matching categories.
#### Changed
- Routing evaluations choose among installed GStack skills, keeping built-in CLI skills outside the evaluated catalog. The model still chooses by matching the request to each skill's description.
## [1.87.3.0] - 2026-09-15 ## [1.87.3.0] - 2026-09-15
**Changed code needs another pass.** **Changed code needs another pass.**
+1 -1
View File
@@ -1 +1 @@
1.87.3.0 1.87.4.0
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.87.3.0 — regenerate/re-copy after upgrading gstack # gstack digest v1.87.4.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows, for agent hosts without a full skill install. The full skills add workflows,
+76 -16
View File
@@ -489,22 +489,55 @@ section in CLAUDE.md:
Run each detected tool. For each tool: Run each detected tool. For each tool:
1. Record the start time 1. Record the start time
2. Run the command, capturing both stdout and stderr 2. Run the command, capturing complete stdout and stderr in a private temporary log
3. Record the exit code 3. Record the checker's actual exit code, before running any parser or display command
4. Record the end time 4. Record the end time
5. Capture the last 50 lines of output for the report 5. Parse counts from the complete log, then display its last 50 lines for the report
```bash ```bash
# Example for each tool — run each independently # Capture example — run each tool independently; adapt the command and parser.
START=$(date +%s) (
tsc --noEmit 2>&1 | tail -50 umask 077
EXIT_CODE=$? health_capture_error() {
END=$(date +%s) printf 'ERROR:typecheck CAPTURE:%s\n' "$1" >&2
echo "TOOL:typecheck EXIT:$EXIT_CODE DURATION:$((END-START))s" exit 125
}
health_log=$(mktemp "${TMPDIR:-/tmp}/gstack-health.XXXXXX") || health_capture_error log_creation
trap 'rm -f -- "$health_log"' EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
health_start=$(date +%s) || health_capture_error timing
# Open separately so a redirection failure cannot masquerade as a checker result.
if ! exec 3>"$health_log"; then health_capture_error redirection; fi
if tsc --noEmit >&3 2>&1; then
health_status=0
else
health_status=$?
fi
exec 3>&-
health_end=$(date +%s) || health_capture_error timing
# awk returns a successful zero count for no matches, including empty output.
health_count=$(awk '/error TS/ { count++ } END { print count+0 }' "$health_log") || health_capture_error parsing
tail -50 "$health_log" || health_capture_error display
printf 'TOOL:typecheck EXIT:%s DURATION:%ss ERRORS:%s\n' "$health_status" "$((health_end-health_start))" "$health_count"
exit "$health_status"
)
``` ```
Run tools sequentially (some may share resources or lock files). If a tool is not Run tools sequentially in independent invocations (some may share resources or lock
installed or not found, record it as `SKIPPED` with reason, not as a failure. files). A failed checker must not prevent later tools from running. Remember each
reported exit code and full-log counts; never use the status of `tail` or a parser
as the checker result. Guard parsers whose no-match exit is expected.
Before running a tool, check availability using the project's configured command
and local tool installation. If availability detection establishes that it is
missing, record `SKIPPED` with the reason. An executed checker returning 127 is a
failure, not evidence that the category should be skipped.
Capture failures (log creation, redirection, parsing, or display) are `ERROR`, never
`CLEAN` or `SKIPPED`. Include the cause and do not invent a category score. Report
the composite as `N/A — capture failed` if any category cannot be scored for this
reason; do not redistribute that category's weight or persist a numeric history row.
--- ---
@@ -522,6 +555,9 @@ Score each category on a 0-10 scale using this rubric:
| GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) | | GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) |
**Parsing tool output for counts:** **Parsing tool output for counts:**
Use the complete captured output, not the displayed tail. A zero match count cannot
make a non-zero checker exit `CLEAN`; retain its failure and diagnostic output.
- **tsc:** Count lines matching `error TS` in output. - **tsc:** Count lines matching `error TS` in output.
- **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available. - **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available.
- **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures). - **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures).
@@ -537,6 +573,11 @@ If a category is skipped (tool not available — includes GBrain when gbrain
is not installed), redistribute its weight proportionally among the is not installed), redistribute its weight proportionally among the
remaining categories. remaining categories.
Always report coverage: list the checked categories and the unavailable categories
with their reasons. Label a numeric composite with skipped categories as **partial
coverage**. If zero checks executed, report **N/A — no checks ran**, do not compute
a numeric composite, and skip numeric history persistence and trend calculation.
**GBrain sub-score computation (D6):** **GBrain sub-score computation (D6):**
``` ```
@@ -579,6 +620,9 @@ Shell lint shellcheck 10/10 CLEAN 1s 0 issues
GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago
COMPOSITE SCORE: 9.1 / 10 COMPOSITE SCORE: 9.1 / 10
Coverage: 6/6 categories checked
Checked: typecheck, lint, test, deadcode, shell, gbrain
Unavailable: none
Duration: 23s total Duration: 23s total
``` ```
@@ -588,6 +632,13 @@ Use these status labels:
- 7-9: `WARNING` - 7-9: `WARNING`
- 4-6: `NEEDS WORK` - 4-6: `NEEDS WORK`
- 0-3: `CRITICAL` - 0-3: `CRITICAL`
- Unavailable tool: `SKIPPED` (no score)
- Capture failure: `ERROR` (no score; composite is N/A)
For partial coverage, show e.g. `COMPOSITE SCORE: 8.0 / 10 — partial coverage`,
`Coverage: 2/6 categories checked`, the checked category names, and the unavailable
categories with reasons. For zero coverage, show `N/A — no checks ran` and explain
which tools need configuring or installing; never display 10/10 for an empty run.
If any category scored below 7, list the top issues from that tool's output: If any category scored below 7, list the top issues from that tool's output:
@@ -607,7 +658,9 @@ DETAILS: Lint (3 warnings)
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gstack/projects/$SLUG eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gstack/projects/$SLUG
``` ```
Append one JSONL line to `~/.gstack/projects/$SLUG/health-history.jsonl`: Only when a numeric composite exists, append one JSONL line to
`~/.gstack/projects/$SLUG/health-history.jsonl`. Zero-check and capture-error runs
must leave any existing history unchanged:
```json ```json
{"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23} {"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23}
@@ -636,7 +689,14 @@ eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gst
tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY" tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY"
``` ```
**If prior entries exist, show the trend:** **Compare like-for-like coverage.** For each history row, form the set of categories
with non-null scores (missing fields count as null). Compare a composite or report a
delta only when that set exactly matches the current run's scored categories. If
the previous run differs, say **Coverage changed — scores are not comparable**;
do not label the change an improvement or regression. Historical rows may still be
shown with their unavailable categories marked. A current N/A result has no trend.
**If comparable prior entries exist, show the trend:**
``` ```
HEALTH TREND (last 5 runs) HEALTH TREND (last 5 runs)
@@ -651,7 +711,7 @@ Date Branch Score TC Lint Test Dead Shell GBrain
Trend: IMPROVING (+0.9 since last run) Trend: IMPROVING (+0.9 since last run)
``` ```
**If score dropped vs the previous run:** **If score dropped vs the previous run with identical coverage:**
1. Identify WHICH categories declined 1. Identify WHICH categories declined
2. Show the delta for each declining category 2. Show the delta for each declining category
3. Correlate with tool output -- what specific errors/warnings appeared? 3. Correlate with tool output -- what specific errors/warnings appeared?
@@ -689,7 +749,7 @@ Rank by `weight * (10 - score)` descending. Only show categories below 10.
1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports. 1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports.
2. **Read-only.** Never fix issues. Present the dashboard and let the user decide. 2. **Read-only.** Never fix issues. Present the dashboard and let the user decide.
3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess. 3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess.
4. **Skipped is not failed.** If a tool isn't available, skip it gracefully and redistribute weight. Do not penalize the score. 4. **Skipped is not failed.** Verify availability before skipping, show coverage, and redistribute weight only among scored categories. An executed command's failure must not become a skip.
5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running. 5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running.
6. **Trends require history.** On first run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." 6. **Trends require comparable history.** On the first scored run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." Changed coverage and N/A runs have no score delta.
7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality. 7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality.
+76 -16
View File
@@ -114,22 +114,55 @@ section in CLAUDE.md:
Run each detected tool. For each tool: Run each detected tool. For each tool:
1. Record the start time 1. Record the start time
2. Run the command, capturing both stdout and stderr 2. Run the command, capturing complete stdout and stderr in a private temporary log
3. Record the exit code 3. Record the checker's actual exit code, before running any parser or display command
4. Record the end time 4. Record the end time
5. Capture the last 50 lines of output for the report 5. Parse counts from the complete log, then display its last 50 lines for the report
```bash ```bash
# Example for each tool — run each independently # Capture example — run each tool independently; adapt the command and parser.
START=$(date +%s) (
tsc --noEmit 2>&1 | tail -50 umask 077
EXIT_CODE=$? health_capture_error() {
END=$(date +%s) printf 'ERROR:typecheck CAPTURE:%s\n' "$1" >&2
echo "TOOL:typecheck EXIT:$EXIT_CODE DURATION:$((END-START))s" exit 125
}
health_log=$(mktemp "${TMPDIR:-/tmp}/gstack-health.XXXXXX") || health_capture_error log_creation
trap 'rm -f -- "$health_log"' EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
health_start=$(date +%s) || health_capture_error timing
# Open separately so a redirection failure cannot masquerade as a checker result.
if ! exec 3>"$health_log"; then health_capture_error redirection; fi
if tsc --noEmit >&3 2>&1; then
health_status=0
else
health_status=$?
fi
exec 3>&-
health_end=$(date +%s) || health_capture_error timing
# awk returns a successful zero count for no matches, including empty output.
health_count=$(awk '/error TS/ { count++ } END { print count+0 }' "$health_log") || health_capture_error parsing
tail -50 "$health_log" || health_capture_error display
printf 'TOOL:typecheck EXIT:%s DURATION:%ss ERRORS:%s\n' "$health_status" "$((health_end-health_start))" "$health_count"
exit "$health_status"
)
``` ```
Run tools sequentially (some may share resources or lock files). If a tool is not Run tools sequentially in independent invocations (some may share resources or lock
installed or not found, record it as `SKIPPED` with reason, not as a failure. files). A failed checker must not prevent later tools from running. Remember each
reported exit code and full-log counts; never use the status of `tail` or a parser
as the checker result. Guard parsers whose no-match exit is expected.
Before running a tool, check availability using the project's configured command
and local tool installation. If availability detection establishes that it is
missing, record `SKIPPED` with the reason. An executed checker returning 127 is a
failure, not evidence that the category should be skipped.
Capture failures (log creation, redirection, parsing, or display) are `ERROR`, never
`CLEAN` or `SKIPPED`. Include the cause and do not invent a category score. Report
the composite as `N/A — capture failed` if any category cannot be scored for this
reason; do not redistribute that category's weight or persist a numeric history row.
--- ---
@@ -147,6 +180,9 @@ Score each category on a 0-10 scale using this rubric:
| GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) | | GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) |
**Parsing tool output for counts:** **Parsing tool output for counts:**
Use the complete captured output, not the displayed tail. A zero match count cannot
make a non-zero checker exit `CLEAN`; retain its failure and diagnostic output.
- **tsc:** Count lines matching `error TS` in output. - **tsc:** Count lines matching `error TS` in output.
- **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available. - **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available.
- **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures). - **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures).
@@ -162,6 +198,11 @@ If a category is skipped (tool not available — includes GBrain when gbrain
is not installed), redistribute its weight proportionally among the is not installed), redistribute its weight proportionally among the
remaining categories. remaining categories.
Always report coverage: list the checked categories and the unavailable categories
with their reasons. Label a numeric composite with skipped categories as **partial
coverage**. If zero checks executed, report **N/A — no checks ran**, do not compute
a numeric composite, and skip numeric history persistence and trend calculation.
**GBrain sub-score computation (D6):** **GBrain sub-score computation (D6):**
``` ```
@@ -204,6 +245,9 @@ Shell lint shellcheck 10/10 CLEAN 1s 0 issues
GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago
COMPOSITE SCORE: 9.1 / 10 COMPOSITE SCORE: 9.1 / 10
Coverage: 6/6 categories checked
Checked: typecheck, lint, test, deadcode, shell, gbrain
Unavailable: none
Duration: 23s total Duration: 23s total
``` ```
@@ -213,6 +257,13 @@ Use these status labels:
- 7-9: `WARNING` - 7-9: `WARNING`
- 4-6: `NEEDS WORK` - 4-6: `NEEDS WORK`
- 0-3: `CRITICAL` - 0-3: `CRITICAL`
- Unavailable tool: `SKIPPED` (no score)
- Capture failure: `ERROR` (no score; composite is N/A)
For partial coverage, show e.g. `COMPOSITE SCORE: 8.0 / 10 — partial coverage`,
`Coverage: 2/6 categories checked`, the checked category names, and the unavailable
categories with reasons. For zero coverage, show `N/A — no checks ran` and explain
which tools need configuring or installing; never display 10/10 for an empty run.
If any category scored below 7, list the top issues from that tool's output: If any category scored below 7, list the top issues from that tool's output:
@@ -232,7 +283,9 @@ DETAILS: Lint (3 warnings)
{{SLUG_SETUP}} {{SLUG_SETUP}}
``` ```
Append one JSONL line to `~/.gstack/projects/$SLUG/health-history.jsonl`: Only when a numeric composite exists, append one JSONL line to
`~/.gstack/projects/$SLUG/health-history.jsonl`. Zero-check and capture-error runs
must leave any existing history unchanged:
```json ```json
{"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23} {"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23}
@@ -261,7 +314,14 @@ file exists and has prior entries).
tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY" tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY"
``` ```
**If prior entries exist, show the trend:** **Compare like-for-like coverage.** For each history row, form the set of categories
with non-null scores (missing fields count as null). Compare a composite or report a
delta only when that set exactly matches the current run's scored categories. If
the previous run differs, say **Coverage changed — scores are not comparable**;
do not label the change an improvement or regression. Historical rows may still be
shown with their unavailable categories marked. A current N/A result has no trend.
**If comparable prior entries exist, show the trend:**
``` ```
HEALTH TREND (last 5 runs) HEALTH TREND (last 5 runs)
@@ -276,7 +336,7 @@ Date Branch Score TC Lint Test Dead Shell GBrain
Trend: IMPROVING (+0.9 since last run) Trend: IMPROVING (+0.9 since last run)
``` ```
**If score dropped vs the previous run:** **If score dropped vs the previous run with identical coverage:**
1. Identify WHICH categories declined 1. Identify WHICH categories declined
2. Show the delta for each declining category 2. Show the delta for each declining category
3. Correlate with tool output -- what specific errors/warnings appeared? 3. Correlate with tool output -- what specific errors/warnings appeared?
@@ -314,7 +374,7 @@ Rank by `weight * (10 - score)` descending. Only show categories below 10.
1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports. 1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports.
2. **Read-only.** Never fix issues. Present the dashboard and let the user decide. 2. **Read-only.** Never fix issues. Present the dashboard and let the user decide.
3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess. 3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess.
4. **Skipped is not failed.** If a tool isn't available, skip it gracefully and redistribute weight. Do not penalize the score. 4. **Skipped is not failed.** Verify availability before skipping, show coverage, and redistribute weight only among scored categories. An executed command's failure must not become a skip.
5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running. 5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running.
6. **Trends require history.** On first run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." 6. **Trends require comparable history.** On the first scored run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." Changed coverage and N/A runs have no score delta.
7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality. 7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality.
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "gstack", "name": "gstack",
"version": "1.87.3", "version": "1.87.4",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT", "license": "MIT",
"type": "module", "type": "module",
+1 -1
View File
@@ -29,7 +29,7 @@ describe('AO completed manual DX handoff preserves report freshness',()=>{
expect(E2E_TOUCHFILES[owner]).toContain('test/fixtures/dx-manual-handoff-ao.json'); expect(E2E_TOUCHFILES[owner]).toContain('test/fixtures/dx-manual-handoff-ao.json');
} }
const arrays=[...Object.values(E2E_TOUCHFILES),...Object.values(LLM_JUDGE_TOUCHFILES),GLOBAL_TOUCHFILES]; const arrays=[...Object.values(E2E_TOUCHFILES),...Object.values(LLM_JUDGE_TOUCHFILES),GLOBAL_TOUCHFILES];
expect(arrays).toHaveLength(210); expect(arrays).toHaveLength(211);
for(const values of arrays)for(let i=0;i<values.length;i++)expect(typeof values[i]).toBe('string'); for(const values of arrays)for(let i=0;i<values.length;i++)expect(typeof values[i]).toBe('string');
}); });
test('exact owned report precedes navigation only, with the current Exit gate recognized',()=>{ test('exact owned report precedes navigation only, with the current Exit gate recognized',()=>{
+92
View File
@@ -0,0 +1,92 @@
import { afterEach, describe, expect, test } from 'bun:test';
import { spawnSync } from 'node:child_process';
import { chmodSync, mkdtempSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
// Execute the documented capture, so the example cannot drift from its tests.
const template = readFileSync(join(import.meta.dir, '../health/SKILL.md.tmpl'), 'utf8');
const capture = template.split('## Step 2: Run Tools')[1].match(/```bash\n([\s\S]*?)```/)![1];
const temporaryDirectories: string[] = [];
afterEach(() => {
for (const directory of temporaryDirectories.splice(0)) rmSync(directory, { recursive: true, force: true });
});
function runCapture(checker: string, setup = '') {
const directory = mkdtempSync(join(tmpdir(), 'health-capture-test-'));
temporaryDirectories.push(directory);
const logs = join(directory, 'logs');
mkdirSync(logs);
const executable = join(directory, 'tsc');
writeFileSync(executable, '#!/usr/bin/env bash\n' + checker + '\n');
chmodSync(executable, 0o755);
const result = spawnSync('/bin/bash', ['-euc', setup + '\n' + capture], {
encoding: 'utf8',
timeout: 10_000,
env: { ...process.env, PATH: directory + ':' + process.env.PATH, TMPDIR: logs },
});
expect(result.error).toBeUndefined();
expect(readdirSync(logs)).toEqual([]);
return result;
}
describe('/health command capture', () => {
test('a successful empty checker reports zero matches under set -e', () => {
const result = runCapture('exit 0');
expect(result.status).toBe(0);
expect(result.stdout).toMatch(/TOOL:typecheck EXIT:0 DURATION:\d+s ERRORS:0/);
});
test('failure survives an earlier success, stderr capture, and display', () => {
const result = runCapture('echo "source.ts: error TS2322: incorrect type" >&2\nexit 2', 'true');
expect(result.status).toBe(2);
expect(result.stdout).toContain('incorrect type');
expect(result.stdout).toMatch(/EXIT:2 DURATION:\d+s ERRORS:1/);
});
test('counts findings outside the displayed tail and prints only fifty log lines', () => {
const result = runCapture([
'for ((i=1; i<=60; i++)); do echo "source.ts: error TS2322: finding $i"; done',
'for ((i=1; i<=80; i++)); do echo "detail $i"; done',
'exit 2',
].join('\n'));
expect(result.status).toBe(2);
expect(result.stdout).toMatch(/EXIT:2 DURATION:\d+s ERRORS:60/);
const lines = result.stdout.trimEnd().split('\n');
expect(lines.length).toBe(51);
expect(lines[0]).toBe('detail 31');
expect(lines[49]).toBe('detail 80');
});
test('an executed checker returning 127 remains a failure', () => {
const result = runCapture('echo "a checker dependency failed" >&2\nexit 127');
expect(result.status).toBe(127);
expect(result.stdout).toContain('EXIT:127');
expect(result.stdout).not.toContain('SKIPPED');
});
test('empty failing output does not inherit a successful parser status', () => {
const result = runCapture('exit 1');
expect(result.status).toBe(1);
expect(result.stdout).toMatch(/EXIT:1 DURATION:\d+s ERRORS:0/);
});
test('capture log permissions are private even with a permissive caller umask', () => {
const result = runCapture('stat -c %a "$TMPDIR"/gstack-health.* 2>/dev/null || stat -f %Lp "$TMPDIR"/gstack-health.*', 'umask 000');
expect(result.status).toBe(0);
expect(result.stdout.split('\n')[0]).toBe('600');
});
test.each([
['log_creation', 'mktemp() { return 1; }'],
['redirection', 'mktemp() { printf "%s/missing/log\\n" "$TMPDIR"; }'],
['parsing', 'awk() { return 2; }'],
['display', 'tail() { return 1; }'],
])('%s failure reports an error instead of a clean result', (phase, setup) => {
const result = runCapture('exit 0', setup);
expect(result.status).toBe(125);
expect(result.stderr).toContain('ERROR:typecheck CAPTURE:' + phase);
expect(result.stdout).not.toContain('TOOL:typecheck EXIT:0');
});
});
+139
View File
@@ -0,0 +1,139 @@
/** Free fixture/recording checks; never import or invoke the paid runner. */
import { afterEach, describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { spawnSync } from 'node:child_process';
import {
createHealthEvalFixture, healthReportingFailures, recordHealthAttempt,
} from './helpers/health-eval-fixture';
import type { SkillTestResult } from './helpers/session-runner';
import type { EvalTestEntry } from './helpers/eval-store';
import { E2E_TIERS, E2E_TOUCHFILES, GLOBAL_TOUCHFILES, selectTests } from './helpers/touchfiles';
import { isPaidTestFile } from './helpers/paid-test-set';
const ROOT = path.resolve(import.meta.dir, '..');
const dirs: string[] = [];
afterEach(() => { for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); });
test('health behavior is selected as periodic and paid while capture regressions stay free', () => {
const selectedBy = (file: string) => selectTests([file], E2E_TOUCHFILES, GLOBAL_TOUCHFILES).selected;
expect(selectedBy('health/SKILL.md.tmpl')).toContain('health-reporting');
expect(selectedBy('test/helpers/health-eval-fixture.ts')).toEqual(['health-reporting']);
expect(E2E_TIERS['health-reporting']).toBe('periodic');
expect(isPaidTestFile('test/skill-e2e-health.test.ts')).toBe(true);
expect(isPaidTestFile('test/health-capture.test.ts')).toBe(false);
});
function fixture(prefix = 'health-fixture-test-') {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
dirs.push(dir);
return createHealthEvalFixture(dir, ROOT);
}
function writePassingEvidence(f: ReturnType<typeof fixture>) {
fs.writeFileSync(path.join(f.dir, 'partial-report.md'), `Type check: 0/10 CRITICAL, 60 errors.
Tests: 10/10 CLEAN, 5 passed.
Checked: type check, tests. Unavailable: lint, dead code, shell lint, GBrain (not installed).
COMPOSITE SCORE: **5.6 / 10 — partial coverage**
Coverage changed; prior test-only history is not comparable.
`);
fs.writeFileSync(path.join(f.dir, 'no-tools-report.md'), 'COMPOSITE SCORE: N/A — no checks ran.\n');
fs.appendFileSync(path.join(f.gstackHome, 'projects', 'partial', 'health-history.jsonl'), JSON.stringify({
score: 5.6, typecheck: 0, test: 10, lint: null, deadcode: null, shell: null, gbrain: null,
}) + '\n');
fs.writeFileSync(f.receipts, 'typecheck\ntest\n');
}
describe('/health eval fixtures', () => {
test('extracts all six real workflow steps and redirects persistent paths', () => {
const f = fixture();
const skill = fs.readFileSync(path.join(f.dir, 'health-SKILL.md'), 'utf-8');
expect(skill).not.toContain('## Preamble (run first)');
expect(skill).not.toContain('~/.gstack');
expect(skill).not.toContain('~/.claude/skills/gstack/bin/gstack-slug');
for (let step = 1; step <= 6; step++) expect(skill).toContain(`## Step ${step}:`);
for (const project of ['partial', 'no-tools']) {
const result = spawnSync(path.join(f.dir, 'bin', 'gstack-slug'), [], {
cwd: path.join(f.dir, project), encoding: 'utf-8', timeout: 10_000,
env: { ...process.env, GSTACK_HOME: f.gstackHome, GSTACK_PROJECT_SLUG: '' },
});
expect(result.status).toBe(0);
expect(result.stdout).toContain(`SLUG=${project}\n`);
}
});
test('real checker exits nonzero and its final 50 lines hide all 60 errors', () => {
const f = fixture();
const result = spawnSync('bash', ['-c', 'bash ./check-typecheck.sh 2>&1'], {
cwd: path.join(f.dir, 'partial'), encoding: 'utf-8', timeout: 10_000,
});
expect(result.status).toBe(2);
expect(result.stdout.match(/error TS/g)).toHaveLength(60);
expect(result.stdout.trimEnd().split('\n').slice(-50).join('\n')).not.toContain('error TS');
expect(fs.readFileSync(f.receipts, 'utf-8')).toBe('typecheck\n');
});
test('both checkers record to their fixed path without shell environment setup', () => {
const f = fixture("health-fixture-test-' space-");
for (const [script, status] of [['check-typecheck.sh', 2], ['check-tests.sh', 0]] as const) {
const result = spawnSync('bash', [script], {
cwd: path.join(f.dir, 'partial'), encoding: 'utf-8', timeout: 10_000,
});
expect(result.status).toBe(status);
}
expect(fs.readFileSync(f.receipts, 'utf-8')).toBe('typecheck\ntest\n');
});
test('accepts truthful reports and history, rejects absent evidence', () => {
const f = fixture();
expect(healthReportingFailures(f).length).toBeGreaterThan(0);
writePassingEvidence(f);
expect(healthReportingFailures(f)).toEqual([]);
});
test.each(['false-clean', 'false-clean-category', 'tail-count', 'numeric-empty', 'hidden-coverage', 'unlike-trend', 'contradictory-trend', 'extra-empty-history', 'missing-receipts', 'project-edit'])('%s is a failed behavior eval', failure => {
const f = fixture();
writePassingEvidence(f);
const report = path.join(f.dir, 'partial-report.md');
if (failure === 'false-clean') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('5.6', '10'));
if (failure === 'false-clean-category') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('0/10 CRITICAL', '10/10 CLEAN'));
if (failure === 'tail-count') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('60 errors', '0 errors'));
if (failure === 'numeric-empty') fs.writeFileSync(path.join(f.dir, 'no-tools-report.md'), 'COMPOSITE SCORE: 10 — no checks ran.');
if (failure === 'hidden-coverage') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace(' — partial coverage', ''));
if (failure === 'unlike-trend') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('Coverage changed; prior test-only history is not comparable.', 'Health worsened from 10 to 5.6.'));
if (failure === 'contradictory-trend') fs.appendFileSync(report, 'Trend: REGRESSING (-4.4 since last run)\n');
if (failure === 'extra-empty-history') fs.appendFileSync(path.join(f.gstackHome, 'projects', 'no-tools', 'health-history.jsonl'), '{"score":10}\n');
if (failure === 'missing-receipts') fs.rmSync(f.receipts);
if (failure === 'project-edit') fs.appendFileSync(path.join(f.dir, 'partial', 'check-typecheck.sh'), '# attempted fix\n');
expect(healthReportingFailures(f).length).toBeGreaterThan(0);
});
});
describe('/health eval recording', () => {
const success = {
exitReason: 'success', duration: 5, output: 'report', transcript: [],
costEstimate: { estimatedCost: 0.1, turnsUsed: 2, estimatedTokens: 100 }, model: 'fixture',
} as SkillTestResult;
test.each(['success', 'assertion', 'timeout', 'throw'])('%s records exactly once with matching pass status', async scenario => {
const entries: EvalTestEntry[] = [];
let verified = false;
const attempt = recordHealthAttempt(
entry => entries.push(entry),
async () => {
if (scenario === 'throw') throw new Error('runner failed');
return { ...success, exitReason: scenario === 'timeout' ? 'timeout' : 'success' };
},
() => { verified = true; if (scenario === 'assertion') throw new Error('false dashboard'); },
);
if (scenario === 'success') await attempt;
else await expect(attempt).rejects.toThrow();
expect(entries).toHaveLength(1);
expect(entries[0].passed).toBe(scenario === 'success');
expect(entries[0].tier).toBe('e2e');
expect(verified).toBe(scenario === 'success' || scenario === 'assertion');
if (scenario === 'assertion') expect(entries[0].output).toContain('false dashboard');
if (scenario === 'timeout') expect(entries[0].exit_reason).toBe('timeout');
});
});
+209
View File
@@ -0,0 +1,209 @@
/** Isolated fixtures and assertions for the single periodic /health capture. */
import * as fs from 'node:fs';
import * as path from 'node:path';
import { extractSkillSections } from './skill-fixture';
import type { SkillTestResult } from './session-runner';
import type { EvalTestEntry } from './eval-store';
export const HEALTH_EVAL_ID = 'health-reporting';
export const HEALTH_EVAL_SECTIONS = [
'Step 1: Detect Health Stack',
'Step 2: Run Tools',
'Step 3: Score Each Category',
'Step 4: Present Dashboard',
'Step 5: Persist to Health History',
'Step 6: Trend Analysis + Recommendations',
'Important Rules',
];
const PRIOR_HISTORY = JSON.stringify({
ts: '2026-01-01T00:00:00Z', branch: 'unknown', score: 10,
typecheck: null, lint: null, test: 10, deadcode: null, shell: null,
gbrain: null, duration_s: 1,
}) + '\n';
export interface HealthEvalFixture {
dir: string;
gstackHome: string;
receipts: string;
prompt: string;
projectFiles: Record<string, string>;
}
function projectSnapshot(dir: string): Record<string, string> {
const files: Record<string, string> = {};
const walk = (relative: string) => {
for (const entry of fs.readdirSync(path.join(dir, relative), { withFileTypes: true })) {
const name = path.join(relative, entry.name);
if (entry.isDirectory()) { files[name + '/'] = '<directory>'; walk(name); }
else files[name] = entry.isSymbolicLink() ? `<symlink:${fs.readlinkSync(path.join(dir, name))}>` : fs.readFileSync(path.join(dir, name), 'utf-8');
}
};
for (const project of ['partial', 'no-tools']) walk(project);
return files;
}
export function createHealthEvalFixture(dir: string, repoRoot: string): HealthEvalFixture {
const gstackHome = path.join(dir, 'gstack-state');
const receipts = path.join(dir, 'checker-runs.txt');
// Instrumentation belongs to the fixture, not to the model's shell setup.
// Single-quote escaping also covers temporary paths containing apostrophes.
const quotedReceipts = "'" + receipts.replaceAll("'", "'\"'\"'") + "'";
fs.mkdirSync(path.join(dir, 'bin'), { recursive: true });
fs.copyFileSync(path.join(repoRoot, 'bin', 'gstack-slug'), path.join(dir, 'bin', 'gstack-slug'));
fs.chmodSync(path.join(dir, 'bin', 'gstack-slug'), 0o755);
// Only redirect installed paths. The workflow and score/history rules come
// from the real generated skill, without its unrelated shared preamble.
const skill = extractSkillSections(path.join(repoRoot, 'health'), HEALTH_EVAL_SECTIONS)
.replaceAll('~/.claude/skills/gstack/bin/gstack-slug', path.join(dir, 'bin', 'gstack-slug'))
.replaceAll('~/.gstack', gstackHome);
fs.writeFileSync(path.join(dir, 'health-SKILL.md'), skill);
for (const project of ['partial', 'no-tools']) {
const projectDir = path.join(dir, project);
fs.mkdirSync(projectDir);
fs.writeFileSync(path.join(projectDir, '.project.yaml'), `name: ${project}\n`);
const historyDir = path.join(gstackHome, 'projects', project);
fs.mkdirSync(historyDir, { recursive: true });
fs.writeFileSync(path.join(historyDir, 'health-history.jsonl'), PRIOR_HISTORY);
}
fs.writeFileSync(path.join(dir, 'partial', 'CLAUDE.md'), `# Partial project
## Health Stack
- typecheck: bash ./check-typecheck.sh
- test: bash ./check-tests.sh
Only the listed tools are available. Lint, dead-code, shell-lint, and GBrain
tools are not installed. This configuration is final; do not install tools.
`);
fs.writeFileSync(path.join(dir, 'no-tools', 'CLAUDE.md'), `# No-tools project
## Health Stack
No health tools are configured or installed for any category.
This configuration is final; do not install tools or substitute other checks.
`);
fs.writeFileSync(path.join(dir, 'partial', 'check-typecheck.sh'), `#!/usr/bin/env bash
printf 'typecheck\\n' >> ${quotedReceipts}
for ((i = 1; i <= 60; i++)); do
printf 'src/file%s.ts(1,1): error TS2322: Type mismatch.\\n' "$i" >&2
done
for ((i = 1; i <= 80; i++)); do
printf 'Additional diagnostic context %s\\n' "$i"
done
exit 2
`);
fs.writeFileSync(path.join(dir, 'partial', 'check-tests.sh'), `#!/usr/bin/env bash
printf 'test\\n' >> ${quotedReceipts}
printf '5 pass\\n0 fail\\n'
`);
return {
dir, gstackHome, receipts, projectFiles: projectSnapshot(dir),
prompt: `Read health-SKILL.md and run its /health workflow, Steps 16, for
the partial project first and the no-tools project second. Each has its own
CLAUDE.md with the final Health Stack configuration. Run commands from the
corresponding project directory. These are local fixtures without Git remotes.
Save each complete dashboard, details, trends, and recommendations as
partial-report.md or no-tools-report.md in ${dir}. Keep the dashboard's
COMPOSITE SCORE label. Existing health histories are available under
${gstackHome}/projects/<project>/health-history.jsonl; apply the skill's normal
history rules. GSTACK_HOME already points at this isolated state directory.
Do not modify project files, install tools, or ask to change either Health Stack.
Only write the reports and any history updates required by the supplied skill.
Finish after producing both reports.`,
};
}
/** Validate behavior, allowing ordinary Markdown/wording variation in reports. */
export function healthReportingFailures(fixture: HealthEvalFixture): string[] {
const failures: string[] = [];
const check = (ok: boolean, message: string) => { if (!ok) failures.push(message); };
const read = (file: string) => fs.existsSync(file) ? fs.readFileSync(file, 'utf-8') : '';
const plain = (text: string) => text.replace(/[*_`]/g, '');
const partial = plain(read(path.join(fixture.dir, 'partial-report.md')));
const empty = plain(read(path.join(fixture.dir, 'no-tools-report.md')));
const composite = (report: string) => report.match(/composite\s+score\s*[:|]?\s*(N\/A|\d+(?:\.\d+)?)/i)?.[1];
check(composite(partial) === '5.6', 'partial coverage must score 5.6, preserving the failing typecheck');
check(/partial\s+coverage/i.test(partial), 'numeric score must be labeled as partial coverage');
const typecheckRows = partial.split('\n').filter(line => /\btype\s*check\b/i.test(line));
check(typecheckRows.some(line => /(?:\b0\s*\/\s*10\b|\|\s*0\s*\|)/.test(line)
&& /\b(?:critical|fail(?:ed|ure)?|error)\b/i.test(line) && !/\bclean\b/i.test(line)),
'dashboard must report the failing typecheck as 0/10, not clean');
check(/\b60\s+(?:\w+\s+){0,2}(?:errors|diagnostics|findings)\b/i.test(partial)
|| /(?:errors|diagnostics|findings)[^\n]{0,20}\b60\b/i.test(partial),
'report must count all 60 errors before the final 50 output lines');
check(/(?:coverage|checked|executed)/i.test(partial), 'partial report must disclose checked coverage');
check(/type\s*check/i.test(partial) && /tests?/i.test(partial), 'checked category names must be visible');
for (const category of ['lint', 'dead[ -]?code', 'shell(?:[ -]?lint)?', 'gbrain']) {
const unavailable = '(?:unavailable|skipped|not (?:installed|configured|available|found))';
check(new RegExp(`${unavailable}[\\s\\S]{0,240}${category}|${category}[^\\n]{0,120}${unavailable}`, 'i').test(partial),
`partial report must name unavailable ${category}`);
}
check(/(?:coverage|categor(?:y|ies)|checks)[\s\S]{0,160}(?:chang|differ|not compar)/i.test(partial)
|| /(?:chang|differ|not compar)[\s\S]{0,160}(?:coverage|categor(?:y|ies)|checks)/i.test(partial),
'partial report must flag changed coverage instead of comparing unlike histories');
check(!/[+-]\s*4\.4\b|trend\s*:\s*(?:improving|regressing|worsening)/i.test(partial),
'partial report must not calculate a trend delta against different coverage');
check(composite(empty)?.toUpperCase() === 'N/A', 'no-tools composite must be N/A');
check(/(?:no|zero|0)\s+(?:health\s+)?checks?\s+(?:ran|run|executed|available)|no\s+tools/i.test(empty),
'no-tools report must explain that no checks ran');
const history = read(path.join(fixture.gstackHome, 'projects', 'partial', 'health-history.jsonl'));
check(history.startsWith(PRIOR_HISTORY), 'partial run must preserve its prior history row');
const rows = history.trim().split('\n').filter(Boolean);
check(rows.length === 2, 'partial run must append exactly one history row');
try {
const row = JSON.parse(rows.at(-1) || '{}');
check(row.score === 5.6 && row.typecheck === 0 && row.test === 10,
'persisted partial scores must reflect all diagnostics and the actual exit status');
check(['lint', 'deadcode', 'shell', 'gbrain'].every(category => row[category] === null),
'unavailable categories must persist as null');
} catch {
failures.push('partial history must remain valid JSONL');
}
check(read(path.join(fixture.gstackHome, 'projects', 'no-tools', 'health-history.jsonl')) === PRIOR_HISTORY,
'no-tools run must leave its history unchanged');
const runs = read(fixture.receipts).trim().split('\n');
check(runs.includes('typecheck') && runs.includes('test'), 'both configured checkers must actually run');
check(JSON.stringify(projectSnapshot(fixture.dir)) === JSON.stringify(fixture.projectFiles),
'health must leave project files unchanged');
return failures;
}
/** One attempt records once, after all assertions; throws remain failures. */
export async function recordHealthAttempt(
record: (entry: EvalTestEntry) => void,
capture: () => Promise<SkillTestResult>,
verify: (result: SkillTestResult) => void,
): Promise<void> {
const started = Date.now();
let result: SkillTestResult | undefined;
let passed = false;
let failure: unknown;
try {
result = await capture();
if (result.exitReason !== 'success') throw new Error(`health capture ended: ${result.exitReason}`);
verify(result);
passed = true;
} catch (error) {
failure = error;
throw error;
} finally {
record({
name: HEALTH_EVAL_ID, suite: 'health', tier: 'e2e', passed,
duration_ms: result?.duration ?? Date.now() - started,
cost_usd: result?.costEstimate.estimatedCost ?? 0,
turns_used: result?.costEstimate.turnsUsed,
tokens_used: result?.costEstimate.estimatedTokens,
transcript: result?.transcript,
output: [result?.output, failure === undefined ? '' : String(failure)].filter(Boolean).join('\n').slice(-4000),
exit_reason: result?.exitReason === 'success' && !passed ? 'assertion_failed' : result?.exitReason ?? 'runner_error',
model: result?.model,
});
}
}
+4
View File
@@ -582,6 +582,9 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
// Document-release // Document-release
'document-release': ['document-release/**', 'test/skill-e2e-workflow.test.ts'], 'document-release': ['document-release/**', 'test/skill-e2e-workflow.test.ts'],
// /health result capture, coverage, and comparable history (model behavior).
'health-reporting': ['health/**', 'test/skill-e2e-health.test.ts', 'test/helpers/health-eval-fixture.ts'],
// Codex (Claude E2E — tests /codex skill via Claude) // Codex (Claude E2E — tests /codex skill via Claude)
'codex-review': ['codex/**', 'test/skill-e2e-workflow.test.ts'], 'codex-review': ['codex/**', 'test/skill-e2e-workflow.test.ts'],
@@ -1186,6 +1189,7 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic'> = {
// Document-release — gate (CHANGELOG guardrail) // Document-release — gate (CHANGELOG guardrail)
'document-release': 'gate', 'document-release': 'gate',
'health-reporting': 'periodic',
// Codex — periodic (Opus, requires codex CLI) // Codex — periodic (Opus, requires codex CLI)
'codex-review': 'periodic', 'codex-review': 'periodic',
+1 -1
View File
@@ -69,7 +69,7 @@ describe('native repeated report permission identity',()=>{
for (const variant of ['basic', 'intervening', 'cropped', 'same-basename', 'path-cropped']) test.skipIf(process.platform==='win32')(`real fake CLI grants each current request once: ${variant}`,async()=>{ for (const variant of ['basic', 'intervening', 'cropped', 'same-basename', 'path-cropped']) test.skipIf(process.platform==='win32')(`real fake CLI grants each current request once: ${variant}`,async()=>{
const intervening = variant === 'intervening'; const intervening = variant === 'intervening';
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'count-edit-pty-'));const fake=path.join(dir,'fake-claude');const worker=path.join(dir,'worker.ts');const events=path.join(dir,'events.jsonl');const output=path.join(dir,'output.json');const expected=path.join(dir,variant==='same-basename'?'PLAN.md':'report.md');fs.writeFileSync(expected,'original'); const dir=fs.mkdtempSync(path.join(os.tmpdir(),'ce-'));const fake=path.join(dir,'fake-claude');const worker=path.join(dir,'worker.ts');const events=path.join(dir,'events.jsonl');const output=path.join(dir,'output.json');const expected=path.join(dir,variant==='same-basename'?'PLAN.md':'report.md');fs.writeFileSync(expected,'original');
const cropped=capturedAc.rows.find(row=>row.job===5)!; const cropped=capturedAc.rows.find(row=>row.job===5)!;
let screen=variant==='path-cropped' ? capturedPath.screen.replace(capturedPath.screen.split('\n')[0]!,expected).replaceAll(path.dirname(capturedPath.expected),path.dirname(expected)).replaceAll(path.basename(capturedPath.expected),'report.md') let screen=variant==='path-cropped' ? capturedPath.screen.replace(capturedPath.screen.split('\n')[0]!,expected).replaceAll(path.dirname(capturedPath.expected),path.dirname(expected)).replaceAll(path.basename(capturedPath.expected),'report.md')
: variant==='cropped' ? cropped.screen.replaceAll(path.dirname(cropped.hook.expected),path.dirname(expected)).replaceAll(path.basename(cropped.hook.expected),'report.md') : variant==='cropped' ? cropped.screen.replaceAll(path.dirname(cropped.hook.expected),path.dirname(expected)).replaceAll(path.basename(cropped.hook.expected),'report.md')
+28 -5
View File
@@ -10,6 +10,26 @@ import owned from './fixtures/plan-count-owned-permission-v.json';
import {E2E_TOUCHFILES,selectTests} from './helpers/touchfiles'; import {E2E_TOUCHFILES,selectTests} from './helpers/touchfiles';
const quote=(s:string)=>s.split('\n').map(row=>'> '+row).join('\n'); const quote=(s:string)=>s.split('\n').map(row=>'> '+row).join('\n');
// A PTY transports bytes, not command-sized stdin events. Share the framing
// code with the fake CLI so fragmented grants exercise the same receiver.
function commandBuffer(){
let pending='';
return (chunk:string)=>{
pending+=chunk;const commands:string[]=[];let end:number;
while((end=pending.indexOf('\r'))!==-1){commands.push(pending.slice(0,end+1));pending=pending.slice(end+1);}
return commands;
};
}
test('fake CLI preserves command bytes across fragmented and coalesced PTY input',()=>{
const expected=['/plan-ceo-review\r','1\r'];
for(const chunks of [expected,['/plan-ceo-review\r','1','\r'],[...expected.join('')],[expected.join('')]]){
const receive=commandBuffer();expect(chunks.flatMap(receive)).toEqual(expected);
}
const receive=commandBuffer();
expect(receive('1')).toEqual([]);expect(receive('\r2\rtrailing')).toEqual(['1\r','2\r']);
expect(receive('\r')).toEqual(['trailing\r']); // no unexpected bytes are discarded
});
test('the exact wholly quoted AK pane is handled so the dispatcher sends no fallback',()=>{ test('the exact wholly quoted AK pane is handled so the dispatcher sends no fallback',()=>{
const screen=quote(exact.screen); const screen=quote(exact.screen);
expect(classifyPlanCountFrame(screen)).toBe('permission'); expect(classifyPlanCountFrame(screen)).toBe('permission');
@@ -44,7 +64,7 @@ test('the regression selects exactly the existing permission consumers',()=>{
test.skipIf(process.platform==='win32')('real dispatcher ignores quoted pane then grants the fresh owned native request once',async()=>{ test.skipIf(process.platform==='win32')('real dispatcher ignores quoted pane then grants the fresh owned native request once',async()=>{
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'count-quoted-frame-')),fake=path.join(dir,'fake-claude'),worker=path.join(dir,'worker.ts'),events=path.join(dir,'events.jsonl'),output=path.join(dir,'result.json'),report=path.join(dir,'report.md'); const dir=fs.mkdtempSync(path.join(os.tmpdir(),'count-quoted-frame-')),fake=path.join(dir,'fake-claude'),worker=path.join(dir,'worker.ts'),events=path.join(dir,'events.jsonl'),output=path.join(dir,'result.json'),report=path.join(dir,'report.md');
fs.writeFileSync(report,'original'); fs.writeFileSync(report,'original');
fs.writeFileSync(fake,`#!${process.execPath}\n`+String.raw` fs.writeFileSync(fake,`#!${process.execPath}\nconst receive=(${commandBuffer.toString()})();\n`+String.raw`
import fs from 'node:fs';import path from 'node:path'; import fs from 'node:fs';import path from 'node:path';
const item=JSON.parse(process.env.QUOTED_FRAME_CASE),sid='quoted-frame-main',log=e=>fs.appendFileSync(item.events,JSON.stringify(e)+'\n'); const item=JSON.parse(process.env.QUOTED_FRAME_CASE),sid='quoted-frame-main',log=e=>fs.appendFileSync(item.events,JSON.stringify(e)+'\n');
const transcript=path.join(process.env.CLAUDE_CONFIG_DIR,'projects','owned',sid+'.jsonl');fs.mkdirSync(path.dirname(transcript),{recursive:true}); const transcript=path.join(process.env.CLAUDE_CONFIG_DIR,'projects','owned',sid+'.jsonl');fs.mkdirSync(path.dirname(transcript),{recursive:true});
@@ -56,14 +76,14 @@ const hook=async name=>{for(const entry of settings.hooks[name]??[]){if(entry.ma
const p=Bun.spawn(['bash','-c',entry.hooks[0].command],{stdin:new Blob([JSON.stringify(event)]),stdout:'pipe',stderr:'pipe'}); const p=Bun.spawn(['bash','-c',entry.hooks[0].command],{stdin:new Blob([JSON.stringify(event)]),stdout:'pipe',stderr:'pipe'});
const [code,out,err]=await Promise.all([p.exited,new Response(p.stdout).text(),new Response(p.stderr).text()]);if(code||out||err)throw Error('Hook failed');}}; const [code,out,err]=await Promise.all([p.exited,new Response(p.stdout).text(),new Response(p.stderr).text()]);if(code||out||err)throw Error('Hook failed');}};
const pane=item.screen.replaceAll('PLAN.md',item.report),paint=s=>process.stdout.write('\x1b[2J\x1b[H'+s.replaceAll('\n','\r\n')); const pane=item.screen.replaceAll('PLAN.md',item.report),paint=s=>process.stdout.write('\x1b[2J\x1b[H'+s.replaceAll('\n','\r\n'));
let stage='startup';process.stdin.setRawMode?.(true);process.stdin.on('data',async data=>{ let stage='startup';process.stdin.setRawMode?.(true);const dispatch=async input=>{
const input=data.toString();log({type:'input',stage,input}); log({type:'input',stage,input});
if(stage==='startup'){stage='quoted';paint(item.quotedScreen);setTimeout(async()=>{await hook('PreToolUse');stage='current';paint(pane);},4200);return;} if(stage==='startup'){stage='quoted';paint(item.quotedScreen);setTimeout(async()=>{await hook('PreToolUse');stage='current';paint(pane);},4200);return;}
if(stage!=='current'){log({type:'unexpected'});return;} if(stage!=='current'){log({type:'unexpected'});return;}
if(input!=='1\r')throw Error('One-time grant changed');stage='done';await hook('PostToolUse'); if(input!=='1\r')throw Error('One-time grant changed');stage='done';await hook('PostToolUse');
const q={header:'Finding',question:'Apply the reviewed fix?',options:[{label:'Fix'},{label:'Keep'}]}; const q={header:'Finding',question:'Apply the reviewed fix?',options:[{label:'Fix'},{label:'Keep'}]};
native('assistant',[{type:'tool_use',name:'AskUserQuestion',id:'finding',input:{questions:[q]}}]);native('user',[{type:'tool_result',tool_use_id:'finding',content:'Answered'}],{toolUseResult:{answers:{[q.question]:'Fix'}}});paint('Done.\n'); native('assistant',[{type:'tool_use',name:'AskUserQuestion',id:'finding',input:{questions:[q]}}]);native('user',[{type:'tool_result',tool_use_id:'finding',content:'Answered'}],{toolUseResult:{answers:{[q.question]:'Fix'}}});paint('Done.\n');
});process.on('SIGINT',()=>process.exit(0));process.stdin.resume(); };process.stdin.on('data',async data=>{const chunk=data.toString();log({type:'chunk',stage,input:chunk});for(const input of receive(chunk))await dispatch(input);});process.on('SIGINT',()=>process.exit(0));process.stdin.resume();
`);fs.chmodSync(fake,0o755); `);fs.chmodSync(fake,0o755);
// Keep every physical terminal row inside the quote; adding a prefix to an // Keep every physical terminal row inside the quote; adding a prefix to an
// already120-column capture would otherwise wrap an unquoted continuation. // already120-column capture would otherwise wrap an unquoted continuation.
@@ -73,7 +93,10 @@ let stage='startup';process.stdin.setRawMode?.(true);process.stdin.on('data',asy
const child=Bun.spawn([process.execPath,worker],{env:{...process.env,BROWSE_TERMINAL_BINARY:fake,EVALS_HERMETIC:'1'},stdout:'pipe',stderr:'pipe'}),timer=setTimeout(()=>child.kill('SIGKILL'),30000); const child=Bun.spawn([process.execPath,worker],{env:{...process.env,BROWSE_TERMINAL_BINARY:fake,EVALS_HERMETIC:'1'},stdout:'pipe',stderr:'pipe'}),timer=setTimeout(()=>child.kill('SIGKILL'),30000);
try{const [code,out,err]=await Promise.all([child.exited,new Response(child.stdout).text(),new Response(child.stderr).text()]);expect(code,out+err).toBe(0); try{const [code,out,err]=await Promise.all([child.exited,new Response(child.stdout).text(),new Response(child.stderr).text()]);expect(code,out+err).toBe(0);
const result=JSON.parse(fs.readFileSync(output,'utf8')),rows=fs.readFileSync(events,'utf8').trim().split('\n').map(s=>JSON.parse(s)); const result=JSON.parse(fs.readFileSync(output,'utf8')),rows=fs.readFileSync(events,'utf8').trim().split('\n').map(s=>JSON.parse(s));
expect(result.outcome,JSON.stringify(result)).toBe('ceiling_reached');expect(result.reviewCount).toBe(1); expect(result.outcome,JSON.stringify({result,rows})).toBe('ceiling_reached');expect(result.reviewCount).toBe(1);
const chunks=rows.filter(r=>r.type==='chunk');
expect(chunks.every(r=>['startup','current'].includes(r.stage))).toBe(true);
expect(chunks.map(r=>r.input).join('')).toBe('/plan-ceo-review\r1\r');
expect(rows.filter(r=>r.type==='input').map(r=>[r.stage,r.input])).toEqual([['startup','/plan-ceo-review\r'],['current','1\r']]);expect(rows.some(r=>r.type==='unexpected')).toBe(false); expect(rows.filter(r=>r.type==='input').map(r=>[r.stage,r.input])).toEqual([['startup','/plan-ceo-review\r'],['current','1\r']]);expect(rows.some(r=>r.type==='unexpected')).toBe(false);
expect(()=>process.kill(rows[0].pid,0)).toThrow();expect(fs.existsSync(rows[0].cwd)).toBe(false); expect(()=>process.kill(rows[0].pid,0)).toThrow();expect(fs.existsSync(rows[0].cwd)).toBe(false);
}finally{clearTimeout(timer);child.kill('SIGKILL');await child.exited; }finally{clearTimeout(timer);child.kill('SIGKILL');await child.exited;
+56
View File
@@ -0,0 +1,56 @@
/**
* Periodic /health behavior: full-log failure counts, visible partial coverage,
* comparable histories, and an unscored no-tools run. One bounded capture;
* wording follows the model, while process receipts and history are asserted.
*/
import { afterAll, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { CAPTURE_MS, CAPTURE_LONG_MS } from './helpers/eval-budgets';
import { describeE2ETier, e2eTierEnabled } from './helpers/e2e-gate';
import { EvalCollector } from './helpers/eval-store';
import {
createHealthEvalFixture, healthReportingFailures, recordHealthAttempt,
} from './helpers/health-eval-fixture';
const ROOT = path.resolve(import.meta.dir, '..');
const describeE2E = describeE2ETier('periodic');
const collector = e2eTierEnabled('periodic') ? new EvalCollector('e2e') : null;
describeE2E('/health trustworthy reporting (periodic)', () => {
test('health-reporting', async () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-health-eval-'));
try {
let fixture: ReturnType<typeof createHealthEvalFixture>;
await recordHealthAttempt(
entry => collector!.addTest(entry),
async () => {
fixture = createHealthEvalFixture(dir, ROOT);
// The runner resolves its transcript directory on import. Keep a
// skipped paid file free of that operator-state lookup/write.
const { runSkillTest } = await import('./helpers/session-runner');
return runSkillTest({
prompt: fixture.prompt,
workingDirectory: dir,
maxTurns: 18,
allowedTools: ['Bash', 'Read', 'Write', 'Glob', 'Grep'],
timeout: CAPTURE_MS,
testName: 'health-reporting',
// The collector retains the transcript in GSTACK_EVAL_DIR. Omit
// runId so the runner does not write a global heartbeat/run log.
env: { GSTACK_HOME: fixture.gstackHome },
});
},
result => {
expect(result.browseErrors).toEqual([]);
expect(healthReportingFailures(fixture)).toEqual([]);
},
);
} finally {
fs.rmSync(dir, { recursive: true, force: true });
}
}, CAPTURE_LONG_MS);
});
afterAll(async () => { await collector?.finalize(); });
+9 -1
View File
@@ -79,6 +79,7 @@ function installSkills(tmpDir: string) {
]; ];
const targetBase = path.join(tmpDir, '.claude', 'skills'); const targetBase = path.join(tmpDir, '.claude', 'skills');
const installedSkills: string[] = [];
for (const skill of skillDirs) { for (const skill of skillDirs) {
const srcPath = path.join(ROOT, skill, 'SKILL.md'); const srcPath = path.join(ROOT, skill, 'SKILL.md');
@@ -88,8 +89,11 @@ function installSkills(tmpDir: string) {
const destDir = path.join(targetBase, skillName); const destDir = path.join(targetBase, skillName);
fs.mkdirSync(destDir, { recursive: true }); fs.mkdirSync(destDir, { recursive: true });
fs.writeFileSync(path.join(destDir, 'SKILL.md'), extractSkillHead(srcPath)); fs.writeFileSync(path.join(destDir, 'SKILL.md'), extractSkillHead(srcPath));
installedSkills.push(skillName);
} }
// The names-only catalog keeps new CLI built-ins from changing the candidate
// set. Descriptions still choose the skill; no request-to-skill answer key.
// Write a CLAUDE.md with a GENERIC invoke-skills nudge — deliberately NO // Write a CLAUDE.md with a GENERIC invoke-skills nudge — deliberately NO
// per-skill routing table. These journey tests exist to catch skill // per-skill routing table. These journey tests exist to catch skill
// DESCRIPTION regressions (their touchfiles key on */SKILL.md.tmpl), and // DESCRIPTION regressions (their touchfiles key on */SKILL.md.tmpl), and
@@ -102,7 +106,11 @@ function installSkills(tmpDir: string) {
## Skill routing ## Skill routing
When the user's request matches an available skill, ALWAYS invoke it using the Skill This project uses the following installed gstack skills: ${installedSkills.join(', ')}.
Choose among this project catalog by matching the request to the skill descriptions.
The CLI's built-in skills are outside this project's workflow.
When the user's request matches an available project skill, ALWAYS invoke it using the Skill
tool as your FIRST action. Do NOT answer directly, do NOT use other tools first. tool as your FIRST action. Do NOT answer directly, do NOT use other tools first.
The skill has specialized workflows that produce better results than ad-hoc answers. The skill has specialized workflows that produce better results than ad-hoc answers.
Choose the skill by matching the request against each skill's description. Choose the skill by matching the request against each skill's description.