mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-16 09:55:29 +02:00
v1.87.4.0 fix: preserve health failures and disclose coverage (#2882)
* fix: report health failures and coverage accurately * chore: prepare health reporting release 1.87.4.0 * test: restrict routing evaluations to installed project skills * test: stabilize health selection and terminal fixtures
This commit is contained in:
@@ -1,5 +1,39 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## [1.87.4.0] - 2026-09-16
|
||||||
|
|
||||||
|
**Failed checks stay failed.**
|
||||||
|
**Health scores show what actually ran.**
|
||||||
|
|
||||||
|
`/health` now keeps each checker's exit status and counts diagnostics from its complete output. Reports still show only the final 50 log lines. Scores name the checked and unavailable categories, so a partial run carries its coverage beside the number. Runs with no checks produce no numeric score or history entry.
|
||||||
|
|
||||||
|
### The three numbers that matter
|
||||||
|
|
||||||
|
Source: the synthetic checker in `test/health-capture.test.ts`, which emits 60 type errors followed by 80 context lines and exits 2. These measurements compare the v1.87.3.0 capture example with this version under default Bash without `pipefail`. Run `bun test test/health-capture.test.ts` to verify current behavior. These are correctness measurements, not production statistics.
|
||||||
|
|
||||||
|
| Metric | Before | After | Δ |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| Reported checker exit status | 0 | 2 | +2 |
|
||||||
|
| Type errors available for scoring | 0 | 60 | +60 |
|
||||||
|
| Displayed checker log lines | 50 | 50 | 0 |
|
||||||
|
|
||||||
|
The failing checker no longer looks successful because `tail` succeeded. All 60 errors count even when the displayed tail contains only context.
|
||||||
|
|
||||||
|
### What this means for developers
|
||||||
|
|
||||||
|
You can distinguish a score backed by several checks from one based on a single available tool. Empty runs report `N/A — no checks ran`; capture errors also remain unscored and leave existing history unchanged. Trends compare only matching categories, so installing a new checker does not manufacture a regression or improvement. Run `/health` to see the score and its coverage together.
|
||||||
|
|
||||||
|
### Itemized changes
|
||||||
|
|
||||||
|
#### Fixed
|
||||||
|
|
||||||
|
- **`/health` preserves failed checks and complete diagnostic counts.** Reports show the final 50 output lines while scoring the full log and the checker's actual exit status. Temporary capture errors remain explicit errors.
|
||||||
|
- **Health scores disclose coverage.** Partial runs list checked and unavailable categories. Runs with no checks report `N/A — no checks ran`, leave numeric history unchanged, and have no trend. Score comparisons require matching categories.
|
||||||
|
|
||||||
|
#### Changed
|
||||||
|
|
||||||
|
- Routing evaluations choose among installed GStack skills, keeping built-in CLI skills outside the evaluated catalog. The model still chooses by matching the request to each skill's description.
|
||||||
|
|
||||||
## [1.87.3.0] - 2026-09-15
|
## [1.87.3.0] - 2026-09-15
|
||||||
|
|
||||||
**Changed code needs another pass.**
|
**Changed code needs another pass.**
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
# gstack digest v1.87.3.0 — regenerate/re-copy after upgrading gstack
|
# gstack digest v1.87.4.0 — regenerate/re-copy after upgrading gstack
|
||||||
|
|
||||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||||
for agent hosts without a full skill install. The full skills add workflows,
|
for agent hosts without a full skill install. The full skills add workflows,
|
||||||
|
|||||||
+76
-16
@@ -489,22 +489,55 @@ section in CLAUDE.md:
|
|||||||
Run each detected tool. For each tool:
|
Run each detected tool. For each tool:
|
||||||
|
|
||||||
1. Record the start time
|
1. Record the start time
|
||||||
2. Run the command, capturing both stdout and stderr
|
2. Run the command, capturing complete stdout and stderr in a private temporary log
|
||||||
3. Record the exit code
|
3. Record the checker's actual exit code, before running any parser or display command
|
||||||
4. Record the end time
|
4. Record the end time
|
||||||
5. Capture the last 50 lines of output for the report
|
5. Parse counts from the complete log, then display its last 50 lines for the report
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Example for each tool — run each independently
|
# Capture example — run each tool independently; adapt the command and parser.
|
||||||
START=$(date +%s)
|
(
|
||||||
tsc --noEmit 2>&1 | tail -50
|
umask 077
|
||||||
EXIT_CODE=$?
|
health_capture_error() {
|
||||||
END=$(date +%s)
|
printf 'ERROR:typecheck CAPTURE:%s\n' "$1" >&2
|
||||||
echo "TOOL:typecheck EXIT:$EXIT_CODE DURATION:$((END-START))s"
|
exit 125
|
||||||
|
}
|
||||||
|
health_log=$(mktemp "${TMPDIR:-/tmp}/gstack-health.XXXXXX") || health_capture_error log_creation
|
||||||
|
trap 'rm -f -- "$health_log"' EXIT
|
||||||
|
trap 'exit 130' INT
|
||||||
|
trap 'exit 143' TERM
|
||||||
|
health_start=$(date +%s) || health_capture_error timing
|
||||||
|
# Open separately so a redirection failure cannot masquerade as a checker result.
|
||||||
|
if ! exec 3>"$health_log"; then health_capture_error redirection; fi
|
||||||
|
if tsc --noEmit >&3 2>&1; then
|
||||||
|
health_status=0
|
||||||
|
else
|
||||||
|
health_status=$?
|
||||||
|
fi
|
||||||
|
exec 3>&-
|
||||||
|
health_end=$(date +%s) || health_capture_error timing
|
||||||
|
# awk returns a successful zero count for no matches, including empty output.
|
||||||
|
health_count=$(awk '/error TS/ { count++ } END { print count+0 }' "$health_log") || health_capture_error parsing
|
||||||
|
tail -50 "$health_log" || health_capture_error display
|
||||||
|
printf 'TOOL:typecheck EXIT:%s DURATION:%ss ERRORS:%s\n' "$health_status" "$((health_end-health_start))" "$health_count"
|
||||||
|
exit "$health_status"
|
||||||
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
Run tools sequentially (some may share resources or lock files). If a tool is not
|
Run tools sequentially in independent invocations (some may share resources or lock
|
||||||
installed or not found, record it as `SKIPPED` with reason, not as a failure.
|
files). A failed checker must not prevent later tools from running. Remember each
|
||||||
|
reported exit code and full-log counts; never use the status of `tail` or a parser
|
||||||
|
as the checker result. Guard parsers whose no-match exit is expected.
|
||||||
|
|
||||||
|
Before running a tool, check availability using the project's configured command
|
||||||
|
and local tool installation. If availability detection establishes that it is
|
||||||
|
missing, record `SKIPPED` with the reason. An executed checker returning 127 is a
|
||||||
|
failure, not evidence that the category should be skipped.
|
||||||
|
|
||||||
|
Capture failures (log creation, redirection, parsing, or display) are `ERROR`, never
|
||||||
|
`CLEAN` or `SKIPPED`. Include the cause and do not invent a category score. Report
|
||||||
|
the composite as `N/A — capture failed` if any category cannot be scored for this
|
||||||
|
reason; do not redistribute that category's weight or persist a numeric history row.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -522,6 +555,9 @@ Score each category on a 0-10 scale using this rubric:
|
|||||||
| GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) |
|
| GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) |
|
||||||
|
|
||||||
**Parsing tool output for counts:**
|
**Parsing tool output for counts:**
|
||||||
|
Use the complete captured output, not the displayed tail. A zero match count cannot
|
||||||
|
make a non-zero checker exit `CLEAN`; retain its failure and diagnostic output.
|
||||||
|
|
||||||
- **tsc:** Count lines matching `error TS` in output.
|
- **tsc:** Count lines matching `error TS` in output.
|
||||||
- **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available.
|
- **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available.
|
||||||
- **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures).
|
- **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures).
|
||||||
@@ -537,6 +573,11 @@ If a category is skipped (tool not available — includes GBrain when gbrain
|
|||||||
is not installed), redistribute its weight proportionally among the
|
is not installed), redistribute its weight proportionally among the
|
||||||
remaining categories.
|
remaining categories.
|
||||||
|
|
||||||
|
Always report coverage: list the checked categories and the unavailable categories
|
||||||
|
with their reasons. Label a numeric composite with skipped categories as **partial
|
||||||
|
coverage**. If zero checks executed, report **N/A — no checks ran**, do not compute
|
||||||
|
a numeric composite, and skip numeric history persistence and trend calculation.
|
||||||
|
|
||||||
**GBrain sub-score computation (D6):**
|
**GBrain sub-score computation (D6):**
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -579,6 +620,9 @@ Shell lint shellcheck 10/10 CLEAN 1s 0 issues
|
|||||||
GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago
|
GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago
|
||||||
|
|
||||||
COMPOSITE SCORE: 9.1 / 10
|
COMPOSITE SCORE: 9.1 / 10
|
||||||
|
Coverage: 6/6 categories checked
|
||||||
|
Checked: typecheck, lint, test, deadcode, shell, gbrain
|
||||||
|
Unavailable: none
|
||||||
|
|
||||||
Duration: 23s total
|
Duration: 23s total
|
||||||
```
|
```
|
||||||
@@ -588,6 +632,13 @@ Use these status labels:
|
|||||||
- 7-9: `WARNING`
|
- 7-9: `WARNING`
|
||||||
- 4-6: `NEEDS WORK`
|
- 4-6: `NEEDS WORK`
|
||||||
- 0-3: `CRITICAL`
|
- 0-3: `CRITICAL`
|
||||||
|
- Unavailable tool: `SKIPPED` (no score)
|
||||||
|
- Capture failure: `ERROR` (no score; composite is N/A)
|
||||||
|
|
||||||
|
For partial coverage, show e.g. `COMPOSITE SCORE: 8.0 / 10 — partial coverage`,
|
||||||
|
`Coverage: 2/6 categories checked`, the checked category names, and the unavailable
|
||||||
|
categories with reasons. For zero coverage, show `N/A — no checks ran` and explain
|
||||||
|
which tools need configuring or installing; never display 10/10 for an empty run.
|
||||||
|
|
||||||
If any category scored below 7, list the top issues from that tool's output:
|
If any category scored below 7, list the top issues from that tool's output:
|
||||||
|
|
||||||
@@ -607,7 +658,9 @@ DETAILS: Lint (3 warnings)
|
|||||||
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gstack/projects/$SLUG
|
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gstack/projects/$SLUG
|
||||||
```
|
```
|
||||||
|
|
||||||
Append one JSONL line to `~/.gstack/projects/$SLUG/health-history.jsonl`:
|
Only when a numeric composite exists, append one JSONL line to
|
||||||
|
`~/.gstack/projects/$SLUG/health-history.jsonl`. Zero-check and capture-error runs
|
||||||
|
must leave any existing history unchanged:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23}
|
{"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23}
|
||||||
@@ -636,7 +689,14 @@ eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)" && mkdir -p ~/.gst
|
|||||||
tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY"
|
tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY"
|
||||||
```
|
```
|
||||||
|
|
||||||
**If prior entries exist, show the trend:**
|
**Compare like-for-like coverage.** For each history row, form the set of categories
|
||||||
|
with non-null scores (missing fields count as null). Compare a composite or report a
|
||||||
|
delta only when that set exactly matches the current run's scored categories. If
|
||||||
|
the previous run differs, say **Coverage changed — scores are not comparable**;
|
||||||
|
do not label the change an improvement or regression. Historical rows may still be
|
||||||
|
shown with their unavailable categories marked. A current N/A result has no trend.
|
||||||
|
|
||||||
|
**If comparable prior entries exist, show the trend:**
|
||||||
|
|
||||||
```
|
```
|
||||||
HEALTH TREND (last 5 runs)
|
HEALTH TREND (last 5 runs)
|
||||||
@@ -651,7 +711,7 @@ Date Branch Score TC Lint Test Dead Shell GBrain
|
|||||||
Trend: IMPROVING (+0.9 since last run)
|
Trend: IMPROVING (+0.9 since last run)
|
||||||
```
|
```
|
||||||
|
|
||||||
**If score dropped vs the previous run:**
|
**If score dropped vs the previous run with identical coverage:**
|
||||||
1. Identify WHICH categories declined
|
1. Identify WHICH categories declined
|
||||||
2. Show the delta for each declining category
|
2. Show the delta for each declining category
|
||||||
3. Correlate with tool output -- what specific errors/warnings appeared?
|
3. Correlate with tool output -- what specific errors/warnings appeared?
|
||||||
@@ -689,7 +749,7 @@ Rank by `weight * (10 - score)` descending. Only show categories below 10.
|
|||||||
1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports.
|
1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports.
|
||||||
2. **Read-only.** Never fix issues. Present the dashboard and let the user decide.
|
2. **Read-only.** Never fix issues. Present the dashboard and let the user decide.
|
||||||
3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess.
|
3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess.
|
||||||
4. **Skipped is not failed.** If a tool isn't available, skip it gracefully and redistribute weight. Do not penalize the score.
|
4. **Skipped is not failed.** Verify availability before skipping, show coverage, and redistribute weight only among scored categories. An executed command's failure must not become a skip.
|
||||||
5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running.
|
5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running.
|
||||||
6. **Trends require history.** On first run, say "First health check -- no trend data yet. Run /health again after making changes to track progress."
|
6. **Trends require comparable history.** On the first scored run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." Changed coverage and N/A runs have no score delta.
|
||||||
7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality.
|
7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality.
|
||||||
|
|||||||
+76
-16
@@ -114,22 +114,55 @@ section in CLAUDE.md:
|
|||||||
Run each detected tool. For each tool:
|
Run each detected tool. For each tool:
|
||||||
|
|
||||||
1. Record the start time
|
1. Record the start time
|
||||||
2. Run the command, capturing both stdout and stderr
|
2. Run the command, capturing complete stdout and stderr in a private temporary log
|
||||||
3. Record the exit code
|
3. Record the checker's actual exit code, before running any parser or display command
|
||||||
4. Record the end time
|
4. Record the end time
|
||||||
5. Capture the last 50 lines of output for the report
|
5. Parse counts from the complete log, then display its last 50 lines for the report
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Example for each tool — run each independently
|
# Capture example — run each tool independently; adapt the command and parser.
|
||||||
START=$(date +%s)
|
(
|
||||||
tsc --noEmit 2>&1 | tail -50
|
umask 077
|
||||||
EXIT_CODE=$?
|
health_capture_error() {
|
||||||
END=$(date +%s)
|
printf 'ERROR:typecheck CAPTURE:%s\n' "$1" >&2
|
||||||
echo "TOOL:typecheck EXIT:$EXIT_CODE DURATION:$((END-START))s"
|
exit 125
|
||||||
|
}
|
||||||
|
health_log=$(mktemp "${TMPDIR:-/tmp}/gstack-health.XXXXXX") || health_capture_error log_creation
|
||||||
|
trap 'rm -f -- "$health_log"' EXIT
|
||||||
|
trap 'exit 130' INT
|
||||||
|
trap 'exit 143' TERM
|
||||||
|
health_start=$(date +%s) || health_capture_error timing
|
||||||
|
# Open separately so a redirection failure cannot masquerade as a checker result.
|
||||||
|
if ! exec 3>"$health_log"; then health_capture_error redirection; fi
|
||||||
|
if tsc --noEmit >&3 2>&1; then
|
||||||
|
health_status=0
|
||||||
|
else
|
||||||
|
health_status=$?
|
||||||
|
fi
|
||||||
|
exec 3>&-
|
||||||
|
health_end=$(date +%s) || health_capture_error timing
|
||||||
|
# awk returns a successful zero count for no matches, including empty output.
|
||||||
|
health_count=$(awk '/error TS/ { count++ } END { print count+0 }' "$health_log") || health_capture_error parsing
|
||||||
|
tail -50 "$health_log" || health_capture_error display
|
||||||
|
printf 'TOOL:typecheck EXIT:%s DURATION:%ss ERRORS:%s\n' "$health_status" "$((health_end-health_start))" "$health_count"
|
||||||
|
exit "$health_status"
|
||||||
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
Run tools sequentially (some may share resources or lock files). If a tool is not
|
Run tools sequentially in independent invocations (some may share resources or lock
|
||||||
installed or not found, record it as `SKIPPED` with reason, not as a failure.
|
files). A failed checker must not prevent later tools from running. Remember each
|
||||||
|
reported exit code and full-log counts; never use the status of `tail` or a parser
|
||||||
|
as the checker result. Guard parsers whose no-match exit is expected.
|
||||||
|
|
||||||
|
Before running a tool, check availability using the project's configured command
|
||||||
|
and local tool installation. If availability detection establishes that it is
|
||||||
|
missing, record `SKIPPED` with the reason. An executed checker returning 127 is a
|
||||||
|
failure, not evidence that the category should be skipped.
|
||||||
|
|
||||||
|
Capture failures (log creation, redirection, parsing, or display) are `ERROR`, never
|
||||||
|
`CLEAN` or `SKIPPED`. Include the cause and do not invent a category score. Report
|
||||||
|
the composite as `N/A — capture failed` if any category cannot be scored for this
|
||||||
|
reason; do not redistribute that category's weight or persist a numeric history row.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -147,6 +180,9 @@ Score each category on a 0-10 scale using this rubric:
|
|||||||
| GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) |
|
| GBrain (D6) | 10% | doctor=ok, queue<10, pushed <24h | doctor=warnings OR queue<100 OR pushed <72h | doctor broken OR queue>=100 OR pushed >=72h | N/A (gbrain not installed) |
|
||||||
|
|
||||||
**Parsing tool output for counts:**
|
**Parsing tool output for counts:**
|
||||||
|
Use the complete captured output, not the displayed tail. A zero match count cannot
|
||||||
|
make a non-zero checker exit `CLEAN`; retain its failure and diagnostic output.
|
||||||
|
|
||||||
- **tsc:** Count lines matching `error TS` in output.
|
- **tsc:** Count lines matching `error TS` in output.
|
||||||
- **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available.
|
- **biome/eslint/ruff:** Count lines matching error/warning patterns. Parse the summary line if available.
|
||||||
- **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures).
|
- **Tests:** Parse pass/fail counts from the test runner output. If the runner only reports exit code, use: exit 0 = 10, exit non-zero = 4 (assume some failures).
|
||||||
@@ -162,6 +198,11 @@ If a category is skipped (tool not available — includes GBrain when gbrain
|
|||||||
is not installed), redistribute its weight proportionally among the
|
is not installed), redistribute its weight proportionally among the
|
||||||
remaining categories.
|
remaining categories.
|
||||||
|
|
||||||
|
Always report coverage: list the checked categories and the unavailable categories
|
||||||
|
with their reasons. Label a numeric composite with skipped categories as **partial
|
||||||
|
coverage**. If zero checks executed, report **N/A — no checks ran**, do not compute
|
||||||
|
a numeric composite, and skip numeric history persistence and trend calculation.
|
||||||
|
|
||||||
**GBrain sub-score computation (D6):**
|
**GBrain sub-score computation (D6):**
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -204,6 +245,9 @@ Shell lint shellcheck 10/10 CLEAN 1s 0 issues
|
|||||||
GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago
|
GBrain gbrain doctor 10/10 CLEAN <1s doctor=ok, queue=3, pushed 2h ago
|
||||||
|
|
||||||
COMPOSITE SCORE: 9.1 / 10
|
COMPOSITE SCORE: 9.1 / 10
|
||||||
|
Coverage: 6/6 categories checked
|
||||||
|
Checked: typecheck, lint, test, deadcode, shell, gbrain
|
||||||
|
Unavailable: none
|
||||||
|
|
||||||
Duration: 23s total
|
Duration: 23s total
|
||||||
```
|
```
|
||||||
@@ -213,6 +257,13 @@ Use these status labels:
|
|||||||
- 7-9: `WARNING`
|
- 7-9: `WARNING`
|
||||||
- 4-6: `NEEDS WORK`
|
- 4-6: `NEEDS WORK`
|
||||||
- 0-3: `CRITICAL`
|
- 0-3: `CRITICAL`
|
||||||
|
- Unavailable tool: `SKIPPED` (no score)
|
||||||
|
- Capture failure: `ERROR` (no score; composite is N/A)
|
||||||
|
|
||||||
|
For partial coverage, show e.g. `COMPOSITE SCORE: 8.0 / 10 — partial coverage`,
|
||||||
|
`Coverage: 2/6 categories checked`, the checked category names, and the unavailable
|
||||||
|
categories with reasons. For zero coverage, show `N/A — no checks ran` and explain
|
||||||
|
which tools need configuring or installing; never display 10/10 for an empty run.
|
||||||
|
|
||||||
If any category scored below 7, list the top issues from that tool's output:
|
If any category scored below 7, list the top issues from that tool's output:
|
||||||
|
|
||||||
@@ -232,7 +283,9 @@ DETAILS: Lint (3 warnings)
|
|||||||
{{SLUG_SETUP}}
|
{{SLUG_SETUP}}
|
||||||
```
|
```
|
||||||
|
|
||||||
Append one JSONL line to `~/.gstack/projects/$SLUG/health-history.jsonl`:
|
Only when a numeric composite exists, append one JSONL line to
|
||||||
|
`~/.gstack/projects/$SLUG/health-history.jsonl`. Zero-check and capture-error runs
|
||||||
|
must leave any existing history unchanged:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23}
|
{"ts":"2026-03-31T14:30:00Z","branch":"main","score":9.1,"typecheck":10,"lint":8,"test":10,"deadcode":7,"shell":10,"gbrain":10,"duration_s":23}
|
||||||
@@ -261,7 +314,14 @@ file exists and has prior entries).
|
|||||||
tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY"
|
tail -10 ~/.gstack/projects/$SLUG/health-history.jsonl 2>/dev/null || echo "NO_HISTORY"
|
||||||
```
|
```
|
||||||
|
|
||||||
**If prior entries exist, show the trend:**
|
**Compare like-for-like coverage.** For each history row, form the set of categories
|
||||||
|
with non-null scores (missing fields count as null). Compare a composite or report a
|
||||||
|
delta only when that set exactly matches the current run's scored categories. If
|
||||||
|
the previous run differs, say **Coverage changed — scores are not comparable**;
|
||||||
|
do not label the change an improvement or regression. Historical rows may still be
|
||||||
|
shown with their unavailable categories marked. A current N/A result has no trend.
|
||||||
|
|
||||||
|
**If comparable prior entries exist, show the trend:**
|
||||||
|
|
||||||
```
|
```
|
||||||
HEALTH TREND (last 5 runs)
|
HEALTH TREND (last 5 runs)
|
||||||
@@ -276,7 +336,7 @@ Date Branch Score TC Lint Test Dead Shell GBrain
|
|||||||
Trend: IMPROVING (+0.9 since last run)
|
Trend: IMPROVING (+0.9 since last run)
|
||||||
```
|
```
|
||||||
|
|
||||||
**If score dropped vs the previous run:**
|
**If score dropped vs the previous run with identical coverage:**
|
||||||
1. Identify WHICH categories declined
|
1. Identify WHICH categories declined
|
||||||
2. Show the delta for each declining category
|
2. Show the delta for each declining category
|
||||||
3. Correlate with tool output -- what specific errors/warnings appeared?
|
3. Correlate with tool output -- what specific errors/warnings appeared?
|
||||||
@@ -314,7 +374,7 @@ Rank by `weight * (10 - score)` descending. Only show categories below 10.
|
|||||||
1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports.
|
1. **Wrap, don't replace.** Run the project's own tools. Never substitute your own analysis for what the tool reports.
|
||||||
2. **Read-only.** Never fix issues. Present the dashboard and let the user decide.
|
2. **Read-only.** Never fix issues. Present the dashboard and let the user decide.
|
||||||
3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess.
|
3. **Respect CLAUDE.md.** If `## Health Stack` is configured, use those exact commands. Do not second-guess.
|
||||||
4. **Skipped is not failed.** If a tool isn't available, skip it gracefully and redistribute weight. Do not penalize the score.
|
4. **Skipped is not failed.** Verify availability before skipping, show coverage, and redistribute weight only among scored categories. An executed command's failure must not become a skip.
|
||||||
5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running.
|
5. **Show raw output for failures.** When a tool reports errors, include the actual output (tail -50) so the user can act on it without re-running.
|
||||||
6. **Trends require history.** On first run, say "First health check -- no trend data yet. Run /health again after making changes to track progress."
|
6. **Trends require comparable history.** On the first scored run, say "First health check -- no trend data yet. Run /health again after making changes to track progress." Changed coverage and N/A runs have no score delta.
|
||||||
7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality.
|
7. **Be honest about scores.** A codebase with 100 type errors and all tests passing is not healthy. The composite score should reflect reality.
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "gstack",
|
"name": "gstack",
|
||||||
"version": "1.87.3",
|
"version": "1.87.4",
|
||||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ describe('AO completed manual DX handoff preserves report freshness',()=>{
|
|||||||
expect(E2E_TOUCHFILES[owner]).toContain('test/fixtures/dx-manual-handoff-ao.json');
|
expect(E2E_TOUCHFILES[owner]).toContain('test/fixtures/dx-manual-handoff-ao.json');
|
||||||
}
|
}
|
||||||
const arrays=[...Object.values(E2E_TOUCHFILES),...Object.values(LLM_JUDGE_TOUCHFILES),GLOBAL_TOUCHFILES];
|
const arrays=[...Object.values(E2E_TOUCHFILES),...Object.values(LLM_JUDGE_TOUCHFILES),GLOBAL_TOUCHFILES];
|
||||||
expect(arrays).toHaveLength(210);
|
expect(arrays).toHaveLength(211);
|
||||||
for(const values of arrays)for(let i=0;i<values.length;i++)expect(typeof values[i]).toBe('string');
|
for(const values of arrays)for(let i=0;i<values.length;i++)expect(typeof values[i]).toBe('string');
|
||||||
});
|
});
|
||||||
test('exact owned report precedes navigation only, with the current Exit gate recognized',()=>{
|
test('exact owned report precedes navigation only, with the current Exit gate recognized',()=>{
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
import { afterEach, describe, expect, test } from 'bun:test';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import { chmodSync, mkdtempSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
|
||||||
|
// Execute the documented capture, so the example cannot drift from its tests.
|
||||||
|
const template = readFileSync(join(import.meta.dir, '../health/SKILL.md.tmpl'), 'utf8');
|
||||||
|
const capture = template.split('## Step 2: Run Tools')[1].match(/```bash\n([\s\S]*?)```/)![1];
|
||||||
|
const temporaryDirectories: string[] = [];
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
for (const directory of temporaryDirectories.splice(0)) rmSync(directory, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
function runCapture(checker: string, setup = '') {
|
||||||
|
const directory = mkdtempSync(join(tmpdir(), 'health-capture-test-'));
|
||||||
|
temporaryDirectories.push(directory);
|
||||||
|
const logs = join(directory, 'logs');
|
||||||
|
mkdirSync(logs);
|
||||||
|
const executable = join(directory, 'tsc');
|
||||||
|
writeFileSync(executable, '#!/usr/bin/env bash\n' + checker + '\n');
|
||||||
|
chmodSync(executable, 0o755);
|
||||||
|
const result = spawnSync('/bin/bash', ['-euc', setup + '\n' + capture], {
|
||||||
|
encoding: 'utf8',
|
||||||
|
timeout: 10_000,
|
||||||
|
env: { ...process.env, PATH: directory + ':' + process.env.PATH, TMPDIR: logs },
|
||||||
|
});
|
||||||
|
expect(result.error).toBeUndefined();
|
||||||
|
expect(readdirSync(logs)).toEqual([]);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('/health command capture', () => {
|
||||||
|
test('a successful empty checker reports zero matches under set -e', () => {
|
||||||
|
const result = runCapture('exit 0');
|
||||||
|
expect(result.status).toBe(0);
|
||||||
|
expect(result.stdout).toMatch(/TOOL:typecheck EXIT:0 DURATION:\d+s ERRORS:0/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('failure survives an earlier success, stderr capture, and display', () => {
|
||||||
|
const result = runCapture('echo "source.ts: error TS2322: incorrect type" >&2\nexit 2', 'true');
|
||||||
|
expect(result.status).toBe(2);
|
||||||
|
expect(result.stdout).toContain('incorrect type');
|
||||||
|
expect(result.stdout).toMatch(/EXIT:2 DURATION:\d+s ERRORS:1/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('counts findings outside the displayed tail and prints only fifty log lines', () => {
|
||||||
|
const result = runCapture([
|
||||||
|
'for ((i=1; i<=60; i++)); do echo "source.ts: error TS2322: finding $i"; done',
|
||||||
|
'for ((i=1; i<=80; i++)); do echo "detail $i"; done',
|
||||||
|
'exit 2',
|
||||||
|
].join('\n'));
|
||||||
|
expect(result.status).toBe(2);
|
||||||
|
expect(result.stdout).toMatch(/EXIT:2 DURATION:\d+s ERRORS:60/);
|
||||||
|
const lines = result.stdout.trimEnd().split('\n');
|
||||||
|
expect(lines.length).toBe(51);
|
||||||
|
expect(lines[0]).toBe('detail 31');
|
||||||
|
expect(lines[49]).toBe('detail 80');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('an executed checker returning 127 remains a failure', () => {
|
||||||
|
const result = runCapture('echo "a checker dependency failed" >&2\nexit 127');
|
||||||
|
expect(result.status).toBe(127);
|
||||||
|
expect(result.stdout).toContain('EXIT:127');
|
||||||
|
expect(result.stdout).not.toContain('SKIPPED');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('empty failing output does not inherit a successful parser status', () => {
|
||||||
|
const result = runCapture('exit 1');
|
||||||
|
expect(result.status).toBe(1);
|
||||||
|
expect(result.stdout).toMatch(/EXIT:1 DURATION:\d+s ERRORS:0/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('capture log permissions are private even with a permissive caller umask', () => {
|
||||||
|
const result = runCapture('stat -c %a "$TMPDIR"/gstack-health.* 2>/dev/null || stat -f %Lp "$TMPDIR"/gstack-health.*', 'umask 000');
|
||||||
|
expect(result.status).toBe(0);
|
||||||
|
expect(result.stdout.split('\n')[0]).toBe('600');
|
||||||
|
});
|
||||||
|
|
||||||
|
test.each([
|
||||||
|
['log_creation', 'mktemp() { return 1; }'],
|
||||||
|
['redirection', 'mktemp() { printf "%s/missing/log\\n" "$TMPDIR"; }'],
|
||||||
|
['parsing', 'awk() { return 2; }'],
|
||||||
|
['display', 'tail() { return 1; }'],
|
||||||
|
])('%s failure reports an error instead of a clean result', (phase, setup) => {
|
||||||
|
const result = runCapture('exit 0', setup);
|
||||||
|
expect(result.status).toBe(125);
|
||||||
|
expect(result.stderr).toContain('ERROR:typecheck CAPTURE:' + phase);
|
||||||
|
expect(result.stdout).not.toContain('TOOL:typecheck EXIT:0');
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,139 @@
|
|||||||
|
/** Free fixture/recording checks; never import or invoke the paid runner. */
|
||||||
|
import { afterEach, describe, expect, test } from 'bun:test';
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import * as os from 'node:os';
|
||||||
|
import * as path from 'node:path';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import {
|
||||||
|
createHealthEvalFixture, healthReportingFailures, recordHealthAttempt,
|
||||||
|
} from './helpers/health-eval-fixture';
|
||||||
|
import type { SkillTestResult } from './helpers/session-runner';
|
||||||
|
import type { EvalTestEntry } from './helpers/eval-store';
|
||||||
|
import { E2E_TIERS, E2E_TOUCHFILES, GLOBAL_TOUCHFILES, selectTests } from './helpers/touchfiles';
|
||||||
|
import { isPaidTestFile } from './helpers/paid-test-set';
|
||||||
|
|
||||||
|
const ROOT = path.resolve(import.meta.dir, '..');
|
||||||
|
const dirs: string[] = [];
|
||||||
|
afterEach(() => { for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); });
|
||||||
|
|
||||||
|
test('health behavior is selected as periodic and paid while capture regressions stay free', () => {
|
||||||
|
const selectedBy = (file: string) => selectTests([file], E2E_TOUCHFILES, GLOBAL_TOUCHFILES).selected;
|
||||||
|
expect(selectedBy('health/SKILL.md.tmpl')).toContain('health-reporting');
|
||||||
|
expect(selectedBy('test/helpers/health-eval-fixture.ts')).toEqual(['health-reporting']);
|
||||||
|
expect(E2E_TIERS['health-reporting']).toBe('periodic');
|
||||||
|
expect(isPaidTestFile('test/skill-e2e-health.test.ts')).toBe(true);
|
||||||
|
expect(isPaidTestFile('test/health-capture.test.ts')).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
function fixture(prefix = 'health-fixture-test-') {
|
||||||
|
const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
|
||||||
|
dirs.push(dir);
|
||||||
|
return createHealthEvalFixture(dir, ROOT);
|
||||||
|
}
|
||||||
|
|
||||||
|
function writePassingEvidence(f: ReturnType<typeof fixture>) {
|
||||||
|
fs.writeFileSync(path.join(f.dir, 'partial-report.md'), `Type check: 0/10 CRITICAL, 60 errors.
|
||||||
|
Tests: 10/10 CLEAN, 5 passed.
|
||||||
|
Checked: type check, tests. Unavailable: lint, dead code, shell lint, GBrain (not installed).
|
||||||
|
COMPOSITE SCORE: **5.6 / 10 — partial coverage**
|
||||||
|
Coverage changed; prior test-only history is not comparable.
|
||||||
|
`);
|
||||||
|
fs.writeFileSync(path.join(f.dir, 'no-tools-report.md'), 'COMPOSITE SCORE: N/A — no checks ran.\n');
|
||||||
|
fs.appendFileSync(path.join(f.gstackHome, 'projects', 'partial', 'health-history.jsonl'), JSON.stringify({
|
||||||
|
score: 5.6, typecheck: 0, test: 10, lint: null, deadcode: null, shell: null, gbrain: null,
|
||||||
|
}) + '\n');
|
||||||
|
fs.writeFileSync(f.receipts, 'typecheck\ntest\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('/health eval fixtures', () => {
|
||||||
|
test('extracts all six real workflow steps and redirects persistent paths', () => {
|
||||||
|
const f = fixture();
|
||||||
|
const skill = fs.readFileSync(path.join(f.dir, 'health-SKILL.md'), 'utf-8');
|
||||||
|
expect(skill).not.toContain('## Preamble (run first)');
|
||||||
|
expect(skill).not.toContain('~/.gstack');
|
||||||
|
expect(skill).not.toContain('~/.claude/skills/gstack/bin/gstack-slug');
|
||||||
|
for (let step = 1; step <= 6; step++) expect(skill).toContain(`## Step ${step}:`);
|
||||||
|
for (const project of ['partial', 'no-tools']) {
|
||||||
|
const result = spawnSync(path.join(f.dir, 'bin', 'gstack-slug'), [], {
|
||||||
|
cwd: path.join(f.dir, project), encoding: 'utf-8', timeout: 10_000,
|
||||||
|
env: { ...process.env, GSTACK_HOME: f.gstackHome, GSTACK_PROJECT_SLUG: '' },
|
||||||
|
});
|
||||||
|
expect(result.status).toBe(0);
|
||||||
|
expect(result.stdout).toContain(`SLUG=${project}\n`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('real checker exits nonzero and its final 50 lines hide all 60 errors', () => {
|
||||||
|
const f = fixture();
|
||||||
|
const result = spawnSync('bash', ['-c', 'bash ./check-typecheck.sh 2>&1'], {
|
||||||
|
cwd: path.join(f.dir, 'partial'), encoding: 'utf-8', timeout: 10_000,
|
||||||
|
});
|
||||||
|
expect(result.status).toBe(2);
|
||||||
|
expect(result.stdout.match(/error TS/g)).toHaveLength(60);
|
||||||
|
expect(result.stdout.trimEnd().split('\n').slice(-50).join('\n')).not.toContain('error TS');
|
||||||
|
expect(fs.readFileSync(f.receipts, 'utf-8')).toBe('typecheck\n');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('both checkers record to their fixed path without shell environment setup', () => {
|
||||||
|
const f = fixture("health-fixture-test-' space-");
|
||||||
|
for (const [script, status] of [['check-typecheck.sh', 2], ['check-tests.sh', 0]] as const) {
|
||||||
|
const result = spawnSync('bash', [script], {
|
||||||
|
cwd: path.join(f.dir, 'partial'), encoding: 'utf-8', timeout: 10_000,
|
||||||
|
});
|
||||||
|
expect(result.status).toBe(status);
|
||||||
|
}
|
||||||
|
expect(fs.readFileSync(f.receipts, 'utf-8')).toBe('typecheck\ntest\n');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('accepts truthful reports and history, rejects absent evidence', () => {
|
||||||
|
const f = fixture();
|
||||||
|
expect(healthReportingFailures(f).length).toBeGreaterThan(0);
|
||||||
|
writePassingEvidence(f);
|
||||||
|
expect(healthReportingFailures(f)).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test.each(['false-clean', 'false-clean-category', 'tail-count', 'numeric-empty', 'hidden-coverage', 'unlike-trend', 'contradictory-trend', 'extra-empty-history', 'missing-receipts', 'project-edit'])('%s is a failed behavior eval', failure => {
|
||||||
|
const f = fixture();
|
||||||
|
writePassingEvidence(f);
|
||||||
|
const report = path.join(f.dir, 'partial-report.md');
|
||||||
|
if (failure === 'false-clean') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('5.6', '10'));
|
||||||
|
if (failure === 'false-clean-category') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('0/10 CRITICAL', '10/10 CLEAN'));
|
||||||
|
if (failure === 'tail-count') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('60 errors', '0 errors'));
|
||||||
|
if (failure === 'numeric-empty') fs.writeFileSync(path.join(f.dir, 'no-tools-report.md'), 'COMPOSITE SCORE: 10 — no checks ran.');
|
||||||
|
if (failure === 'hidden-coverage') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace(' — partial coverage', ''));
|
||||||
|
if (failure === 'unlike-trend') fs.writeFileSync(report, fs.readFileSync(report, 'utf-8').replace('Coverage changed; prior test-only history is not comparable.', 'Health worsened from 10 to 5.6.'));
|
||||||
|
if (failure === 'contradictory-trend') fs.appendFileSync(report, 'Trend: REGRESSING (-4.4 since last run)\n');
|
||||||
|
if (failure === 'extra-empty-history') fs.appendFileSync(path.join(f.gstackHome, 'projects', 'no-tools', 'health-history.jsonl'), '{"score":10}\n');
|
||||||
|
if (failure === 'missing-receipts') fs.rmSync(f.receipts);
|
||||||
|
if (failure === 'project-edit') fs.appendFileSync(path.join(f.dir, 'partial', 'check-typecheck.sh'), '# attempted fix\n');
|
||||||
|
expect(healthReportingFailures(f).length).toBeGreaterThan(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('/health eval recording', () => {
|
||||||
|
const success = {
|
||||||
|
exitReason: 'success', duration: 5, output: 'report', transcript: [],
|
||||||
|
costEstimate: { estimatedCost: 0.1, turnsUsed: 2, estimatedTokens: 100 }, model: 'fixture',
|
||||||
|
} as SkillTestResult;
|
||||||
|
|
||||||
|
test.each(['success', 'assertion', 'timeout', 'throw'])('%s records exactly once with matching pass status', async scenario => {
|
||||||
|
const entries: EvalTestEntry[] = [];
|
||||||
|
let verified = false;
|
||||||
|
const attempt = recordHealthAttempt(
|
||||||
|
entry => entries.push(entry),
|
||||||
|
async () => {
|
||||||
|
if (scenario === 'throw') throw new Error('runner failed');
|
||||||
|
return { ...success, exitReason: scenario === 'timeout' ? 'timeout' : 'success' };
|
||||||
|
},
|
||||||
|
() => { verified = true; if (scenario === 'assertion') throw new Error('false dashboard'); },
|
||||||
|
);
|
||||||
|
if (scenario === 'success') await attempt;
|
||||||
|
else await expect(attempt).rejects.toThrow();
|
||||||
|
expect(entries).toHaveLength(1);
|
||||||
|
expect(entries[0].passed).toBe(scenario === 'success');
|
||||||
|
expect(entries[0].tier).toBe('e2e');
|
||||||
|
expect(verified).toBe(scenario === 'success' || scenario === 'assertion');
|
||||||
|
if (scenario === 'assertion') expect(entries[0].output).toContain('false dashboard');
|
||||||
|
if (scenario === 'timeout') expect(entries[0].exit_reason).toBe('timeout');
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,209 @@
|
|||||||
|
/** Isolated fixtures and assertions for the single periodic /health capture. */
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import * as path from 'node:path';
|
||||||
|
import { extractSkillSections } from './skill-fixture';
|
||||||
|
import type { SkillTestResult } from './session-runner';
|
||||||
|
import type { EvalTestEntry } from './eval-store';
|
||||||
|
|
||||||
|
export const HEALTH_EVAL_ID = 'health-reporting';
|
||||||
|
export const HEALTH_EVAL_SECTIONS = [
|
||||||
|
'Step 1: Detect Health Stack',
|
||||||
|
'Step 2: Run Tools',
|
||||||
|
'Step 3: Score Each Category',
|
||||||
|
'Step 4: Present Dashboard',
|
||||||
|
'Step 5: Persist to Health History',
|
||||||
|
'Step 6: Trend Analysis + Recommendations',
|
||||||
|
'Important Rules',
|
||||||
|
];
|
||||||
|
|
||||||
|
const PRIOR_HISTORY = JSON.stringify({
|
||||||
|
ts: '2026-01-01T00:00:00Z', branch: 'unknown', score: 10,
|
||||||
|
typecheck: null, lint: null, test: 10, deadcode: null, shell: null,
|
||||||
|
gbrain: null, duration_s: 1,
|
||||||
|
}) + '\n';
|
||||||
|
|
||||||
|
export interface HealthEvalFixture {
|
||||||
|
dir: string;
|
||||||
|
gstackHome: string;
|
||||||
|
receipts: string;
|
||||||
|
prompt: string;
|
||||||
|
projectFiles: Record<string, string>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function projectSnapshot(dir: string): Record<string, string> {
|
||||||
|
const files: Record<string, string> = {};
|
||||||
|
const walk = (relative: string) => {
|
||||||
|
for (const entry of fs.readdirSync(path.join(dir, relative), { withFileTypes: true })) {
|
||||||
|
const name = path.join(relative, entry.name);
|
||||||
|
if (entry.isDirectory()) { files[name + '/'] = '<directory>'; walk(name); }
|
||||||
|
else files[name] = entry.isSymbolicLink() ? `<symlink:${fs.readlinkSync(path.join(dir, name))}>` : fs.readFileSync(path.join(dir, name), 'utf-8');
|
||||||
|
}
|
||||||
|
};
|
||||||
|
for (const project of ['partial', 'no-tools']) walk(project);
|
||||||
|
return files;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createHealthEvalFixture(dir: string, repoRoot: string): HealthEvalFixture {
|
||||||
|
const gstackHome = path.join(dir, 'gstack-state');
|
||||||
|
const receipts = path.join(dir, 'checker-runs.txt');
|
||||||
|
// Instrumentation belongs to the fixture, not to the model's shell setup.
|
||||||
|
// Single-quote escaping also covers temporary paths containing apostrophes.
|
||||||
|
const quotedReceipts = "'" + receipts.replaceAll("'", "'\"'\"'") + "'";
|
||||||
|
fs.mkdirSync(path.join(dir, 'bin'), { recursive: true });
|
||||||
|
fs.copyFileSync(path.join(repoRoot, 'bin', 'gstack-slug'), path.join(dir, 'bin', 'gstack-slug'));
|
||||||
|
fs.chmodSync(path.join(dir, 'bin', 'gstack-slug'), 0o755);
|
||||||
|
|
||||||
|
// Only redirect installed paths. The workflow and score/history rules come
|
||||||
|
// from the real generated skill, without its unrelated shared preamble.
|
||||||
|
const skill = extractSkillSections(path.join(repoRoot, 'health'), HEALTH_EVAL_SECTIONS)
|
||||||
|
.replaceAll('~/.claude/skills/gstack/bin/gstack-slug', path.join(dir, 'bin', 'gstack-slug'))
|
||||||
|
.replaceAll('~/.gstack', gstackHome);
|
||||||
|
fs.writeFileSync(path.join(dir, 'health-SKILL.md'), skill);
|
||||||
|
|
||||||
|
for (const project of ['partial', 'no-tools']) {
|
||||||
|
const projectDir = path.join(dir, project);
|
||||||
|
fs.mkdirSync(projectDir);
|
||||||
|
fs.writeFileSync(path.join(projectDir, '.project.yaml'), `name: ${project}\n`);
|
||||||
|
const historyDir = path.join(gstackHome, 'projects', project);
|
||||||
|
fs.mkdirSync(historyDir, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(historyDir, 'health-history.jsonl'), PRIOR_HISTORY);
|
||||||
|
}
|
||||||
|
fs.writeFileSync(path.join(dir, 'partial', 'CLAUDE.md'), `# Partial project
|
||||||
|
|
||||||
|
## Health Stack
|
||||||
|
|
||||||
|
- typecheck: bash ./check-typecheck.sh
|
||||||
|
- test: bash ./check-tests.sh
|
||||||
|
|
||||||
|
Only the listed tools are available. Lint, dead-code, shell-lint, and GBrain
|
||||||
|
tools are not installed. This configuration is final; do not install tools.
|
||||||
|
`);
|
||||||
|
fs.writeFileSync(path.join(dir, 'no-tools', 'CLAUDE.md'), `# No-tools project
|
||||||
|
|
||||||
|
## Health Stack
|
||||||
|
|
||||||
|
No health tools are configured or installed for any category.
|
||||||
|
This configuration is final; do not install tools or substitute other checks.
|
||||||
|
`);
|
||||||
|
fs.writeFileSync(path.join(dir, 'partial', 'check-typecheck.sh'), `#!/usr/bin/env bash
|
||||||
|
printf 'typecheck\\n' >> ${quotedReceipts}
|
||||||
|
for ((i = 1; i <= 60; i++)); do
|
||||||
|
printf 'src/file%s.ts(1,1): error TS2322: Type mismatch.\\n' "$i" >&2
|
||||||
|
done
|
||||||
|
for ((i = 1; i <= 80; i++)); do
|
||||||
|
printf 'Additional diagnostic context %s\\n' "$i"
|
||||||
|
done
|
||||||
|
exit 2
|
||||||
|
`);
|
||||||
|
fs.writeFileSync(path.join(dir, 'partial', 'check-tests.sh'), `#!/usr/bin/env bash
|
||||||
|
printf 'test\\n' >> ${quotedReceipts}
|
||||||
|
printf '5 pass\\n0 fail\\n'
|
||||||
|
`);
|
||||||
|
|
||||||
|
return {
|
||||||
|
dir, gstackHome, receipts, projectFiles: projectSnapshot(dir),
|
||||||
|
prompt: `Read health-SKILL.md and run its /health workflow, Steps 1–6, for
|
||||||
|
the partial project first and the no-tools project second. Each has its own
|
||||||
|
CLAUDE.md with the final Health Stack configuration. Run commands from the
|
||||||
|
corresponding project directory. These are local fixtures without Git remotes.
|
||||||
|
|
||||||
|
Save each complete dashboard, details, trends, and recommendations as
|
||||||
|
partial-report.md or no-tools-report.md in ${dir}. Keep the dashboard's
|
||||||
|
COMPOSITE SCORE label. Existing health histories are available under
|
||||||
|
${gstackHome}/projects/<project>/health-history.jsonl; apply the skill's normal
|
||||||
|
history rules. GSTACK_HOME already points at this isolated state directory.
|
||||||
|
|
||||||
|
Do not modify project files, install tools, or ask to change either Health Stack.
|
||||||
|
Only write the reports and any history updates required by the supplied skill.
|
||||||
|
Finish after producing both reports.`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Validate behavior, allowing ordinary Markdown/wording variation in reports. */
|
||||||
|
export function healthReportingFailures(fixture: HealthEvalFixture): string[] {
|
||||||
|
const failures: string[] = [];
|
||||||
|
const check = (ok: boolean, message: string) => { if (!ok) failures.push(message); };
|
||||||
|
const read = (file: string) => fs.existsSync(file) ? fs.readFileSync(file, 'utf-8') : '';
|
||||||
|
const plain = (text: string) => text.replace(/[*_`]/g, '');
|
||||||
|
const partial = plain(read(path.join(fixture.dir, 'partial-report.md')));
|
||||||
|
const empty = plain(read(path.join(fixture.dir, 'no-tools-report.md')));
|
||||||
|
const composite = (report: string) => report.match(/composite\s+score\s*[:|]?\s*(N\/A|\d+(?:\.\d+)?)/i)?.[1];
|
||||||
|
check(composite(partial) === '5.6', 'partial coverage must score 5.6, preserving the failing typecheck');
|
||||||
|
check(/partial\s+coverage/i.test(partial), 'numeric score must be labeled as partial coverage');
|
||||||
|
const typecheckRows = partial.split('\n').filter(line => /\btype\s*check\b/i.test(line));
|
||||||
|
check(typecheckRows.some(line => /(?:\b0\s*\/\s*10\b|\|\s*0\s*\|)/.test(line)
|
||||||
|
&& /\b(?:critical|fail(?:ed|ure)?|error)\b/i.test(line) && !/\bclean\b/i.test(line)),
|
||||||
|
'dashboard must report the failing typecheck as 0/10, not clean');
|
||||||
|
check(/\b60\s+(?:\w+\s+){0,2}(?:errors|diagnostics|findings)\b/i.test(partial)
|
||||||
|
|| /(?:errors|diagnostics|findings)[^\n]{0,20}\b60\b/i.test(partial),
|
||||||
|
'report must count all 60 errors before the final 50 output lines');
|
||||||
|
check(/(?:coverage|checked|executed)/i.test(partial), 'partial report must disclose checked coverage');
|
||||||
|
check(/type\s*check/i.test(partial) && /tests?/i.test(partial), 'checked category names must be visible');
|
||||||
|
for (const category of ['lint', 'dead[ -]?code', 'shell(?:[ -]?lint)?', 'gbrain']) {
|
||||||
|
const unavailable = '(?:unavailable|skipped|not (?:installed|configured|available|found))';
|
||||||
|
check(new RegExp(`${unavailable}[\\s\\S]{0,240}${category}|${category}[^\\n]{0,120}${unavailable}`, 'i').test(partial),
|
||||||
|
`partial report must name unavailable ${category}`);
|
||||||
|
}
|
||||||
|
check(/(?:coverage|categor(?:y|ies)|checks)[\s\S]{0,160}(?:chang|differ|not compar)/i.test(partial)
|
||||||
|
|| /(?:chang|differ|not compar)[\s\S]{0,160}(?:coverage|categor(?:y|ies)|checks)/i.test(partial),
|
||||||
|
'partial report must flag changed coverage instead of comparing unlike histories');
|
||||||
|
check(!/[+-]\s*4\.4\b|trend\s*:\s*(?:improving|regressing|worsening)/i.test(partial),
|
||||||
|
'partial report must not calculate a trend delta against different coverage');
|
||||||
|
check(composite(empty)?.toUpperCase() === 'N/A', 'no-tools composite must be N/A');
|
||||||
|
check(/(?:no|zero|0)\s+(?:health\s+)?checks?\s+(?:ran|run|executed|available)|no\s+tools/i.test(empty),
|
||||||
|
'no-tools report must explain that no checks ran');
|
||||||
|
|
||||||
|
const history = read(path.join(fixture.gstackHome, 'projects', 'partial', 'health-history.jsonl'));
|
||||||
|
check(history.startsWith(PRIOR_HISTORY), 'partial run must preserve its prior history row');
|
||||||
|
const rows = history.trim().split('\n').filter(Boolean);
|
||||||
|
check(rows.length === 2, 'partial run must append exactly one history row');
|
||||||
|
try {
|
||||||
|
const row = JSON.parse(rows.at(-1) || '{}');
|
||||||
|
check(row.score === 5.6 && row.typecheck === 0 && row.test === 10,
|
||||||
|
'persisted partial scores must reflect all diagnostics and the actual exit status');
|
||||||
|
check(['lint', 'deadcode', 'shell', 'gbrain'].every(category => row[category] === null),
|
||||||
|
'unavailable categories must persist as null');
|
||||||
|
} catch {
|
||||||
|
failures.push('partial history must remain valid JSONL');
|
||||||
|
}
|
||||||
|
check(read(path.join(fixture.gstackHome, 'projects', 'no-tools', 'health-history.jsonl')) === PRIOR_HISTORY,
|
||||||
|
'no-tools run must leave its history unchanged');
|
||||||
|
const runs = read(fixture.receipts).trim().split('\n');
|
||||||
|
check(runs.includes('typecheck') && runs.includes('test'), 'both configured checkers must actually run');
|
||||||
|
check(JSON.stringify(projectSnapshot(fixture.dir)) === JSON.stringify(fixture.projectFiles),
|
||||||
|
'health must leave project files unchanged');
|
||||||
|
return failures;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One attempt records once, after all assertions; throws remain failures. */
|
||||||
|
export async function recordHealthAttempt(
|
||||||
|
record: (entry: EvalTestEntry) => void,
|
||||||
|
capture: () => Promise<SkillTestResult>,
|
||||||
|
verify: (result: SkillTestResult) => void,
|
||||||
|
): Promise<void> {
|
||||||
|
const started = Date.now();
|
||||||
|
let result: SkillTestResult | undefined;
|
||||||
|
let passed = false;
|
||||||
|
let failure: unknown;
|
||||||
|
try {
|
||||||
|
result = await capture();
|
||||||
|
if (result.exitReason !== 'success') throw new Error(`health capture ended: ${result.exitReason}`);
|
||||||
|
verify(result);
|
||||||
|
passed = true;
|
||||||
|
} catch (error) {
|
||||||
|
failure = error;
|
||||||
|
throw error;
|
||||||
|
} finally {
|
||||||
|
record({
|
||||||
|
name: HEALTH_EVAL_ID, suite: 'health', tier: 'e2e', passed,
|
||||||
|
duration_ms: result?.duration ?? Date.now() - started,
|
||||||
|
cost_usd: result?.costEstimate.estimatedCost ?? 0,
|
||||||
|
turns_used: result?.costEstimate.turnsUsed,
|
||||||
|
tokens_used: result?.costEstimate.estimatedTokens,
|
||||||
|
transcript: result?.transcript,
|
||||||
|
output: [result?.output, failure === undefined ? '' : String(failure)].filter(Boolean).join('\n').slice(-4000),
|
||||||
|
exit_reason: result?.exitReason === 'success' && !passed ? 'assertion_failed' : result?.exitReason ?? 'runner_error',
|
||||||
|
model: result?.model,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -582,6 +582,9 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
|
|||||||
// Document-release
|
// Document-release
|
||||||
'document-release': ['document-release/**', 'test/skill-e2e-workflow.test.ts'],
|
'document-release': ['document-release/**', 'test/skill-e2e-workflow.test.ts'],
|
||||||
|
|
||||||
|
// /health result capture, coverage, and comparable history (model behavior).
|
||||||
|
'health-reporting': ['health/**', 'test/skill-e2e-health.test.ts', 'test/helpers/health-eval-fixture.ts'],
|
||||||
|
|
||||||
// Codex (Claude E2E — tests /codex skill via Claude)
|
// Codex (Claude E2E — tests /codex skill via Claude)
|
||||||
'codex-review': ['codex/**', 'test/skill-e2e-workflow.test.ts'],
|
'codex-review': ['codex/**', 'test/skill-e2e-workflow.test.ts'],
|
||||||
|
|
||||||
@@ -1186,6 +1189,7 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic'> = {
|
|||||||
|
|
||||||
// Document-release — gate (CHANGELOG guardrail)
|
// Document-release — gate (CHANGELOG guardrail)
|
||||||
'document-release': 'gate',
|
'document-release': 'gate',
|
||||||
|
'health-reporting': 'periodic',
|
||||||
|
|
||||||
// Codex — periodic (Opus, requires codex CLI)
|
// Codex — periodic (Opus, requires codex CLI)
|
||||||
'codex-review': 'periodic',
|
'codex-review': 'periodic',
|
||||||
|
|||||||
@@ -69,7 +69,7 @@ describe('native repeated report permission identity',()=>{
|
|||||||
|
|
||||||
for (const variant of ['basic', 'intervening', 'cropped', 'same-basename', 'path-cropped']) test.skipIf(process.platform==='win32')(`real fake CLI grants each current request once: ${variant}`,async()=>{
|
for (const variant of ['basic', 'intervening', 'cropped', 'same-basename', 'path-cropped']) test.skipIf(process.platform==='win32')(`real fake CLI grants each current request once: ${variant}`,async()=>{
|
||||||
const intervening = variant === 'intervening';
|
const intervening = variant === 'intervening';
|
||||||
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'count-edit-pty-'));const fake=path.join(dir,'fake-claude');const worker=path.join(dir,'worker.ts');const events=path.join(dir,'events.jsonl');const output=path.join(dir,'output.json');const expected=path.join(dir,variant==='same-basename'?'PLAN.md':'report.md');fs.writeFileSync(expected,'original');
|
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'ce-'));const fake=path.join(dir,'fake-claude');const worker=path.join(dir,'worker.ts');const events=path.join(dir,'events.jsonl');const output=path.join(dir,'output.json');const expected=path.join(dir,variant==='same-basename'?'PLAN.md':'report.md');fs.writeFileSync(expected,'original');
|
||||||
const cropped=capturedAc.rows.find(row=>row.job===5)!;
|
const cropped=capturedAc.rows.find(row=>row.job===5)!;
|
||||||
let screen=variant==='path-cropped' ? capturedPath.screen.replace(capturedPath.screen.split('\n')[0]!,expected).replaceAll(path.dirname(capturedPath.expected),path.dirname(expected)).replaceAll(path.basename(capturedPath.expected),'report.md')
|
let screen=variant==='path-cropped' ? capturedPath.screen.replace(capturedPath.screen.split('\n')[0]!,expected).replaceAll(path.dirname(capturedPath.expected),path.dirname(expected)).replaceAll(path.basename(capturedPath.expected),'report.md')
|
||||||
: variant==='cropped' ? cropped.screen.replaceAll(path.dirname(cropped.hook.expected),path.dirname(expected)).replaceAll(path.basename(cropped.hook.expected),'report.md')
|
: variant==='cropped' ? cropped.screen.replaceAll(path.dirname(cropped.hook.expected),path.dirname(expected)).replaceAll(path.basename(cropped.hook.expected),'report.md')
|
||||||
|
|||||||
@@ -10,6 +10,26 @@ import owned from './fixtures/plan-count-owned-permission-v.json';
|
|||||||
import {E2E_TOUCHFILES,selectTests} from './helpers/touchfiles';
|
import {E2E_TOUCHFILES,selectTests} from './helpers/touchfiles';
|
||||||
const quote=(s:string)=>s.split('\n').map(row=>'> '+row).join('\n');
|
const quote=(s:string)=>s.split('\n').map(row=>'> '+row).join('\n');
|
||||||
|
|
||||||
|
// A PTY transports bytes, not command-sized stdin events. Share the framing
|
||||||
|
// code with the fake CLI so fragmented grants exercise the same receiver.
|
||||||
|
function commandBuffer(){
|
||||||
|
let pending='';
|
||||||
|
return (chunk:string)=>{
|
||||||
|
pending+=chunk;const commands:string[]=[];let end:number;
|
||||||
|
while((end=pending.indexOf('\r'))!==-1){commands.push(pending.slice(0,end+1));pending=pending.slice(end+1);}
|
||||||
|
return commands;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
test('fake CLI preserves command bytes across fragmented and coalesced PTY input',()=>{
|
||||||
|
const expected=['/plan-ceo-review\r','1\r'];
|
||||||
|
for(const chunks of [expected,['/plan-ceo-review\r','1','\r'],[...expected.join('')],[expected.join('')]]){
|
||||||
|
const receive=commandBuffer();expect(chunks.flatMap(receive)).toEqual(expected);
|
||||||
|
}
|
||||||
|
const receive=commandBuffer();
|
||||||
|
expect(receive('1')).toEqual([]);expect(receive('\r2\rtrailing')).toEqual(['1\r','2\r']);
|
||||||
|
expect(receive('\r')).toEqual(['trailing\r']); // no unexpected bytes are discarded
|
||||||
|
});
|
||||||
|
|
||||||
test('the exact wholly quoted AK pane is handled so the dispatcher sends no fallback',()=>{
|
test('the exact wholly quoted AK pane is handled so the dispatcher sends no fallback',()=>{
|
||||||
const screen=quote(exact.screen);
|
const screen=quote(exact.screen);
|
||||||
expect(classifyPlanCountFrame(screen)).toBe('permission');
|
expect(classifyPlanCountFrame(screen)).toBe('permission');
|
||||||
@@ -44,7 +64,7 @@ test('the regression selects exactly the existing permission consumers',()=>{
|
|||||||
test.skipIf(process.platform==='win32')('real dispatcher ignores quoted pane then grants the fresh owned native request once',async()=>{
|
test.skipIf(process.platform==='win32')('real dispatcher ignores quoted pane then grants the fresh owned native request once',async()=>{
|
||||||
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'count-quoted-frame-')),fake=path.join(dir,'fake-claude'),worker=path.join(dir,'worker.ts'),events=path.join(dir,'events.jsonl'),output=path.join(dir,'result.json'),report=path.join(dir,'report.md');
|
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'count-quoted-frame-')),fake=path.join(dir,'fake-claude'),worker=path.join(dir,'worker.ts'),events=path.join(dir,'events.jsonl'),output=path.join(dir,'result.json'),report=path.join(dir,'report.md');
|
||||||
fs.writeFileSync(report,'original');
|
fs.writeFileSync(report,'original');
|
||||||
fs.writeFileSync(fake,`#!${process.execPath}\n`+String.raw`
|
fs.writeFileSync(fake,`#!${process.execPath}\nconst receive=(${commandBuffer.toString()})();\n`+String.raw`
|
||||||
import fs from 'node:fs';import path from 'node:path';
|
import fs from 'node:fs';import path from 'node:path';
|
||||||
const item=JSON.parse(process.env.QUOTED_FRAME_CASE),sid='quoted-frame-main',log=e=>fs.appendFileSync(item.events,JSON.stringify(e)+'\n');
|
const item=JSON.parse(process.env.QUOTED_FRAME_CASE),sid='quoted-frame-main',log=e=>fs.appendFileSync(item.events,JSON.stringify(e)+'\n');
|
||||||
const transcript=path.join(process.env.CLAUDE_CONFIG_DIR,'projects','owned',sid+'.jsonl');fs.mkdirSync(path.dirname(transcript),{recursive:true});
|
const transcript=path.join(process.env.CLAUDE_CONFIG_DIR,'projects','owned',sid+'.jsonl');fs.mkdirSync(path.dirname(transcript),{recursive:true});
|
||||||
@@ -56,14 +76,14 @@ const hook=async name=>{for(const entry of settings.hooks[name]??[]){if(entry.ma
|
|||||||
const p=Bun.spawn(['bash','-c',entry.hooks[0].command],{stdin:new Blob([JSON.stringify(event)]),stdout:'pipe',stderr:'pipe'});
|
const p=Bun.spawn(['bash','-c',entry.hooks[0].command],{stdin:new Blob([JSON.stringify(event)]),stdout:'pipe',stderr:'pipe'});
|
||||||
const [code,out,err]=await Promise.all([p.exited,new Response(p.stdout).text(),new Response(p.stderr).text()]);if(code||out||err)throw Error('Hook failed');}};
|
const [code,out,err]=await Promise.all([p.exited,new Response(p.stdout).text(),new Response(p.stderr).text()]);if(code||out||err)throw Error('Hook failed');}};
|
||||||
const pane=item.screen.replaceAll('PLAN.md',item.report),paint=s=>process.stdout.write('\x1b[2J\x1b[H'+s.replaceAll('\n','\r\n'));
|
const pane=item.screen.replaceAll('PLAN.md',item.report),paint=s=>process.stdout.write('\x1b[2J\x1b[H'+s.replaceAll('\n','\r\n'));
|
||||||
let stage='startup';process.stdin.setRawMode?.(true);process.stdin.on('data',async data=>{
|
let stage='startup';process.stdin.setRawMode?.(true);const dispatch=async input=>{
|
||||||
const input=data.toString();log({type:'input',stage,input});
|
log({type:'input',stage,input});
|
||||||
if(stage==='startup'){stage='quoted';paint(item.quotedScreen);setTimeout(async()=>{await hook('PreToolUse');stage='current';paint(pane);},4200);return;}
|
if(stage==='startup'){stage='quoted';paint(item.quotedScreen);setTimeout(async()=>{await hook('PreToolUse');stage='current';paint(pane);},4200);return;}
|
||||||
if(stage!=='current'){log({type:'unexpected'});return;}
|
if(stage!=='current'){log({type:'unexpected'});return;}
|
||||||
if(input!=='1\r')throw Error('One-time grant changed');stage='done';await hook('PostToolUse');
|
if(input!=='1\r')throw Error('One-time grant changed');stage='done';await hook('PostToolUse');
|
||||||
const q={header:'Finding',question:'Apply the reviewed fix?',options:[{label:'Fix'},{label:'Keep'}]};
|
const q={header:'Finding',question:'Apply the reviewed fix?',options:[{label:'Fix'},{label:'Keep'}]};
|
||||||
native('assistant',[{type:'tool_use',name:'AskUserQuestion',id:'finding',input:{questions:[q]}}]);native('user',[{type:'tool_result',tool_use_id:'finding',content:'Answered'}],{toolUseResult:{answers:{[q.question]:'Fix'}}});paint('Done.\n');
|
native('assistant',[{type:'tool_use',name:'AskUserQuestion',id:'finding',input:{questions:[q]}}]);native('user',[{type:'tool_result',tool_use_id:'finding',content:'Answered'}],{toolUseResult:{answers:{[q.question]:'Fix'}}});paint('Done.\n');
|
||||||
});process.on('SIGINT',()=>process.exit(0));process.stdin.resume();
|
};process.stdin.on('data',async data=>{const chunk=data.toString();log({type:'chunk',stage,input:chunk});for(const input of receive(chunk))await dispatch(input);});process.on('SIGINT',()=>process.exit(0));process.stdin.resume();
|
||||||
`);fs.chmodSync(fake,0o755);
|
`);fs.chmodSync(fake,0o755);
|
||||||
// Keep every physical terminal row inside the quote; adding a prefix to an
|
// Keep every physical terminal row inside the quote; adding a prefix to an
|
||||||
// already120-column capture would otherwise wrap an unquoted continuation.
|
// already120-column capture would otherwise wrap an unquoted continuation.
|
||||||
@@ -73,7 +93,10 @@ let stage='startup';process.stdin.setRawMode?.(true);process.stdin.on('data',asy
|
|||||||
const child=Bun.spawn([process.execPath,worker],{env:{...process.env,BROWSE_TERMINAL_BINARY:fake,EVALS_HERMETIC:'1'},stdout:'pipe',stderr:'pipe'}),timer=setTimeout(()=>child.kill('SIGKILL'),30000);
|
const child=Bun.spawn([process.execPath,worker],{env:{...process.env,BROWSE_TERMINAL_BINARY:fake,EVALS_HERMETIC:'1'},stdout:'pipe',stderr:'pipe'}),timer=setTimeout(()=>child.kill('SIGKILL'),30000);
|
||||||
try{const [code,out,err]=await Promise.all([child.exited,new Response(child.stdout).text(),new Response(child.stderr).text()]);expect(code,out+err).toBe(0);
|
try{const [code,out,err]=await Promise.all([child.exited,new Response(child.stdout).text(),new Response(child.stderr).text()]);expect(code,out+err).toBe(0);
|
||||||
const result=JSON.parse(fs.readFileSync(output,'utf8')),rows=fs.readFileSync(events,'utf8').trim().split('\n').map(s=>JSON.parse(s));
|
const result=JSON.parse(fs.readFileSync(output,'utf8')),rows=fs.readFileSync(events,'utf8').trim().split('\n').map(s=>JSON.parse(s));
|
||||||
expect(result.outcome,JSON.stringify(result)).toBe('ceiling_reached');expect(result.reviewCount).toBe(1);
|
expect(result.outcome,JSON.stringify({result,rows})).toBe('ceiling_reached');expect(result.reviewCount).toBe(1);
|
||||||
|
const chunks=rows.filter(r=>r.type==='chunk');
|
||||||
|
expect(chunks.every(r=>['startup','current'].includes(r.stage))).toBe(true);
|
||||||
|
expect(chunks.map(r=>r.input).join('')).toBe('/plan-ceo-review\r1\r');
|
||||||
expect(rows.filter(r=>r.type==='input').map(r=>[r.stage,r.input])).toEqual([['startup','/plan-ceo-review\r'],['current','1\r']]);expect(rows.some(r=>r.type==='unexpected')).toBe(false);
|
expect(rows.filter(r=>r.type==='input').map(r=>[r.stage,r.input])).toEqual([['startup','/plan-ceo-review\r'],['current','1\r']]);expect(rows.some(r=>r.type==='unexpected')).toBe(false);
|
||||||
expect(()=>process.kill(rows[0].pid,0)).toThrow();expect(fs.existsSync(rows[0].cwd)).toBe(false);
|
expect(()=>process.kill(rows[0].pid,0)).toThrow();expect(fs.existsSync(rows[0].cwd)).toBe(false);
|
||||||
}finally{clearTimeout(timer);child.kill('SIGKILL');await child.exited;
|
}finally{clearTimeout(timer);child.kill('SIGKILL');await child.exited;
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
/**
|
||||||
|
* Periodic /health behavior: full-log failure counts, visible partial coverage,
|
||||||
|
* comparable histories, and an unscored no-tools run. One bounded capture;
|
||||||
|
* wording follows the model, while process receipts and history are asserted.
|
||||||
|
*/
|
||||||
|
import { afterAll, expect, test } from 'bun:test';
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import * as os from 'node:os';
|
||||||
|
import * as path from 'node:path';
|
||||||
|
import { CAPTURE_MS, CAPTURE_LONG_MS } from './helpers/eval-budgets';
|
||||||
|
import { describeE2ETier, e2eTierEnabled } from './helpers/e2e-gate';
|
||||||
|
import { EvalCollector } from './helpers/eval-store';
|
||||||
|
import {
|
||||||
|
createHealthEvalFixture, healthReportingFailures, recordHealthAttempt,
|
||||||
|
} from './helpers/health-eval-fixture';
|
||||||
|
|
||||||
|
const ROOT = path.resolve(import.meta.dir, '..');
|
||||||
|
const describeE2E = describeE2ETier('periodic');
|
||||||
|
const collector = e2eTierEnabled('periodic') ? new EvalCollector('e2e') : null;
|
||||||
|
|
||||||
|
describeE2E('/health trustworthy reporting (periodic)', () => {
|
||||||
|
test('health-reporting', async () => {
|
||||||
|
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-health-eval-'));
|
||||||
|
try {
|
||||||
|
let fixture: ReturnType<typeof createHealthEvalFixture>;
|
||||||
|
await recordHealthAttempt(
|
||||||
|
entry => collector!.addTest(entry),
|
||||||
|
async () => {
|
||||||
|
fixture = createHealthEvalFixture(dir, ROOT);
|
||||||
|
// The runner resolves its transcript directory on import. Keep a
|
||||||
|
// skipped paid file free of that operator-state lookup/write.
|
||||||
|
const { runSkillTest } = await import('./helpers/session-runner');
|
||||||
|
return runSkillTest({
|
||||||
|
prompt: fixture.prompt,
|
||||||
|
workingDirectory: dir,
|
||||||
|
maxTurns: 18,
|
||||||
|
allowedTools: ['Bash', 'Read', 'Write', 'Glob', 'Grep'],
|
||||||
|
timeout: CAPTURE_MS,
|
||||||
|
testName: 'health-reporting',
|
||||||
|
// The collector retains the transcript in GSTACK_EVAL_DIR. Omit
|
||||||
|
// runId so the runner does not write a global heartbeat/run log.
|
||||||
|
env: { GSTACK_HOME: fixture.gstackHome },
|
||||||
|
});
|
||||||
|
},
|
||||||
|
result => {
|
||||||
|
expect(result.browseErrors).toEqual([]);
|
||||||
|
expect(healthReportingFailures(fixture)).toEqual([]);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
fs.rmSync(dir, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
}, CAPTURE_LONG_MS);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterAll(async () => { await collector?.finalize(); });
|
||||||
@@ -79,6 +79,7 @@ function installSkills(tmpDir: string) {
|
|||||||
];
|
];
|
||||||
|
|
||||||
const targetBase = path.join(tmpDir, '.claude', 'skills');
|
const targetBase = path.join(tmpDir, '.claude', 'skills');
|
||||||
|
const installedSkills: string[] = [];
|
||||||
|
|
||||||
for (const skill of skillDirs) {
|
for (const skill of skillDirs) {
|
||||||
const srcPath = path.join(ROOT, skill, 'SKILL.md');
|
const srcPath = path.join(ROOT, skill, 'SKILL.md');
|
||||||
@@ -88,8 +89,11 @@ function installSkills(tmpDir: string) {
|
|||||||
const destDir = path.join(targetBase, skillName);
|
const destDir = path.join(targetBase, skillName);
|
||||||
fs.mkdirSync(destDir, { recursive: true });
|
fs.mkdirSync(destDir, { recursive: true });
|
||||||
fs.writeFileSync(path.join(destDir, 'SKILL.md'), extractSkillHead(srcPath));
|
fs.writeFileSync(path.join(destDir, 'SKILL.md'), extractSkillHead(srcPath));
|
||||||
|
installedSkills.push(skillName);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The names-only catalog keeps new CLI built-ins from changing the candidate
|
||||||
|
// set. Descriptions still choose the skill; no request-to-skill answer key.
|
||||||
// Write a CLAUDE.md with a GENERIC invoke-skills nudge — deliberately NO
|
// Write a CLAUDE.md with a GENERIC invoke-skills nudge — deliberately NO
|
||||||
// per-skill routing table. These journey tests exist to catch skill
|
// per-skill routing table. These journey tests exist to catch skill
|
||||||
// DESCRIPTION regressions (their touchfiles key on */SKILL.md.tmpl), and
|
// DESCRIPTION regressions (their touchfiles key on */SKILL.md.tmpl), and
|
||||||
@@ -102,7 +106,11 @@ function installSkills(tmpDir: string) {
|
|||||||
|
|
||||||
## Skill routing
|
## Skill routing
|
||||||
|
|
||||||
When the user's request matches an available skill, ALWAYS invoke it using the Skill
|
This project uses the following installed gstack skills: ${installedSkills.join(', ')}.
|
||||||
|
Choose among this project catalog by matching the request to the skill descriptions.
|
||||||
|
The CLI's built-in skills are outside this project's workflow.
|
||||||
|
|
||||||
|
When the user's request matches an available project skill, ALWAYS invoke it using the Skill
|
||||||
tool as your FIRST action. Do NOT answer directly, do NOT use other tools first.
|
tool as your FIRST action. Do NOT answer directly, do NOT use other tools first.
|
||||||
The skill has specialized workflows that produce better results than ad-hoc answers.
|
The skill has specialized workflows that produce better results than ad-hoc answers.
|
||||||
Choose the skill by matching the request against each skill's description.
|
Choose the skill by matching the request against each skill's description.
|
||||||
|
|||||||
Reference in New Issue
Block a user