mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 22:45:02 +01:00
v1.92.26.0 feat(retro): friction by skill from the project timeline (GST-632) (#3757)
* feat(retro): show friction by skill from the project timeline (GST-632) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (question coverage follows question_tuning; example row wording) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
f739756389
commit
6a2c38da05
11 files changed
+424
-15
No files matched your search
@@ -1,5 +1,22 @@
|
||||
# Changelog
|
||||
|
||||
## [1.92.26.0] - 2026-11-01
|
||||
|
||||
**/retro shows which skills fought you this week: a "Friction by skill" row lists the top three skills by questions asked, errors and aborts in this project.** Before, /retro counted skill runs but said nothing about which skills caused back-and-forth, so a skill that kept stopping to ask questions or kept failing never showed up in the report you use to decide what to fix next.
|
||||
|
||||
### Added
|
||||
- **`gstack-retro-metrics` prints `FRICTION:` lines** (top three skills, with runs, questions, typed answers, errors and aborts) plus a `FRICTION_SOURCES:` line, computed by `lib/retro-friction.ts` for the exact retro window. Runs and outcomes come from this project's `timeline.jsonl`, which is written on every run whether or not telemetry is on, unattended sessions included. Questions come from this project's question log when question tuning is on; when it is off, the row says questions are not tracked instead of showing zero. Only outcomes actually recorded as error or abort count, machine-made answers are not counted as questions, and questions the Claude Code hook logs without knowing which skill asked are left out, so no question counts twice and no `/unknown` row appears. Another project's logs never count. If Bun is missing or the helper fails, the script prints `FRICTION: unavailable` and the rest of the metrics are unchanged.
|
||||
- **/retro shows a "Friction by skill" row** when any skill had friction in the window, for example `/ship 7 in 12 runs (4 questions, 3 aborts)`.
|
||||
|
||||
### Changed
|
||||
- **/retro's optional-input notes are shorter**: each one says "Read the path it prints" instead of repeating the path, which keeps /retro's always-loaded prompt under its size cap with the new row.
|
||||
|
||||
### Tests
|
||||
- `test/retro-friction.test.ts`: window edges, explicit outcomes only (the stop hook's `unknown` completions are ignored), hook rows taking precedence over agent rows, hook rows with no skill left out, repeated tool calls, machine answers, unsafe skill names, malformed lines, ties and the top three.
|
||||
- `test/gstack-retro-metrics.test.ts`: a project with both logs plus another project's timeline (not counted), an empty prior window, and a failing `bun` that still ends with `RETRO_METRICS_END: ok`.
|
||||
|
||||
Thanks to @thesohamdatta for the idea in #3417. This is a new implementation; no code from that PR was used.
|
||||
|
||||
## [1.92.25.0] - 2026-11-01
|
||||
|
||||
- In a repository that merges the base during `/ship` (any `queue_mode` but `stamp-at-merge`), Step 3 now stops by name when an earlier merge, cherry-pick or revert is still unfinished (`STOP: an unfinished cherry-pick is in progress (CHERRY_PICK_HEAD): …`), with any conflicted files listed, and runs no `git merge`. Conflicted files left with no operation in progress, as after a conflicted `git stash pop`, stop with those files named instead of a merge message. Before, both fell through to `git merge`, so on a machine whose shared-tree guard refuses `git merge`, the guard's refusal was the only message (GST-732). A branch or tag named `MERGE_HEAD`, `CHERRY_PICK_HEAD` or `REVERT_HEAD` is not mistaken for one. The ship skill's Step 3 now names the `MERGE: up to date` outcome.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.92.25.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.92.26.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
@@ -29,6 +29,8 @@
|
||||
# expression ("24 hours ago"). Default: "7 days ago".
|
||||
# --until optional window end (compare mode's prior window).
|
||||
|
||||
GSTACK_BUN_CONFIG=/dev/null; case "${OSTYPE:-}" in msys*|cygwin*) if command -v cygpath >/dev/null 2>&1; then GSTACK_BUN_CONFIG=NUL; fi ;; esac
|
||||
|
||||
BASE=""
|
||||
SINCE="7 days ago"
|
||||
UNTIL=""
|
||||
@@ -379,4 +381,35 @@ fi
|
||||
[ -f "$_GH/analytics/skill-usage.jsonl" ] && echo "SKILL_USAGE_LOG: present ($_GH/analytics/skill-usage.jsonl)" || echo "SKILL_USAGE_LOG: absent"
|
||||
[ -f "$_GH/analytics/eureka.jsonl" ] && echo "EUREKA_LOG: present ($_GH/analytics/eureka.jsonl)" || echo "EUREKA_LOG: absent"
|
||||
|
||||
# ── Friction by skill (this project's timeline + question log) ──────────────
|
||||
# lib/retro-friction.ts does the counting; the window bounds come from git's own
|
||||
# date parser so they match the commit queries above. Any failure prints
|
||||
# FRICTION: unavailable and the script still reaches RETRO_METRICS_END.
|
||||
_FR_SLUG=$("$_BIN/gstack-slug" --get SLUG 2>/dev/null || true)
|
||||
_FR_SINCE=$(git rev-parse "--since=$SINCE" 2>/dev/null | sed -n 's/^--max-age=//p' | head -1)
|
||||
_FR_UNTIL=""
|
||||
[ -n "$UNTIL" ] && _FR_UNTIL=$(git rev-parse "--until=$UNTIL" 2>/dev/null | sed -n 's/^--min-age=//p' | head -1)
|
||||
_FR_QLOG=""
|
||||
[ "$("$_BIN/gstack-config" get question_tuning 2>/dev/null)" = "true" ] && _FR_QLOG="$_GH/projects/$_FR_SLUG/question-log.jsonl"
|
||||
_FR_LIB="$_SCRIPT_DIR/../lib/retro-friction.ts"
|
||||
case "$(uname -s)" in
|
||||
MINGW*|MSYS*|CYGWIN*) command -v cygpath >/dev/null 2>&1 && _FR_LIB="$(cygpath -m "$_FR_LIB")" ;;
|
||||
esac
|
||||
if [ -z "$_FR_SLUG" ]; then
|
||||
echo "FRICTION: unavailable (no project slug)"
|
||||
elif [ -z "$_FR_SINCE" ] || { [ -n "$UNTIL" ] && [ -z "$_FR_UNTIL" ]; }; then
|
||||
echo "FRICTION: unavailable (window bounds did not resolve)"
|
||||
elif ! command -v bun >/dev/null 2>&1; then
|
||||
echo "FRICTION: unavailable (bun not found)"
|
||||
else
|
||||
_FR_OUT=$(bun --no-env-file --config="${GSTACK_BUN_CONFIG:-/dev/null}" "$_FR_LIB" \
|
||||
--timeline "$_GH/projects/$_FR_SLUG/timeline.jsonl" \
|
||||
--questions "$_FR_QLOG" \
|
||||
--since-epoch "$_FR_SINCE" --until-epoch "$_FR_UNTIL" 2>/dev/null) || _FR_OUT=""
|
||||
case "$_FR_OUT" in
|
||||
*FRICTION:*) printf '%s\n' "$_FR_OUT" ;;
|
||||
*) echo "FRICTION: unavailable (friction helper failed)" ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
echo "RETRO_METRICS_END: ok"
|
||||
@@ -0,0 +1,172 @@
|
||||
/**
|
||||
* retro-friction — the "Friction by skill" numbers /retro shows, computed from
|
||||
* two per-project logs so the retro prompt reads finished lines instead of
|
||||
* aggregating JSONL itself.
|
||||
*
|
||||
* projects/<slug>/timeline.jsonl gstack-skill-start / gstack-skill-end,
|
||||
* written on every run: runs (started)
|
||||
* and explicit error / abort outcomes
|
||||
* projects/<slug>/question-log.jsonl gstack-question-log (question tuning
|
||||
* on): questions a person answered
|
||||
*
|
||||
* Only explicit outcomes count: the stop hook's `unknown` completions and
|
||||
* unmatched starts are not aborts. Machine answers (auto-decided, spawned-env
|
||||
* denies, Codex imports) are not questions. The Claude Code question hook logs
|
||||
* skill `unknown` (it cannot tell which skill asked), so those rows are not
|
||||
* counted; the agent's own rows carry the skill. Per skill and session,
|
||||
* hook-captured rows that do name a skill win over the agent's rows for the
|
||||
* same questions, so one question counts once. Skill names outside ^[a-z0-9-]{1,64}$ are dropped so log text
|
||||
* never reaches the prompt. Idea credit: @thesohamdatta (#3417).
|
||||
*
|
||||
* CLI (bin/gstack-retro-metrics):
|
||||
* --timeline <file> --questions <file> --since-epoch <s> [--until-epoch <s>]
|
||||
* prints FRICTION_SOURCES: and FRICTION: lines.
|
||||
*/
|
||||
import * as fs from 'node:fs';
|
||||
|
||||
export interface SkillFriction {
|
||||
skill: string;
|
||||
runs: number;
|
||||
questions: number;
|
||||
typed: number;
|
||||
errors: number;
|
||||
aborts: number;
|
||||
score: number;
|
||||
}
|
||||
|
||||
export interface FrictionInput {
|
||||
timelineLines: string[];
|
||||
questionLines: string[];
|
||||
sinceMs: number;
|
||||
untilMs?: number;
|
||||
}
|
||||
|
||||
const SKILL_RE = /^[a-z0-9-]{1,64}$/;
|
||||
const HUMAN_SOURCES = new Set(['agent', 'hook', 'auq-other']);
|
||||
export const FRICTION_TOP = 3;
|
||||
|
||||
function parse(line: string): Record<string, unknown> | null {
|
||||
if (!line.trim()) return null;
|
||||
try {
|
||||
const j = JSON.parse(line);
|
||||
return j && typeof j === 'object' && !Array.isArray(j) ? j : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function skillOf(j: Record<string, unknown>): string | null {
|
||||
if (typeof j.skill !== 'string') return null;
|
||||
const s = j.skill.replace(/^\//, '').toLowerCase();
|
||||
return SKILL_RE.test(s) ? s : null;
|
||||
}
|
||||
|
||||
function inWindow(j: Record<string, unknown>, sinceMs: number, untilMs?: number): boolean {
|
||||
if (typeof j.ts !== 'string') return false;
|
||||
const t = Date.parse(j.ts);
|
||||
if (Number.isNaN(t) || t < sinceMs) return false;
|
||||
return untilMs === undefined || t <= untilMs;
|
||||
}
|
||||
|
||||
export function frictionBySkill(input: FrictionInput): SkillFriction[] {
|
||||
const bySkill = new Map<string, SkillFriction>();
|
||||
const entry = (skill: string): SkillFriction => {
|
||||
let e = bySkill.get(skill);
|
||||
if (!e) {
|
||||
e = { skill, runs: 0, questions: 0, typed: 0, errors: 0, aborts: 0, score: 0 };
|
||||
bySkill.set(skill, e);
|
||||
}
|
||||
return e;
|
||||
};
|
||||
|
||||
for (const line of input.timelineLines) {
|
||||
const j = parse(line);
|
||||
if (!j || !inWindow(j, input.sinceMs, input.untilMs)) continue;
|
||||
const skill = skillOf(j);
|
||||
if (!skill) continue;
|
||||
if (j.event === 'started') entry(skill).runs++;
|
||||
else if (j.event === 'completed' && j.outcome === 'error') entry(skill).errors++;
|
||||
else if (j.event === 'completed' && j.outcome === 'abort') entry(skill).aborts++;
|
||||
}
|
||||
|
||||
const groups = new Map<string, { skill: string; hook: Record<string, unknown>[]; agent: Record<string, unknown>[] }>();
|
||||
const seenToolUse = new Set<string>();
|
||||
for (const line of input.questionLines) {
|
||||
const j = parse(line);
|
||||
if (!j || !inWindow(j, input.sinceMs, input.untilMs)) continue;
|
||||
const skill = skillOf(j);
|
||||
const source = typeof j.source === 'string' ? j.source : 'agent';
|
||||
if (!skill || skill === 'unknown' || !HUMAN_SOURCES.has(source)) continue;
|
||||
const toolUse = typeof j.tool_use_id === 'string' && j.tool_use_id ? j.tool_use_id : '';
|
||||
if (toolUse) {
|
||||
const key = `${source}:${toolUse}`;
|
||||
if (seenToolUse.has(key)) continue;
|
||||
seenToolUse.add(key);
|
||||
}
|
||||
const session = typeof j.session_id === 'string' ? j.session_id : '';
|
||||
const gkey = `${skill}\u0000${session}`;
|
||||
let g = groups.get(gkey);
|
||||
if (!g) {
|
||||
g = { skill, hook: [], agent: [] };
|
||||
groups.set(gkey, g);
|
||||
}
|
||||
const hookCaptured = source === 'hook' || (source === 'auq-other' && toolUse !== '');
|
||||
(hookCaptured ? g.hook : g.agent).push(j);
|
||||
}
|
||||
for (const g of groups.values()) {
|
||||
const rows = g.hook.length > 0 ? g.hook : g.agent;
|
||||
const e = entry(g.skill);
|
||||
e.questions += rows.length;
|
||||
e.typed += rows.filter((r) => r.source === 'auq-other').length;
|
||||
}
|
||||
|
||||
for (const e of bySkill.values()) e.score = e.questions + e.errors + e.aborts;
|
||||
return [...bySkill.values()]
|
||||
.filter((e) => e.score > 0)
|
||||
.sort((a, b) => b.score - a.score || (a.skill < b.skill ? -1 : a.skill > b.skill ? 1 : 0));
|
||||
}
|
||||
|
||||
export function formatFriction(
|
||||
rows: SkillFriction[],
|
||||
sources: { timeline: boolean; questions: boolean },
|
||||
): string[] {
|
||||
const out = [
|
||||
`FRICTION_SOURCES: timeline=${sources.timeline ? 'present' : 'absent'} questions=${sources.questions ? 'present' : 'absent'}`,
|
||||
];
|
||||
if (rows.length === 0) return [...out, 'FRICTION: none'];
|
||||
for (const e of rows.slice(0, FRICTION_TOP)) {
|
||||
out.push(
|
||||
`FRICTION: /${e.skill} score=${e.score} runs=${e.runs} questions=${e.questions} typed=${e.typed} errors=${e.errors} aborts=${e.aborts}`,
|
||||
);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function readLines(file: string | undefined): string[] | null {
|
||||
if (!file || !fs.existsSync(file)) return null;
|
||||
return fs.readFileSync(file, 'utf-8').split('\n');
|
||||
}
|
||||
|
||||
if (import.meta.main) {
|
||||
const args = process.argv.slice(2);
|
||||
const arg = (name: string): string | undefined => {
|
||||
const i = args.indexOf(name);
|
||||
return i >= 0 ? args[i + 1] : undefined;
|
||||
};
|
||||
const since = Number(arg('--since-epoch'));
|
||||
const untilRaw = arg('--until-epoch');
|
||||
const until = untilRaw === undefined || untilRaw === '' ? undefined : Number(untilRaw);
|
||||
if (!Number.isFinite(since) || (until !== undefined && !Number.isFinite(until))) {
|
||||
console.log('FRICTION: unavailable (window bounds did not resolve)');
|
||||
process.exit(0);
|
||||
}
|
||||
const timeline = readLines(arg('--timeline'));
|
||||
const questions = readLines(arg('--questions'));
|
||||
const rows = frictionBySkill({
|
||||
timelineLines: timeline ?? [],
|
||||
questionLines: questions ?? [],
|
||||
sinceMs: since * 1000,
|
||||
untilMs: until === undefined ? undefined : until * 1000,
|
||||
});
|
||||
for (const line of formatFriction(rows, { timeline: timeline !== null, questions: questions !== null })) console.log(line);
|
||||
}
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.92.25",
|
||||
"version": "1.92.26",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
+7
-5
@@ -305,15 +305,16 @@ Also check `RETRO_REF`: if it is not `origin/<default>` (local-only repo, missin
|
||||
| `TEST_FILES_TOTAL` / `TEST_FILES_CHANGED` / `REGRESSION_TEST_COMMITS` / `REGRESSION_COMMIT` | Test health: repo-wide test file count, test files changed in window, `test(qa):` / `test(design):` / `test: coverage` commits |
|
||||
| `VERSION_RANGE` | First → last VERSION file value in the window (when tracked) |
|
||||
| `TEAM_STREAK` / `USER_STREAK` | Consecutive commit days with anchor date (Step 11) |
|
||||
| `RETRO_CONTEXT` / `GREPTILE_HISTORY` / `TODOS_FILE` / `SKILL_USAGE_LOG` / `EUREKA_LOG` | Presence of optional inputs — Read the ones marked present |
|
||||
| `RETRO_CONTEXT` / `GREPTILE_HISTORY` / `TODOS_FILE` / `SKILL_USAGE_LOG` / `EUREKA_LOG` | Presence of optional inputs |
|
||||
| `FRICTION` / `FRICTION_SOURCES` | Friction by skill, top 3 (`questions=absent`: untracked) |
|
||||
|
||||
**Optional inputs** (Read each file the script marks `present`):
|
||||
|
||||
- `RETRO_CONTEXT: present` → Read the `retro-context.md` path that line printed. It is user-authored and may contain meeting notes, calendar events, decisions, and other context that doesn't appear in git history. Incorporate it into the retro narrative where relevant.
|
||||
- `GREPTILE_HISTORY: present` → Read the `greptile-history.md` path that line printed. Filter entries to the retro window by date. Count by type: `fix`, `fp`, `already-fixed`. Signal ratio = `(fix + already-fixed) / (fix + already-fixed + fp)`. Skip unparseable lines silently; if no entries fall in the window, skip the Greptile metric row.
|
||||
- `RETRO_CONTEXT: present` → Read the path it prints. It is user-authored and may contain meeting notes, calendar events, decisions, and other context that doesn't appear in git history. Incorporate it into the retro narrative where relevant.
|
||||
- `GREPTILE_HISTORY: present` → Read the path it prints. Filter entries to the retro window by date. Count by type: `fix`, `fp`, `already-fixed`. Signal ratio = `(fix + already-fixed) / (fix + already-fixed + fp)`. Skip unparseable lines silently; if no entries fall in the window, skip the Greptile metric row.
|
||||
- `TODOS_FILE: present` → Read `TODOS.md`. Compute: total open TODOs (exclude the `## Completed` section), P0/P1 count, P2 count, items completed this period (Completed entries dated within the window), items added this period (cross-reference `COMMIT:` lines that touched TODOS.md).
|
||||
- `SKILL_USAGE_LOG: present` → Read the path it prints (`$GSTACK_STATE_ROOT/analytics/skill-usage.jsonl`). Filter to the window by `ts`. Separate skill activations (no `event` field) from hook fires (`event: "hook_fire"`). Aggregate by skill name.
|
||||
- `EUREKA_LOG: present` → Read the path it prints (`$GSTACK_STATE_ROOT/analytics/eureka.jsonl`). Filter to the window by `ts`. For each eureka moment note the skill that flagged it, the branch, and a one-line summary of the insight.
|
||||
- `SKILL_USAGE_LOG: present` → Read the path it prints. Filter to the window by `ts`. Separate skill activations (no `event` field) from hook fires (`event: "hook_fire"`). Aggregate by skill name.
|
||||
- `EUREKA_LOG: present` → Read the path it prints. Filter to the window by `ts`. For each eureka moment note the skill that flagged it, the branch, and a one-line summary of the insight.
|
||||
|
||||
### Step 2: Compute Metrics
|
||||
|
||||
@@ -363,6 +364,7 @@ Conditional rows (skip each when its input is absent or empty in the window):
|
||||
```
|
||||
| Backlog Health | N open (X P0/P1, Y P2) · Z completed this period |
|
||||
| Skill Usage | /ship(12) /qa(8) /review(5) · 3 safety hook fires |
|
||||
| Friction by skill | /ship 7 in 12 runs (4 questions, 3 aborts) · /qa 5 in 6 (5 questions) |
|
||||
| Eureka Moments | 2 this period |
|
||||
```
|
||||
|
||||
|
||||
+7
-5
@@ -150,15 +150,16 @@ Also check `RETRO_REF`: if it is not `origin/<default>` (local-only repo, missin
|
||||
| `TEST_FILES_TOTAL` / `TEST_FILES_CHANGED` / `REGRESSION_TEST_COMMITS` / `REGRESSION_COMMIT` | Test health: repo-wide test file count, test files changed in window, `test(qa):` / `test(design):` / `test: coverage` commits |
|
||||
| `VERSION_RANGE` | First → last VERSION file value in the window (when tracked) |
|
||||
| `TEAM_STREAK` / `USER_STREAK` | Consecutive commit days with anchor date (Step 11) |
|
||||
| `RETRO_CONTEXT` / `GREPTILE_HISTORY` / `TODOS_FILE` / `SKILL_USAGE_LOG` / `EUREKA_LOG` | Presence of optional inputs — Read the ones marked present |
|
||||
| `RETRO_CONTEXT` / `GREPTILE_HISTORY` / `TODOS_FILE` / `SKILL_USAGE_LOG` / `EUREKA_LOG` | Presence of optional inputs |
|
||||
| `FRICTION` / `FRICTION_SOURCES` | Friction by skill, top 3 (`questions=absent`: untracked) |
|
||||
|
||||
**Optional inputs** (Read each file the script marks `present`):
|
||||
|
||||
- `RETRO_CONTEXT: present` → Read the `retro-context.md` path that line printed. It is user-authored and may contain meeting notes, calendar events, decisions, and other context that doesn't appear in git history. Incorporate it into the retro narrative where relevant.
|
||||
- `GREPTILE_HISTORY: present` → Read the `greptile-history.md` path that line printed. Filter entries to the retro window by date. Count by type: `fix`, `fp`, `already-fixed`. Signal ratio = `(fix + already-fixed) / (fix + already-fixed + fp)`. Skip unparseable lines silently; if no entries fall in the window, skip the Greptile metric row.
|
||||
- `RETRO_CONTEXT: present` → Read the path it prints. It is user-authored and may contain meeting notes, calendar events, decisions, and other context that doesn't appear in git history. Incorporate it into the retro narrative where relevant.
|
||||
- `GREPTILE_HISTORY: present` → Read the path it prints. Filter entries to the retro window by date. Count by type: `fix`, `fp`, `already-fixed`. Signal ratio = `(fix + already-fixed) / (fix + already-fixed + fp)`. Skip unparseable lines silently; if no entries fall in the window, skip the Greptile metric row.
|
||||
- `TODOS_FILE: present` → Read `TODOS.md`. Compute: total open TODOs (exclude the `## Completed` section), P0/P1 count, P2 count, items completed this period (Completed entries dated within the window), items added this period (cross-reference `COMMIT:` lines that touched TODOS.md).
|
||||
- `SKILL_USAGE_LOG: present` → Read the path it prints (`$GSTACK_STATE_ROOT/analytics/skill-usage.jsonl`). Filter to the window by `ts`. Separate skill activations (no `event` field) from hook fires (`event: "hook_fire"`). Aggregate by skill name.
|
||||
- `EUREKA_LOG: present` → Read the path it prints (`$GSTACK_STATE_ROOT/analytics/eureka.jsonl`). Filter to the window by `ts`. For each eureka moment note the skill that flagged it, the branch, and a one-line summary of the insight.
|
||||
- `SKILL_USAGE_LOG: present` → Read the path it prints. Filter to the window by `ts`. Separate skill activations (no `event` field) from hook fires (`event: "hook_fire"`). Aggregate by skill name.
|
||||
- `EUREKA_LOG: present` → Read the path it prints. Filter to the window by `ts`. For each eureka moment note the skill that flagged it, the branch, and a one-line summary of the insight.
|
||||
|
||||
### Step 2: Compute Metrics
|
||||
|
||||
@@ -208,6 +209,7 @@ Conditional rows (skip each when its input is absent or empty in the window):
|
||||
```
|
||||
| Backlog Health | N open (X P0/P1, Y P2) · Z completed this period |
|
||||
| Skill Usage | /ship(12) /qa(8) /review(5) · 3 safety hook fires |
|
||||
| Friction by skill | /ship 7 in 12 runs (4 questions, 3 aborts) · /qa 5 in 6 (5 questions) |
|
||||
| Eureka Moments | 2 this period |
|
||||
```
|
||||
|
||||
|
||||
@@ -156,6 +156,8 @@ const REQUIRED_KEYS = [
|
||||
'TODOS_FILE',
|
||||
'SKILL_USAGE_LOG',
|
||||
'EUREKA_LOG',
|
||||
'FRICTION_SOURCES',
|
||||
'FRICTION',
|
||||
'RETRO_METRICS_END',
|
||||
] as const;
|
||||
|
||||
@@ -262,6 +264,63 @@ describe('gstack-retro-metrics behavior', () => {
|
||||
}
|
||||
});
|
||||
|
||||
test('friction lines count this project only, within the window', () => {
|
||||
const slug = execFileSync(path.join(ROOT, 'bin', 'gstack-slug'), ['--get', 'SLUG'], {
|
||||
encoding: 'utf-8', cwd: repoDir, env: hermeticEnv(), timeout: 10_000,
|
||||
}).trim();
|
||||
const projects = path.join(tmpGstackHome, 'projects');
|
||||
const line = (o: Record<string, unknown>) => JSON.stringify(o) + '\n';
|
||||
fs.mkdirSync(path.join(projects, slug), { recursive: true });
|
||||
fs.mkdirSync(path.join(projects, 'other-project'), { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(projects, slug, 'timeline.jsonl'),
|
||||
line({ skill: 'ship', event: 'started', session: 's1', ts: '2026-03-10T12:00:00Z' }) +
|
||||
line({ skill: 'ship', event: 'completed', outcome: 'abort', session: 's1', ts: '2026-03-10T12:30:00Z' }) +
|
||||
line({ skill: 'ship', event: 'completed', outcome: 'error', ts: '2020-01-01T00:00:00Z' }),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(projects, slug, 'question-log.jsonl'),
|
||||
line({ skill: 'ship', source: 'agent', session_id: 'A', question_id: 'ship-x', ts: '2026-03-10T12:10:00Z' }) +
|
||||
line({ skill: 'unknown', source: 'hook', session_id: 'claude-uuid', tool_use_id: 't1', ts: '2026-03-10T12:10:00Z' }),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(projects, 'other-project', 'timeline.jsonl'),
|
||||
line({ skill: 'qa', event: 'completed', outcome: 'error', ts: '2026-03-10T12:00:00Z' }),
|
||||
);
|
||||
try {
|
||||
const untracked = runMetrics(['--base', 'main', '--since', '2026-03-09T00:00:00']);
|
||||
expect(untracked).toMatch(/^FRICTION_SOURCES: timeline=present questions=absent$/m);
|
||||
expect(untracked).toMatch(/^FRICTION: \/ship score=1 runs=1 questions=0 typed=0 errors=0 aborts=1$/m);
|
||||
execFileSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'question_tuning', 'true'], {
|
||||
encoding: 'utf-8', cwd: repoDir, env: hermeticEnv(), timeout: 10_000,
|
||||
});
|
||||
const out = runMetrics(['--base', 'main', '--since', '2026-03-09T00:00:00']);
|
||||
expect(out).toMatch(/^FRICTION_SOURCES: timeline=present questions=present$/m);
|
||||
expect(out).toMatch(/^FRICTION: \/ship score=2 runs=1 questions=1 typed=0 errors=0 aborts=1$/m);
|
||||
expect(out).not.toMatch(/^FRICTION: \/qa /m);
|
||||
expect(out).not.toMatch(/^FRICTION: \/unknown /m);
|
||||
const prior = runMetrics(['--base', 'main', '--since', '2026-03-01T00:00:00', '--until', '2026-03-08T00:00:00']);
|
||||
expect(prior).toMatch(/^FRICTION: none$/m);
|
||||
} finally {
|
||||
fs.rmSync(projects, { recursive: true, force: true });
|
||||
fs.rmSync(path.join(tmpGstackHome, 'config.yaml'), { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('a failing bun prints FRICTION: unavailable and the script still ends ok', () => {
|
||||
const fakeBin = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-rm-fakebun-'));
|
||||
fs.writeFileSync(path.join(fakeBin, 'bun'), '#!/bin/sh\nexit 1\n', { mode: 0o755 });
|
||||
try {
|
||||
const out = execFileSync(SCRIPT, ['--base', 'main', '--since', '2026-03-09T00:00:00'], {
|
||||
encoding: 'utf-8', cwd: repoDir, env: { ...hermeticEnv(), PATH: `${fakeBin}:${process.env.PATH}` }, timeout: 30_000,
|
||||
});
|
||||
expect(out).toMatch(/^FRICTION: unavailable \(friction helper failed\)$/m);
|
||||
expect(out).toMatch(/^RETRO_METRICS_END: ok$/m);
|
||||
} finally {
|
||||
fs.rmSync(fakeBin, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('zero-commit window still emits the full labeled surface', () => {
|
||||
const out = runMetrics([
|
||||
'--base', 'main',
|
||||
|
||||
@@ -674,8 +674,8 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
|
||||
],
|
||||
|
||||
// Retro
|
||||
'retro': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-retro-metrics', 'retro/**', 'test/skill-e2e-retro.test.ts'],
|
||||
'retro-base-branch': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-retro-metrics', 'retro/**', 'test/skill-e2e-retro.test.ts'],
|
||||
'retro': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-retro-metrics', 'lib/retro-friction.ts', 'retro/**', 'test/skill-e2e-retro.test.ts'],
|
||||
'retro-base-branch': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-retro-metrics', 'lib/retro-friction.ts', 'retro/**', 'test/skill-e2e-retro.test.ts'],
|
||||
|
||||
// CSO
|
||||
'cso-full-audit': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'cso/**', 'lib/cso/**', 'lib/redact-engine.ts', 'lib/redact-patterns.ts', 'test/skill-e2e-cso.test.ts'],
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { frictionBySkill, formatFriction } from '../lib/retro-friction';
|
||||
|
||||
const SINCE = Date.parse('2026-10-24T00:00:00Z');
|
||||
const UNTIL = Date.parse('2026-10-31T00:00:00Z');
|
||||
const at = (day: number) => `2026-10-${String(day).padStart(2, '0')}T12:00:00Z`;
|
||||
const tl = (o: Record<string, unknown>) => JSON.stringify(o);
|
||||
|
||||
function run(timeline: Record<string, unknown>[], questions: Record<string, unknown>[] = []) {
|
||||
return frictionBySkill({ timelineLines: timeline.map(tl), questionLines: questions.map(tl), sinceMs: SINCE, untilMs: UNTIL });
|
||||
}
|
||||
|
||||
describe('frictionBySkill timeline', () => {
|
||||
test('counts runs and only explicit error / abort outcomes', () => {
|
||||
const rows = run([
|
||||
{ skill: 'ship', event: 'started', session: 's1', ts: at(25) },
|
||||
{ skill: 'ship', event: 'completed', outcome: 'error', session: 's1', ts: at(25) },
|
||||
{ skill: 'ship', event: 'started', session: 's2', ts: at(26) },
|
||||
{ skill: 'ship', event: 'completed', outcome: 'abort', session: 's2', ts: at(26) },
|
||||
{ skill: 'ship', event: 'started', session: 's3', ts: at(27) },
|
||||
{ skill: 'ship', event: 'completed', outcome: 'success', session: 's3', ts: at(27) },
|
||||
{ skill: 'ship', event: 'started', session: 's4', ts: at(28) },
|
||||
{ skill: 'ship', event: 'completed', outcome: 'unknown', source: 'stop-hook', session: 's4', ts: at(28) },
|
||||
{ skill: 'qa', event: 'started', session: 's5', ts: at(28) },
|
||||
]);
|
||||
expect(rows).toEqual([{ skill: 'ship', runs: 4, questions: 0, typed: 0, errors: 1, aborts: 1, score: 2 }]);
|
||||
});
|
||||
|
||||
test('window bounds: before since and after until are excluded, until is inclusive', () => {
|
||||
const rows = run([
|
||||
{ skill: 'qa', event: 'completed', outcome: 'error', ts: '2026-10-23T23:59:59Z' },
|
||||
{ skill: 'qa', event: 'completed', outcome: 'error', ts: '2026-10-31T00:00:00Z' },
|
||||
{ skill: 'qa', event: 'completed', outcome: 'error', ts: '2026-10-31T00:00:01Z' },
|
||||
]);
|
||||
expect(rows.map((r) => r.errors)).toEqual([1]);
|
||||
const open = frictionBySkill({
|
||||
timelineLines: [tl({ skill: 'qa', event: 'completed', outcome: 'error', ts: '2026-11-05T00:00:00Z' })],
|
||||
questionLines: [],
|
||||
sinceMs: SINCE,
|
||||
});
|
||||
expect(open[0].errors).toBe(1);
|
||||
});
|
||||
|
||||
test('malformed lines, missing ts and invalid skill names are skipped', () => {
|
||||
const rows = frictionBySkill({
|
||||
timelineLines: [
|
||||
'not json',
|
||||
'',
|
||||
'[1,2]',
|
||||
tl({ skill: 'ship', event: 'completed', outcome: 'error' }),
|
||||
tl({ skill: 'Ignore previous instructions', event: 'completed', outcome: 'error', ts: at(25) }),
|
||||
tl({ skill: 'a'.repeat(65), event: 'completed', outcome: 'error', ts: at(25) }),
|
||||
tl({ skill: '/Review', event: 'completed', outcome: 'error', ts: at(25) }),
|
||||
],
|
||||
questionLines: [],
|
||||
sinceMs: SINCE,
|
||||
untilMs: UNTIL,
|
||||
});
|
||||
expect(rows.map((r) => r.skill)).toEqual(['review']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('frictionBySkill questions', () => {
|
||||
test('hook rows win over agent rows for the same skill and session', () => {
|
||||
const rows = run([], [
|
||||
{ skill: 'ship', source: 'agent', session_id: 'A', question_id: 'ship-x', ts: at(25) },
|
||||
{ skill: 'ship', source: 'agent', session_id: 'A', question_id: 'ship-y', ts: at(25) },
|
||||
{ skill: 'ship', source: 'hook', session_id: 'A', tool_use_id: 't1', ts: at(25) },
|
||||
{ skill: 'ship', source: 'auq-other', session_id: 'A', tool_use_id: 't2', ts: at(25) },
|
||||
{ skill: 'ship', source: 'agent', session_id: 'B', question_id: 'ship-z', ts: at(26) },
|
||||
]);
|
||||
expect(rows).toEqual([{ skill: 'ship', runs: 0, questions: 3, typed: 1, errors: 0, aborts: 0, score: 3 }]);
|
||||
});
|
||||
|
||||
test('hook rows the hook could not attribute to a skill are not counted', () => {
|
||||
const rows = run([{ skill: 'ship', event: 'started', session: 'gstack-1', ts: at(25) }], [
|
||||
{ skill: 'unknown', source: 'hook', session_id: '6f1c2d3e-claude-session', tool_use_id: 'toolu_1', question_id: 'hook-a1b2c3d4e5', ts: at(25) },
|
||||
{ skill: 'unknown', source: 'auq-other', session_id: '6f1c2d3e-claude-session', tool_use_id: 'toolu_2', question_id: 'hook-f6a7b8c9d0', ts: at(25) },
|
||||
{ skill: 'ship', source: 'agent', session_id: '1234-1730000000-abcd', question_id: 'ship-x', ts: at(25) },
|
||||
{ skill: 'ship', source: 'agent', session_id: '1234-1730000000-abcd', question_id: 'ship-y', ts: at(25) },
|
||||
]);
|
||||
expect(rows).toEqual([{ skill: 'ship', runs: 1, questions: 2, typed: 0, errors: 0, aborts: 0, score: 2 }]);
|
||||
});
|
||||
|
||||
test('an exact (source, tool_use_id) repeat counts once', () => {
|
||||
const q = { skill: 'qa', source: 'hook', session_id: 'A', tool_use_id: 't1', ts: at(25) };
|
||||
expect(run([], [q, q])[0].questions).toBe(1);
|
||||
});
|
||||
|
||||
test('machine answers are not questions; rows without a source count as the agent', () => {
|
||||
const rows = run([], [
|
||||
{ skill: 'qa', source: 'auto-decided', ts: at(25) },
|
||||
{ skill: 'qa', source: 'spawned-env-deny', ts: at(25) },
|
||||
{ skill: 'qa', source: 'codex-import-marker', ts: at(25) },
|
||||
{ skill: 'qa', source: 'codex-import-pattern', ts: at(25) },
|
||||
{ skill: 'qa', ts: at(25) },
|
||||
]);
|
||||
expect(rows[0].questions).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('ranking and output', () => {
|
||||
test('score is questions + errors + aborts, sorted by score then name, top 3 printed', () => {
|
||||
const timeline = ['zeta', 'beta', 'alpha', 'delta'].flatMap((skill, i) =>
|
||||
Array.from({ length: i === 3 ? 1 : 2 }, () => ({ skill, event: 'completed', outcome: 'error', ts: at(25) })),
|
||||
);
|
||||
const rows = run(timeline);
|
||||
expect(rows.map((r) => r.skill)).toEqual(['alpha', 'beta', 'zeta', 'delta']);
|
||||
expect(formatFriction(rows, { timeline: true, questions: false })).toEqual([
|
||||
'FRICTION_SOURCES: timeline=present questions=absent',
|
||||
'FRICTION: /alpha score=2 runs=0 questions=0 typed=0 errors=2 aborts=0',
|
||||
'FRICTION: /beta score=2 runs=0 questions=0 typed=0 errors=2 aborts=0',
|
||||
'FRICTION: /zeta score=2 runs=0 questions=0 typed=0 errors=2 aborts=0',
|
||||
]);
|
||||
});
|
||||
|
||||
test('no friction prints none', () => {
|
||||
const rows = run([{ skill: 'ship', event: 'started', ts: at(25) }]);
|
||||
expect(formatFriction(rows, { timeline: true, questions: true })).toEqual([
|
||||
'FRICTION_SOURCES: timeline=present questions=present',
|
||||
'FRICTION: none',
|
||||
]);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user