mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-22 16:11:34 +01:00
Telemetry records why skill runs end, which host and model ran them, and how upgrades went (GST-3234) (#4542)
gstack's anonymous usage telemetry now records nine more fields: the host and model family, how a skill was started, why it ended, whether checks passed, how skill routing is set up, a random per-session id, and for upgrades the version left and any error. This tells us where skills stop working and which setups break upgrades, which the current outcome-only data can't show. Deploy in order after merge: apply migration 009, deploy the telemetry-ingest function, then release the client; apply migration 007 after that. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> Review-Finding: p1/specialist-batched#2 reader=specialist-batched Review-Finding: p1/specialist-batched#3 reader=specialist-batched Review-Finding: p1/adversarial-native#1 reader=adversarial-native Review-Finding: p1/outside-voice#2 reader=outside-voice Review-Finding: v1/outside-voice#1 reader=outside-voice Review-Finding: v1/outside-voice#2 reader=outside-voice
This commit is contained in:
1 parent
db2a56e679
commit
9ea26ecc42
74 files changed
+1180
-695
No files matched your search
+11
-11
@@ -110,26 +110,26 @@ jq -nc --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg skill "SKILL_NAME" --arg
|
||||
|
||||
## Telemetry (run last)
|
||||
|
||||
After workflow completion, log telemetry with ONE command. OUTCOME is
|
||||
success/error/abort/unknown; `SESSION_ID` and `TEL_START` are the values the
|
||||
preamble's skill-start output echoed. It also drains the artifacts-sync queue
|
||||
(the former skill-end sync step — do not run gstack-brain-sync separately).
|
||||
When done, make ONE call — it also drains the artifacts-sync
|
||||
queue (don't also run gstack-brain-sync).
|
||||
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** This writes telemetry to
|
||||
`$GSTACK_STATE_ROOT/analytics/`, matching preamble analytics writes.
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** writes telemetry to `$GSTACK_STATE_ROOT/analytics/`.
|
||||
|
||||
```bash
|
||||
[ -d "${GSTACK_ROOT:-/-}/bin" ]&&[ -d "$GSTACK_ROOT/lib" ]||{ _r=$(git rev-parse --show-toplevel 2>/dev/null)/.agents/skills/gstack;[ -d "$_r/bin" ]||_r=~/.gemini/antigravity-cli/skills/gstack;[ -d "$_r/bin" ]||{ echo "gstack: no install found (tried $_r). Fix: ./setup --host agy from your gstack checkout; ./setup --status shows it.">&2;exit 1;};GSTACK_ROOT=$_r;}
|
||||
GSTACK_BIN=$GSTACK_ROOT/bin
|
||||
$GSTACK_BIN/gstack-skill-end --skill "ship" --outcome OUTCOME \
|
||||
--session-id "SESSION_ID" --tel-start "TEL_START" --used-browse USED_BROWSE \
|
||||
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" 2>/dev/null || true
|
||||
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" \
|
||||
--exit-reason EXIT_REASON --invoked INVOKED --model-family MODEL_FAMILY --checks CHECKS 2>/dev/null || true
|
||||
```
|
||||
|
||||
Replace `OUTCOME` and `USED_BROWSE` (yes/no) before running; substitute
|
||||
`SESSION_ID`/`TEL_START` from the skill-start echoes. `ERROR_MESSAGE`/`FAILED_STEP`
|
||||
are "" unless outcome is error. If the command is missing (stale install), skip
|
||||
telemetry — it never blocks the workflow.
|
||||
OUTCOME: success/error/abort/unknown. USED_BROWSE: yes/no. SESSION_ID, TEL_START:
|
||||
skill-start echoes. ERROR_MESSAGE, FAILED_STEP: "" unless error. EXIT_REASON:
|
||||
completed/user_stopped/blocked/tool_error/test_failed/context_limit/gave_up.
|
||||
INVOKED: user/model/chained. MODEL_FAMILY (yours): claude_opus/claude_sonnet/
|
||||
claude_haiku/claude_other/gpt/gemini/other. CHECKS: passed/failed/not_run. No command
|
||||
(stale install) — skip.
|
||||
|
||||
## Third-Party Web Actions
|
||||
|
||||
|
||||
+11
-11
@@ -117,24 +117,24 @@ jq -nc --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg skill "SKILL_NAME" --arg
|
||||
|
||||
## Telemetry (run last)
|
||||
|
||||
After workflow completion, log telemetry with ONE command. OUTCOME is
|
||||
success/error/abort/unknown; `SESSION_ID` and `TEL_START` are the values the
|
||||
preamble's skill-start output echoed. It also drains the artifacts-sync queue
|
||||
(the former skill-end sync step — do not run gstack-brain-sync separately).
|
||||
When done, make ONE call — it also drains the artifacts-sync
|
||||
queue (don't also run gstack-brain-sync).
|
||||
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** This writes telemetry to
|
||||
`$GSTACK_STATE_ROOT/analytics/`, matching preamble analytics writes.
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** writes telemetry to `$GSTACK_STATE_ROOT/analytics/`.
|
||||
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-skill-end --skill "ship" --outcome OUTCOME \
|
||||
--session-id "SESSION_ID" --tel-start "TEL_START" --used-browse USED_BROWSE \
|
||||
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" 2>/dev/null || true
|
||||
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" \
|
||||
--exit-reason EXIT_REASON --invoked INVOKED --model-family MODEL_FAMILY --checks CHECKS 2>/dev/null || true
|
||||
```
|
||||
|
||||
Replace `OUTCOME` and `USED_BROWSE` (yes/no) before running; substitute
|
||||
`SESSION_ID`/`TEL_START` from the skill-start echoes. `ERROR_MESSAGE`/`FAILED_STEP`
|
||||
are "" unless outcome is error. If the command is missing (stale install), skip
|
||||
telemetry — it never blocks the workflow.
|
||||
OUTCOME: success/error/abort/unknown. USED_BROWSE: yes/no. SESSION_ID, TEL_START:
|
||||
skill-start echoes. ERROR_MESSAGE, FAILED_STEP: "" unless error. EXIT_REASON:
|
||||
completed/user_stopped/blocked/tool_error/test_failed/context_limit/gave_up.
|
||||
INVOKED: user/model/chained. MODEL_FAMILY (yours): claude_opus/claude_sonnet/
|
||||
claude_haiku/claude_other/gpt/gemini/other. CHECKS: passed/failed/not_run. No command
|
||||
(stale install) — skip.
|
||||
|
||||
## Third-Party Web Actions
|
||||
|
||||
|
||||
+11
-11
@@ -108,26 +108,26 @@ jq -nc --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg skill "SKILL_NAME" --arg
|
||||
|
||||
## Telemetry (run last)
|
||||
|
||||
After workflow completion, log telemetry with ONE command. OUTCOME is
|
||||
success/error/abort/unknown; `SESSION_ID` and `TEL_START` are the values the
|
||||
preamble's skill-start output echoed. It also drains the artifacts-sync queue
|
||||
(the former skill-end sync step — do not run gstack-brain-sync separately).
|
||||
When done, make ONE call — it also drains the artifacts-sync
|
||||
queue (don't also run gstack-brain-sync).
|
||||
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** This writes telemetry to
|
||||
`$GSTACK_STATE_ROOT/analytics/`, matching preamble analytics writes.
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** writes telemetry to `$GSTACK_STATE_ROOT/analytics/`.
|
||||
|
||||
```bash
|
||||
[ -d "${GSTACK_ROOT:-/-}/bin" ]&&[ -d "$GSTACK_ROOT/lib" ]||{ _r=$(git rev-parse --show-toplevel 2>/dev/null)/.agents/skills/gstack;[ -d "$_r/bin" ]||_r=${CODEX_HOME:-~/.codex}/skills/gstack;[ -d "$_r/bin" ]||{ echo "gstack: no install found (tried $_r). Fix: ./setup --host codex from your gstack checkout; ./setup --status shows it.">&2;exit 1;};GSTACK_ROOT=$_r;}
|
||||
GSTACK_BIN=$GSTACK_ROOT/bin
|
||||
$GSTACK_BIN/gstack-skill-end --skill "ship" --outcome OUTCOME \
|
||||
--session-id "SESSION_ID" --tel-start "TEL_START" --used-browse USED_BROWSE \
|
||||
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" 2>/dev/null || true
|
||||
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" \
|
||||
--exit-reason EXIT_REASON --invoked INVOKED --model-family MODEL_FAMILY --checks CHECKS 2>/dev/null || true
|
||||
```
|
||||
|
||||
Replace `OUTCOME` and `USED_BROWSE` (yes/no) before running; substitute
|
||||
`SESSION_ID`/`TEL_START` from the skill-start echoes. `ERROR_MESSAGE`/`FAILED_STEP`
|
||||
are "" unless outcome is error. If the command is missing (stale install), skip
|
||||
telemetry — it never blocks the workflow.
|
||||
OUTCOME: success/error/abort/unknown. USED_BROWSE: yes/no. SESSION_ID, TEL_START:
|
||||
skill-start echoes. ERROR_MESSAGE, FAILED_STEP: "" unless error. EXIT_REASON:
|
||||
completed/user_stopped/blocked/tool_error/test_failed/context_limit/gave_up.
|
||||
INVOKED: user/model/chained. MODEL_FAMILY (yours): claude_opus/claude_sonnet/
|
||||
claude_haiku/claude_other/gpt/gemini/other. CHECKS: passed/failed/not_run. No command
|
||||
(stale install) — skip.
|
||||
|
||||
## Third-Party Web Actions
|
||||
|
||||
|
||||
+11
-11
@@ -110,26 +110,26 @@ jq -nc --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg skill "SKILL_NAME" --arg
|
||||
|
||||
## Telemetry (run last)
|
||||
|
||||
After workflow completion, log telemetry with ONE command. OUTCOME is
|
||||
success/error/abort/unknown; `SESSION_ID` and `TEL_START` are the values the
|
||||
preamble's skill-start output echoed. It also drains the artifacts-sync queue
|
||||
(the former skill-end sync step — do not run gstack-brain-sync separately).
|
||||
When done, make ONE call — it also drains the artifacts-sync
|
||||
queue (don't also run gstack-brain-sync).
|
||||
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** This writes telemetry to
|
||||
`$GSTACK_STATE_ROOT/analytics/`, matching preamble analytics writes.
|
||||
**PLAN MODE EXCEPTION — ALWAYS RUN:** writes telemetry to `$GSTACK_STATE_ROOT/analytics/`.
|
||||
|
||||
```bash
|
||||
[ -d "${GSTACK_ROOT:-/-}/bin" ]&&[ -d "$GSTACK_ROOT/lib" ]||{ _r=$(git rev-parse --show-toplevel 2>/dev/null)/.factory/skills/gstack;[ -d "$_r/bin" ]||_r=~/.factory/skills/gstack;[ -d "$_r/bin" ]||{ echo "gstack: no install found (tried $_r). Fix: ./setup --host factory from your gstack checkout; ./setup --status shows it.">&2;exit 1;};GSTACK_ROOT=$_r;}
|
||||
GSTACK_BIN=$GSTACK_ROOT/bin
|
||||
$GSTACK_BIN/gstack-skill-end --skill "ship" --outcome OUTCOME \
|
||||
--session-id "SESSION_ID" --tel-start "TEL_START" --used-browse USED_BROWSE \
|
||||
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" 2>/dev/null || true
|
||||
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" \
|
||||
--exit-reason EXIT_REASON --invoked INVOKED --model-family MODEL_FAMILY --checks CHECKS 2>/dev/null || true
|
||||
```
|
||||
|
||||
Replace `OUTCOME` and `USED_BROWSE` (yes/no) before running; substitute
|
||||
`SESSION_ID`/`TEL_START` from the skill-start echoes. `ERROR_MESSAGE`/`FAILED_STEP`
|
||||
are "" unless outcome is error. If the command is missing (stale install), skip
|
||||
telemetry — it never blocks the workflow.
|
||||
OUTCOME: success/error/abort/unknown. USED_BROWSE: yes/no. SESSION_ID, TEL_START:
|
||||
skill-start echoes. ERROR_MESSAGE, FAILED_STEP: "" unless error. EXIT_REASON:
|
||||
completed/user_stopped/blocked/tool_error/test_failed/context_limit/gave_up.
|
||||
INVOKED: user/model/chained. MODEL_FAMILY (yours): claude_opus/claude_sonnet/
|
||||
claude_haiku/claude_other/gpt/gemini/other. CHECKS: passed/failed/not_run. No command
|
||||
(stale install) — skip.
|
||||
|
||||
## Third-Party Web Actions
|
||||
|
||||
|
||||
@@ -130,6 +130,27 @@ describe('gstack-upgrade', () => {
|
||||
expect(f.run().stdout).toContain('gstack-upgrade: current 1.0.1');
|
||||
});
|
||||
|
||||
test('telemetry: one upgrade_completed per run with its upgrade_error and from-version; --check sends nothing', () => {
|
||||
const f = fixture();
|
||||
const telLog = path.join(f.host.base, 'calls.log.tel');
|
||||
write(path.join(f.root, 'bin/gstack-telemetry-log'), '#!/usr/bin/env bash\necho "$*" >> "$UPGRADE_LOG.tel"\n', 0o755);
|
||||
const tel = () => fs.existsSync(telLog) ? fs.readFileSync(telLog, 'utf8').trim().split('\n').filter(Boolean) : [];
|
||||
expect(f.run(['--check']).status).toBe(1);
|
||||
expect(tel()).toEqual([]);
|
||||
fs.writeFileSync(path.join(f.root, 'README.md'), 'local edit\n');
|
||||
expect(f.run().status).toBe(3);
|
||||
f.git(['checkout', '--', 'README.md']);
|
||||
expect(f.run([], { STUB_SETUP_EXIT: '1' }).status).toBe(1);
|
||||
const retry = f.run();
|
||||
expect(retry.status, retry.stdout + retry.stderr).toBe(0);
|
||||
const ev = '--no-sweep --event-type upgrade_completed';
|
||||
expect(tel()).toEqual([
|
||||
`${ev} --outcome error --from-version 1.0.0 --upgrade-error refused_dirty`,
|
||||
`${ev} --outcome error --from-version 1.0.0 --upgrade-error setup_failed`,
|
||||
`${ev} --outcome success --from-version 1.0.0 --upgrade-error none`,
|
||||
]);
|
||||
});
|
||||
|
||||
test('a Capy install (the installer marker) runs bin/gstack-capy-install instead of setup', () => {
|
||||
const f = fixture();
|
||||
write(path.join(f.root, 'bin/gstack-capy-install'), '#!/usr/bin/env bash\necho "capy-install $*" >> "$UPGRADE_LOG"\necho "gstack: ok $(cat VERSION) for=autoplan"\n', 0o755);
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
/** The next telemetry fields (docs/privacy.md, migration 009) as the client
|
||||
* writes them: fixed lists, a random per-agent-session id, and a source label
|
||||
* an explicit setting always wins. Network is stubbed; nothing leaves the box. */
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { spawn, spawnSync } from 'child_process';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const BIN = path.join(ROOT, 'bin');
|
||||
let tmp: string;
|
||||
|
||||
function env(extra: Record<string, string> = {}): Record<string, string> {
|
||||
return {
|
||||
...process.env as Record<string, string>,
|
||||
HOME: path.join(tmp, 'home'), GSTACK_HOME: tmp, GSTACK_STATE_ROOT: tmp, GSTACK_STATE_DIR: tmp,
|
||||
GSTACK_DIR: ROOT, GSTACK_SUPABASE_URL: 'https://telemetry.fixture.invalid', GSTACK_SUPABASE_ANON_KEY: 'fixture',
|
||||
GSTACK_TEST_BLOCK_LIVE_TELEMETRY: '1', GSTACK_SESSION_KIND: 'interactive',
|
||||
CI: '', GSTACK_TELEMETRY_SOURCE: '', EVALS_RUN_ID: '', GSTACK_EVAL_DIR: '',
|
||||
GSTACK_ACTIVE_HOST: '', CLAUDECODE: '', CODEX_THREAD_ID: '',
|
||||
PATH: `${path.join(tmp, 'bin')}:${process.env.PATH}`,
|
||||
...extra,
|
||||
};
|
||||
}
|
||||
function run(bin: string, args: string[], extra: Record<string, string> = {}) {
|
||||
const r = spawnSync(path.join(BIN, bin), args, { cwd: tmp, env: env(extra), encoding: 'utf-8', timeout: 20_000 });
|
||||
expect(r.status).toBe(0);
|
||||
return r.stdout;
|
||||
}
|
||||
function events(): any[] {
|
||||
const f = path.join(tmp, 'analytics', 'skill-usage.jsonl');
|
||||
const lines = fs.existsSync(f) ? fs.readFileSync(f, 'utf-8').trim().split('\n').filter(Boolean).map(l => JSON.parse(l)) : [];
|
||||
return lines.filter(e => e.v === 1);
|
||||
}
|
||||
async function eventually(n: number): Promise<any[]> {
|
||||
for (let i = 0; i < 50 && events().length < n; i++) await Bun.sleep(100);
|
||||
return events();
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-nexttel-'));
|
||||
fs.mkdirSync(path.join(tmp, 'home'));
|
||||
fs.mkdirSync(path.join(tmp, 'bin'));
|
||||
fs.writeFileSync(path.join(tmp, 'bin', 'curl'), '#!/bin/sh\nprintf 200\n', { mode: 0o755 });
|
||||
run('gstack-config', ['set', 'telemetry', 'anonymous']);
|
||||
});
|
||||
afterEach(() => { fs.rmSync(tmp, { recursive: true, force: true }); });
|
||||
|
||||
describe.skipIf(process.platform === 'win32')('gstack-telemetry-log next fields', () => {
|
||||
test('fixed-list fields pass through, off-list values become unknown, absent ones null', () => {
|
||||
run('gstack-telemetry-log', ['--skill', 'qa', '--outcome', 'success', '--host', 'codex', '--model-family', 'gpt',
|
||||
'--invoked', 'model', '--exit-reason', 'rm -rf /', '--checks', 'passed', '--routing', 'block']);
|
||||
const [e] = events();
|
||||
expect([e.host, e.model_family, e.invoked, e.exit_reason, e.checks, e.routing]).toEqual(['codex', 'gpt', 'model', 'unknown', 'passed', 'block']);
|
||||
expect([e.harness_session_id, e.from_version, e.upgrade_error]).toEqual([null, null, null]);
|
||||
});
|
||||
|
||||
test('the session id must be 16 hex and from_version a version, or they are dropped', () => {
|
||||
run('gstack-telemetry-log', ['--event-type', 'upgrade_completed', '--outcome', 'error', '--harness-session', '0123456789abcdef',
|
||||
'--from-version', '1.2.3.4', '--upgrade-error', 'busy']);
|
||||
run('gstack-telemetry-log', ['--outcome', 'success', '--harness-session', 'not-a-session-id', '--from-version', '1.2; DROP']);
|
||||
const [a, b] = events();
|
||||
expect([a.harness_session_id, a.from_version, a.upgrade_error]).toEqual(['0123456789abcdef', '1.2.3.4', 'busy']);
|
||||
expect([b.harness_session_id, b.from_version]).toEqual([null, null]);
|
||||
});
|
||||
|
||||
test('source: flag > GSTACK_TELEMETRY_SOURCE > eval run > CI > live, so CI never relabels a test row', () => {
|
||||
run('gstack-telemetry-log', ['--outcome', 'success', '--source', 'test'], { CI: 'true', EVALS_RUN_ID: 'r1' });
|
||||
run('gstack-telemetry-log', ['--outcome', 'success'], { CI: 'true', GSTACK_TELEMETRY_SOURCE: 'test' });
|
||||
run('gstack-telemetry-log', ['--outcome', 'success'], { CI: 'true', EVALS_RUN_ID: 'r1' });
|
||||
run('gstack-telemetry-log', ['--outcome', 'success'], { CI: 'true' });
|
||||
run('gstack-telemetry-log', ['--outcome', 'success']);
|
||||
expect(events().map(e => e.source)).toEqual(['test', 'test', 'eval', 'ci', 'live']);
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(process.platform === 'win32')('gstack-skill-end next fields', () => {
|
||||
test('one random id per harness process, kept in its sessions marker without adding a session', async () => {
|
||||
fs.mkdirSync(path.join(tmp, 'sessions'));
|
||||
const marker = path.join(tmp, 'sessions', '4242');
|
||||
fs.writeFileSync(marker, '');
|
||||
const args = ['--skill', 'qa', '--outcome', 'success', '--session-id', '4242-1700000000-1', '--tel-start', '1700000000',
|
||||
'--exit-reason', 'completed', '--invoked', 'user', '--model-family', 'claude_opus', '--checks', 'not_run'];
|
||||
run('gstack-skill-end', args, { CLAUDECODE: '1' });
|
||||
run('gstack-skill-end', args, { CLAUDECODE: '1' });
|
||||
const [a, b] = await eventually(2);
|
||||
expect(a.harness_session_id).toMatch(/^[0-9a-f]{16}$/);
|
||||
expect(b.harness_session_id).toBe(a.harness_session_id);
|
||||
expect(fs.readFileSync(marker, 'utf-8')).toBe(a.harness_session_id);
|
||||
expect(fs.readdirSync(path.join(tmp, 'sessions'))).toEqual(['4242']);
|
||||
expect([a.host, a.exit_reason, a.invoked, a.model_family, a.checks, a.routing]).toEqual(['claude_code', 'completed', 'user', 'claude_opus', 'not_run', 'none']);
|
||||
});
|
||||
|
||||
test('under a shared harness (capy-server) every session has one pid, so no session id is sent', async () => {
|
||||
fs.copyFileSync('/bin/sleep', path.join(tmp, 'capy-server'));
|
||||
fs.chmodSync(path.join(tmp, 'capy-server'), 0o755);
|
||||
const harness = spawn(path.join(tmp, 'capy-server'), ['30'], { stdio: 'ignore' });
|
||||
try {
|
||||
fs.mkdirSync(path.join(tmp, 'sessions'));
|
||||
fs.writeFileSync(path.join(tmp, 'sessions', String(harness.pid)), '');
|
||||
run('gstack-skill-end', ['--skill', 'qa', '--outcome', 'success', '--session-id', `${harness.pid}-1700000000-1`, '--tel-start', '1700000000']);
|
||||
const [e] = await eventually(1);
|
||||
expect(e.harness_session_id).toBeNull();
|
||||
expect(fs.readFileSync(path.join(tmp, 'sessions', String(harness.pid)), 'utf-8')).toBe('');
|
||||
} finally { harness.kill('SIGKILL'); }
|
||||
});
|
||||
|
||||
test('a recycled pid with a stale marker gets a new session id, never the old one', async () => {
|
||||
fs.appendFileSync(path.join(tmp, 'config.yaml'), 'update_check: false\n');
|
||||
fs.mkdirSync(path.join(tmp, 'sessions'));
|
||||
const marker = path.join(tmp, 'sessions', '4343');
|
||||
fs.writeFileSync(marker, 'aaaaaaaaaaaaaaaa');
|
||||
const threeHoursAgo = new Date(Date.now() - 3 * 3600_000);
|
||||
fs.utimesSync(marker, threeHoursAgo, threeHoursAgo);
|
||||
run('gstack-skill-start', ['--skill', 'qa', '--parent-pid', '4343']);
|
||||
expect(fs.readFileSync(marker, 'utf-8')).toBe('');
|
||||
run('gstack-skill-end', ['--skill', 'qa', '--outcome', 'success', '--session-id', '4343-1700000000-1', '--tel-start', '1700000000'], { CLAUDECODE: '1' });
|
||||
const [e] = await eventually(1);
|
||||
expect(e.harness_session_id).toMatch(/^[0-9a-f]{16}$/);
|
||||
expect(e.harness_session_id).not.toBe('aaaaaaaaaaaaaaaa');
|
||||
}, 30_000);
|
||||
|
||||
test('an unknown start time sends a null duration, not 0', async () => {
|
||||
const out = run('gstack-skill-end', ['--skill', 'qa', '--outcome', 'success', '--session-id', '99-1']);
|
||||
expect(out).toContain('duration_s=unknown');
|
||||
const [e] = await eventually(1);
|
||||
expect(e.duration_s).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(process.platform === 'win32')('gstack-telemetry-sync batch size', () => {
|
||||
test('a backlog of full-size events is sent in bodies under ingest\'s 50,000-byte limit', () => {
|
||||
fs.writeFileSync(path.join(tmp, 'bin', 'curl'),
|
||||
'#!/bin/sh\nfor a in "$@"; do case "$a" in @*) wc -c < "${a#@}" >> "$SIZES" ;; esac; done\nprintf 500\n', { mode: 0o755 });
|
||||
const event = JSON.stringify({ v: 1, ts: '2026-11-19T21:00:00Z', event_type: 'skill_run', skill: 'plan-design-review', session_id: '12345-1763600000-abc',
|
||||
gstack_version: '1.102.0.0', os: 'darwin', arch: 'arm64', duration_s: 1234, outcome: 'success', error_class: null, error_message: null, failed_step: null,
|
||||
used_browse: false, sessions: 1, installation_id: null, source: 'live', security_url_domain: null, security_payload_hash: null, security_confidence: null,
|
||||
security_layer: null, security_verdict: null, host: 'claude_code', model_family: 'claude_opus', invoked: 'user', exit_reason: 'completed', checks: 'passed',
|
||||
routing: 'both', harness_session_id: '0123456789abcdef', from_version: null, upgrade_error: null, _repo_slug: 'owner/repo', _branch: 'main' });
|
||||
fs.mkdirSync(path.join(tmp, 'analytics'), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmp, 'analytics', 'skill-usage.jsonl'), `${event}\n`.repeat(120));
|
||||
const sizes = path.join(tmp, 'sizes');
|
||||
run('gstack-telemetry-sync', [], { SIZES: sizes });
|
||||
const sent = fs.readFileSync(sizes, 'utf-8').trim().split('\n').map(Number);
|
||||
expect(sent).toHaveLength(1);
|
||||
expect(sent[0]).toBeGreaterThan(40_000);
|
||||
expect(sent[0]).toBeLessThanOrEqual(45_000);
|
||||
});
|
||||
|
||||
test('multi-byte events are measured in bytes, so the body stays under the limit', () => {
|
||||
fs.writeFileSync(path.join(tmp, 'bin', 'curl'),
|
||||
'#!/bin/sh\nfor a in "$@"; do case "$a" in @*) wc -c < "${a#@}" >> "$SIZES" ;; esac; done\nprintf 500\n', { mode: 0o755 });
|
||||
const event = JSON.stringify({ v: 1, ts: '2026-11-19T21:00:00Z', event_type: 'skill_run', skill: 'qa', session_id: '12345-1763600000-abc',
|
||||
outcome: 'error', error_message: '\u20ac'.repeat(300), source: 'live', host: 'claude_code' });
|
||||
fs.mkdirSync(path.join(tmp, 'analytics'), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmp, 'analytics', 'skill-usage.jsonl'), `${event}\n`.repeat(100));
|
||||
const sizes = path.join(tmp, 'sizes');
|
||||
run('gstack-telemetry-sync', [], { SIZES: sizes, LC_ALL: process.platform === 'darwin' ? 'en_US.UTF-8' : 'C.UTF-8' });
|
||||
const sent = fs.readFileSync(sizes, 'utf-8').trim().split('\n').map(Number);
|
||||
expect(sent).toHaveLength(1);
|
||||
expect(sent[0]).toBeGreaterThan(40_000);
|
||||
expect(sent[0]).toBeLessThanOrEqual(45_000);
|
||||
});
|
||||
});
|
||||
@@ -94,3 +94,55 @@ describe('migrations keep the public key and signed-in users out', () => {
|
||||
expect([...created].filter(p => /^anon_(insert|update)/.test(p))).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the next telemetry fields stay in step across client, ingest and schema', () => {
|
||||
const LOG = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-telemetry-log'), 'utf-8');
|
||||
const sql = fs.readFileSync(path.join(MIGRATIONS, '009_next_telemetry_fields.sql'), 'utf-8');
|
||||
const ingestEnums = (field: string) => {
|
||||
const m = INGEST.match(new RegExp(`${field}: \\[([^\\]]*)\\]`));
|
||||
return [...(m?.[1] ?? '').matchAll(/"([a-z_]+)"/g)].map(x => x[1]).sort();
|
||||
};
|
||||
const fields: Array<[string, string]> = [['host', 'HOST_FIELD'], ['model_family', 'MODEL_FIELD'], ['invoked', 'INVOKED_FIELD'],
|
||||
['exit_reason', 'EXIT_FIELD'], ['checks', 'CHECKS_FIELD'], ['routing', 'ROUTING_FIELD'], ['upgrade_error', 'UPGRADE_ERR_FIELD']];
|
||||
|
||||
test('each fixed list is the same in the client, the ingest and the CHECK (plus unknown)', () => {
|
||||
for (const [col, v] of fields) {
|
||||
const client = (LOG.match(new RegExp(`${v}="\\$\\(enum_field "\\$[A-Z_]+" ([^)]*)\\)"`))?.[1] ?? '').split(' ').filter(Boolean);
|
||||
const check = [...(sql.match(new RegExp(`ADD COLUMN IF NOT EXISTS ${col} TEXT CHECK \\(${col} IN \\(([^)]*)\\)`))?.[1] ?? '').matchAll(/'([a-z_]+)'/g)].map(x => x[1]);
|
||||
expect(client.length, col).toBeGreaterThan(2);
|
||||
expect(ingestEnums(col), col).toEqual([...client, 'unknown'].sort());
|
||||
expect(check.sort(), col).toEqual([...client, 'unknown'].sort());
|
||||
}
|
||||
});
|
||||
|
||||
test('every column the ingest writes is added by a migration', () => {
|
||||
const cols = new Set<string>();
|
||||
for (const f of fs.readdirSync(MIGRATIONS).filter(x => x.endsWith('.sql'))) {
|
||||
const text = fs.readFileSync(path.join(MIGRATIONS, f), 'utf-8');
|
||||
const table = text.match(/CREATE TABLE telemetry_events \(([\s\S]*?)\n\);/)?.[1] ?? '';
|
||||
for (const m of table.matchAll(/^\s+([a-z_]+)\s+[A-Z]/gm)) cols.add(m[1]);
|
||||
for (const m of text.matchAll(/ADD COLUMN IF NOT EXISTS ([a-z_]+)/g)) cols.add(m[1]);
|
||||
}
|
||||
const rows = INGEST.match(/rows\.push\(\{([\s\S]*?)\n\s*\}\);/)?.[1] ?? '';
|
||||
const written = [...rows.matchAll(/^\s+([a-z_]+):/gm)].map(m => m[1])
|
||||
.concat(Object.keys({ outcome: 1, outcome_raw: 1, harness_session_id: 1, from_version: 1 }))
|
||||
.concat(fields.map(([c]) => c))
|
||||
.concat(['security_url_domain', 'security_payload_hash', 'security_confidence', 'security_layer', 'security_verdict']);
|
||||
expect(written.length).toBeGreaterThan(25);
|
||||
expect(written.filter(c => !cols.has(c))).toEqual([]);
|
||||
});
|
||||
|
||||
test('009 gives the backend role what the pulse needs and nobody else anything', () => {
|
||||
expect(sql).toContain('GRANT SELECT ON telemetry_live_events TO service_role;');
|
||||
expect(sql).toContain('GRANT SELECT, INSERT, UPDATE ON community_pulse_cache TO service_role;');
|
||||
expect(sql).not.toMatch(/GRANT [^;]* TO [^;]*\b(anon|authenticated|PUBLIC)\b/i);
|
||||
});
|
||||
|
||||
test('007 drops the live-only user_id policies before the columns, and treats backup_version 0 as empty', () => {
|
||||
const s007 = fs.readFileSync(path.join(MIGRATIONS, '007_drop_unused_installation_columns.sql'), 'utf-8');
|
||||
const drops = ['auth_read_own', 'auth_update_own', 'auth_write_own'].map(p => s007.indexOf(`DROP POLICY IF EXISTS ${p} ON public.installations`));
|
||||
expect(drops.every(i => i > 0 && i < s007.indexOf('DROP COLUMN'))).toBe(true);
|
||||
expect(s007).toContain("CASE WHEN col = 'backup_version' THEN ' AND backup_version <> 0' ELSE '' END");
|
||||
});
|
||||
});
|
||||
|
||||
Reference in new issue
Block a user