Telemetry records why skill runs end, which host and model ran them, and how upgrades went (GST-3234) (#4542)

gstack's anonymous usage telemetry now records nine more fields: the
host and model family, how a skill was started, why it ended, whether
checks passed, how skill routing is set up, a random per-session id, and
for upgrades the version left and any error.

This tells us where skills stop working and which setups break upgrades,
which the current outcome-only data can't show.

Deploy in order after merge: apply migration 009, deploy the
telemetry-ingest function, then release the client; apply migration 007
after that.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Review-Finding: p1/specialist-batched#2 reader=specialist-batched
Review-Finding: p1/specialist-batched#3 reader=specialist-batched
Review-Finding: p1/adversarial-native#1 reader=adversarial-native
Review-Finding: p1/outside-voice#2 reader=outside-voice
Review-Finding: v1/outside-voice#1 reader=outside-voice
Review-Finding: v1/outside-voice#2 reader=outside-voice
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-19 23:10:19 +00:00
1 parent db2a56e679
commit 9ea26ecc42
74 files changed
+1180 -695

No files matched your search

+11 -11
View File
@@ -110,26 +110,26 @@ jq -nc --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg skill "SKILL_NAME" --arg
## Telemetry (run last)
After workflow completion, log telemetry with ONE command. OUTCOME is
success/error/abort/unknown; `SESSION_ID` and `TEL_START` are the values the
preamble's skill-start output echoed. It also drains the artifacts-sync queue
(the former skill-end sync step — do not run gstack-brain-sync separately).
When done, make ONE call — it also drains the artifacts-sync
queue (don't also run gstack-brain-sync).
**PLAN MODE EXCEPTION — ALWAYS RUN:** This writes telemetry to
`$GSTACK_STATE_ROOT/analytics/`, matching preamble analytics writes.
**PLAN MODE EXCEPTION — ALWAYS RUN:** writes telemetry to `$GSTACK_STATE_ROOT/analytics/`.
```bash
[ -d "${GSTACK_ROOT:-/-}/bin" ]&&[ -d "$GSTACK_ROOT/lib" ]||{ _r=$(git rev-parse --show-toplevel 2>/dev/null)/.agents/skills/gstack;[ -d "$_r/bin" ]||_r=~/.gemini/antigravity-cli/skills/gstack;[ -d "$_r/bin" ]||{ echo "gstack: no install found (tried $_r). Fix: ./setup --host agy from your gstack checkout; ./setup --status shows it.">&2;exit 1;};GSTACK_ROOT=$_r;}
GSTACK_BIN=$GSTACK_ROOT/bin
$GSTACK_BIN/gstack-skill-end --skill "ship" --outcome OUTCOME \
--session-id "SESSION_ID" --tel-start "TEL_START" --used-browse USED_BROWSE \
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" 2>/dev/null || true
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" \
--exit-reason EXIT_REASON --invoked INVOKED --model-family MODEL_FAMILY --checks CHECKS 2>/dev/null || true
```
Replace `OUTCOME` and `USED_BROWSE` (yes/no) before running; substitute
`SESSION_ID`/`TEL_START` from the skill-start echoes. `ERROR_MESSAGE`/`FAILED_STEP`
are "" unless outcome is error. If the command is missing (stale install), skip
telemetry — it never blocks the workflow.
OUTCOME: success/error/abort/unknown. USED_BROWSE: yes/no. SESSION_ID, TEL_START:
skill-start echoes. ERROR_MESSAGE, FAILED_STEP: "" unless error. EXIT_REASON:
completed/user_stopped/blocked/tool_error/test_failed/context_limit/gave_up.
INVOKED: user/model/chained. MODEL_FAMILY (yours): claude_opus/claude_sonnet/
claude_haiku/claude_other/gpt/gemini/other. CHECKS: passed/failed/not_run. No command
(stale install) — skip.
## Third-Party Web Actions
+11 -11
View File
@@ -117,24 +117,24 @@ jq -nc --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg skill "SKILL_NAME" --arg
## Telemetry (run last)
After workflow completion, log telemetry with ONE command. OUTCOME is
success/error/abort/unknown; `SESSION_ID` and `TEL_START` are the values the
preamble's skill-start output echoed. It also drains the artifacts-sync queue
(the former skill-end sync step — do not run gstack-brain-sync separately).
When done, make ONE call — it also drains the artifacts-sync
queue (don't also run gstack-brain-sync).
**PLAN MODE EXCEPTION — ALWAYS RUN:** This writes telemetry to
`$GSTACK_STATE_ROOT/analytics/`, matching preamble analytics writes.
**PLAN MODE EXCEPTION — ALWAYS RUN:** writes telemetry to `$GSTACK_STATE_ROOT/analytics/`.
```bash
~/.claude/skills/gstack/bin/gstack-skill-end --skill "ship" --outcome OUTCOME \
--session-id "SESSION_ID" --tel-start "TEL_START" --used-browse USED_BROWSE \
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" 2>/dev/null || true
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" \
--exit-reason EXIT_REASON --invoked INVOKED --model-family MODEL_FAMILY --checks CHECKS 2>/dev/null || true
```
Replace `OUTCOME` and `USED_BROWSE` (yes/no) before running; substitute
`SESSION_ID`/`TEL_START` from the skill-start echoes. `ERROR_MESSAGE`/`FAILED_STEP`
are "" unless outcome is error. If the command is missing (stale install), skip
telemetry — it never blocks the workflow.
OUTCOME: success/error/abort/unknown. USED_BROWSE: yes/no. SESSION_ID, TEL_START:
skill-start echoes. ERROR_MESSAGE, FAILED_STEP: "" unless error. EXIT_REASON:
completed/user_stopped/blocked/tool_error/test_failed/context_limit/gave_up.
INVOKED: user/model/chained. MODEL_FAMILY (yours): claude_opus/claude_sonnet/
claude_haiku/claude_other/gpt/gemini/other. CHECKS: passed/failed/not_run. No command
(stale install) — skip.
## Third-Party Web Actions
+11 -11
View File
@@ -108,26 +108,26 @@ jq -nc --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg skill "SKILL_NAME" --arg
## Telemetry (run last)
After workflow completion, log telemetry with ONE command. OUTCOME is
success/error/abort/unknown; `SESSION_ID` and `TEL_START` are the values the
preamble's skill-start output echoed. It also drains the artifacts-sync queue
(the former skill-end sync step — do not run gstack-brain-sync separately).
When done, make ONE call — it also drains the artifacts-sync
queue (don't also run gstack-brain-sync).
**PLAN MODE EXCEPTION — ALWAYS RUN:** This writes telemetry to
`$GSTACK_STATE_ROOT/analytics/`, matching preamble analytics writes.
**PLAN MODE EXCEPTION — ALWAYS RUN:** writes telemetry to `$GSTACK_STATE_ROOT/analytics/`.
```bash
[ -d "${GSTACK_ROOT:-/-}/bin" ]&&[ -d "$GSTACK_ROOT/lib" ]||{ _r=$(git rev-parse --show-toplevel 2>/dev/null)/.agents/skills/gstack;[ -d "$_r/bin" ]||_r=${CODEX_HOME:-~/.codex}/skills/gstack;[ -d "$_r/bin" ]||{ echo "gstack: no install found (tried $_r). Fix: ./setup --host codex from your gstack checkout; ./setup --status shows it.">&2;exit 1;};GSTACK_ROOT=$_r;}
GSTACK_BIN=$GSTACK_ROOT/bin
$GSTACK_BIN/gstack-skill-end --skill "ship" --outcome OUTCOME \
--session-id "SESSION_ID" --tel-start "TEL_START" --used-browse USED_BROWSE \
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" 2>/dev/null || true
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" \
--exit-reason EXIT_REASON --invoked INVOKED --model-family MODEL_FAMILY --checks CHECKS 2>/dev/null || true
```
Replace `OUTCOME` and `USED_BROWSE` (yes/no) before running; substitute
`SESSION_ID`/`TEL_START` from the skill-start echoes. `ERROR_MESSAGE`/`FAILED_STEP`
are "" unless outcome is error. If the command is missing (stale install), skip
telemetry — it never blocks the workflow.
OUTCOME: success/error/abort/unknown. USED_BROWSE: yes/no. SESSION_ID, TEL_START:
skill-start echoes. ERROR_MESSAGE, FAILED_STEP: "" unless error. EXIT_REASON:
completed/user_stopped/blocked/tool_error/test_failed/context_limit/gave_up.
INVOKED: user/model/chained. MODEL_FAMILY (yours): claude_opus/claude_sonnet/
claude_haiku/claude_other/gpt/gemini/other. CHECKS: passed/failed/not_run. No command
(stale install) — skip.
## Third-Party Web Actions
+11 -11
View File
@@ -110,26 +110,26 @@ jq -nc --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg skill "SKILL_NAME" --arg
## Telemetry (run last)
After workflow completion, log telemetry with ONE command. OUTCOME is
success/error/abort/unknown; `SESSION_ID` and `TEL_START` are the values the
preamble's skill-start output echoed. It also drains the artifacts-sync queue
(the former skill-end sync step — do not run gstack-brain-sync separately).
When done, make ONE call — it also drains the artifacts-sync
queue (don't also run gstack-brain-sync).
**PLAN MODE EXCEPTION — ALWAYS RUN:** This writes telemetry to
`$GSTACK_STATE_ROOT/analytics/`, matching preamble analytics writes.
**PLAN MODE EXCEPTION — ALWAYS RUN:** writes telemetry to `$GSTACK_STATE_ROOT/analytics/`.
```bash
[ -d "${GSTACK_ROOT:-/-}/bin" ]&&[ -d "$GSTACK_ROOT/lib" ]||{ _r=$(git rev-parse --show-toplevel 2>/dev/null)/.factory/skills/gstack;[ -d "$_r/bin" ]||_r=~/.factory/skills/gstack;[ -d "$_r/bin" ]||{ echo "gstack: no install found (tried $_r). Fix: ./setup --host factory from your gstack checkout; ./setup --status shows it.">&2;exit 1;};GSTACK_ROOT=$_r;}
GSTACK_BIN=$GSTACK_ROOT/bin
$GSTACK_BIN/gstack-skill-end --skill "ship" --outcome OUTCOME \
--session-id "SESSION_ID" --tel-start "TEL_START" --used-browse USED_BROWSE \
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" 2>/dev/null || true
--error-message "ERROR_MESSAGE" --failed-step "FAILED_STEP" \
--exit-reason EXIT_REASON --invoked INVOKED --model-family MODEL_FAMILY --checks CHECKS 2>/dev/null || true
```
Replace `OUTCOME` and `USED_BROWSE` (yes/no) before running; substitute
`SESSION_ID`/`TEL_START` from the skill-start echoes. `ERROR_MESSAGE`/`FAILED_STEP`
are "" unless outcome is error. If the command is missing (stale install), skip
telemetry — it never blocks the workflow.
OUTCOME: success/error/abort/unknown. USED_BROWSE: yes/no. SESSION_ID, TEL_START:
skill-start echoes. ERROR_MESSAGE, FAILED_STEP: "" unless error. EXIT_REASON:
completed/user_stopped/blocked/tool_error/test_failed/context_limit/gave_up.
INVOKED: user/model/chained. MODEL_FAMILY (yours): claude_opus/claude_sonnet/
claude_haiku/claude_other/gpt/gemini/other. CHECKS: passed/failed/not_run. No command
(stale install) — skip.
## Third-Party Web Actions
+21
View File
@@ -130,6 +130,27 @@ describe('gstack-upgrade', () => {
expect(f.run().stdout).toContain('gstack-upgrade: current 1.0.1');
});
test('telemetry: one upgrade_completed per run with its upgrade_error and from-version; --check sends nothing', () => {
const f = fixture();
const telLog = path.join(f.host.base, 'calls.log.tel');
write(path.join(f.root, 'bin/gstack-telemetry-log'), '#!/usr/bin/env bash\necho "$*" >> "$UPGRADE_LOG.tel"\n', 0o755);
const tel = () => fs.existsSync(telLog) ? fs.readFileSync(telLog, 'utf8').trim().split('\n').filter(Boolean) : [];
expect(f.run(['--check']).status).toBe(1);
expect(tel()).toEqual([]);
fs.writeFileSync(path.join(f.root, 'README.md'), 'local edit\n');
expect(f.run().status).toBe(3);
f.git(['checkout', '--', 'README.md']);
expect(f.run([], { STUB_SETUP_EXIT: '1' }).status).toBe(1);
const retry = f.run();
expect(retry.status, retry.stdout + retry.stderr).toBe(0);
const ev = '--no-sweep --event-type upgrade_completed';
expect(tel()).toEqual([
`${ev} --outcome error --from-version 1.0.0 --upgrade-error refused_dirty`,
`${ev} --outcome error --from-version 1.0.0 --upgrade-error setup_failed`,
`${ev} --outcome success --from-version 1.0.0 --upgrade-error none`,
]);
});
test('a Capy install (the installer marker) runs bin/gstack-capy-install instead of setup', () => {
const f = fixture();
write(path.join(f.root, 'bin/gstack-capy-install'), '#!/usr/bin/env bash\necho "capy-install $*" >> "$UPGRADE_LOG"\necho "gstack: ok $(cat VERSION) for=autoplan"\n', 0o755);
+165
View File
@@ -0,0 +1,165 @@
/** The next telemetry fields (docs/privacy.md, migration 009) as the client
* writes them: fixed lists, a random per-agent-session id, and a source label
* an explicit setting always wins. Network is stubbed; nothing leaves the box. */
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
import { spawn, spawnSync } from 'child_process';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
const ROOT = path.resolve(import.meta.dir, '..');
const BIN = path.join(ROOT, 'bin');
let tmp: string;
function env(extra: Record<string, string> = {}): Record<string, string> {
return {
...process.env as Record<string, string>,
HOME: path.join(tmp, 'home'), GSTACK_HOME: tmp, GSTACK_STATE_ROOT: tmp, GSTACK_STATE_DIR: tmp,
GSTACK_DIR: ROOT, GSTACK_SUPABASE_URL: 'https://telemetry.fixture.invalid', GSTACK_SUPABASE_ANON_KEY: 'fixture',
GSTACK_TEST_BLOCK_LIVE_TELEMETRY: '1', GSTACK_SESSION_KIND: 'interactive',
CI: '', GSTACK_TELEMETRY_SOURCE: '', EVALS_RUN_ID: '', GSTACK_EVAL_DIR: '',
GSTACK_ACTIVE_HOST: '', CLAUDECODE: '', CODEX_THREAD_ID: '',
PATH: `${path.join(tmp, 'bin')}:${process.env.PATH}`,
...extra,
};
}
function run(bin: string, args: string[], extra: Record<string, string> = {}) {
const r = spawnSync(path.join(BIN, bin), args, { cwd: tmp, env: env(extra), encoding: 'utf-8', timeout: 20_000 });
expect(r.status).toBe(0);
return r.stdout;
}
function events(): any[] {
const f = path.join(tmp, 'analytics', 'skill-usage.jsonl');
const lines = fs.existsSync(f) ? fs.readFileSync(f, 'utf-8').trim().split('\n').filter(Boolean).map(l => JSON.parse(l)) : [];
return lines.filter(e => e.v === 1);
}
async function eventually(n: number): Promise<any[]> {
for (let i = 0; i < 50 && events().length < n; i++) await Bun.sleep(100);
return events();
}
beforeEach(() => {
tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-nexttel-'));
fs.mkdirSync(path.join(tmp, 'home'));
fs.mkdirSync(path.join(tmp, 'bin'));
fs.writeFileSync(path.join(tmp, 'bin', 'curl'), '#!/bin/sh\nprintf 200\n', { mode: 0o755 });
run('gstack-config', ['set', 'telemetry', 'anonymous']);
});
afterEach(() => { fs.rmSync(tmp, { recursive: true, force: true }); });
describe.skipIf(process.platform === 'win32')('gstack-telemetry-log next fields', () => {
test('fixed-list fields pass through, off-list values become unknown, absent ones null', () => {
run('gstack-telemetry-log', ['--skill', 'qa', '--outcome', 'success', '--host', 'codex', '--model-family', 'gpt',
'--invoked', 'model', '--exit-reason', 'rm -rf /', '--checks', 'passed', '--routing', 'block']);
const [e] = events();
expect([e.host, e.model_family, e.invoked, e.exit_reason, e.checks, e.routing]).toEqual(['codex', 'gpt', 'model', 'unknown', 'passed', 'block']);
expect([e.harness_session_id, e.from_version, e.upgrade_error]).toEqual([null, null, null]);
});
test('the session id must be 16 hex and from_version a version, or they are dropped', () => {
run('gstack-telemetry-log', ['--event-type', 'upgrade_completed', '--outcome', 'error', '--harness-session', '0123456789abcdef',
'--from-version', '1.2.3.4', '--upgrade-error', 'busy']);
run('gstack-telemetry-log', ['--outcome', 'success', '--harness-session', 'not-a-session-id', '--from-version', '1.2; DROP']);
const [a, b] = events();
expect([a.harness_session_id, a.from_version, a.upgrade_error]).toEqual(['0123456789abcdef', '1.2.3.4', 'busy']);
expect([b.harness_session_id, b.from_version]).toEqual([null, null]);
});
test('source: flag > GSTACK_TELEMETRY_SOURCE > eval run > CI > live, so CI never relabels a test row', () => {
run('gstack-telemetry-log', ['--outcome', 'success', '--source', 'test'], { CI: 'true', EVALS_RUN_ID: 'r1' });
run('gstack-telemetry-log', ['--outcome', 'success'], { CI: 'true', GSTACK_TELEMETRY_SOURCE: 'test' });
run('gstack-telemetry-log', ['--outcome', 'success'], { CI: 'true', EVALS_RUN_ID: 'r1' });
run('gstack-telemetry-log', ['--outcome', 'success'], { CI: 'true' });
run('gstack-telemetry-log', ['--outcome', 'success']);
expect(events().map(e => e.source)).toEqual(['test', 'test', 'eval', 'ci', 'live']);
});
});
describe.skipIf(process.platform === 'win32')('gstack-skill-end next fields', () => {
test('one random id per harness process, kept in its sessions marker without adding a session', async () => {
fs.mkdirSync(path.join(tmp, 'sessions'));
const marker = path.join(tmp, 'sessions', '4242');
fs.writeFileSync(marker, '');
const args = ['--skill', 'qa', '--outcome', 'success', '--session-id', '4242-1700000000-1', '--tel-start', '1700000000',
'--exit-reason', 'completed', '--invoked', 'user', '--model-family', 'claude_opus', '--checks', 'not_run'];
run('gstack-skill-end', args, { CLAUDECODE: '1' });
run('gstack-skill-end', args, { CLAUDECODE: '1' });
const [a, b] = await eventually(2);
expect(a.harness_session_id).toMatch(/^[0-9a-f]{16}$/);
expect(b.harness_session_id).toBe(a.harness_session_id);
expect(fs.readFileSync(marker, 'utf-8')).toBe(a.harness_session_id);
expect(fs.readdirSync(path.join(tmp, 'sessions'))).toEqual(['4242']);
expect([a.host, a.exit_reason, a.invoked, a.model_family, a.checks, a.routing]).toEqual(['claude_code', 'completed', 'user', 'claude_opus', 'not_run', 'none']);
});
test('under a shared harness (capy-server) every session has one pid, so no session id is sent', async () => {
fs.copyFileSync('/bin/sleep', path.join(tmp, 'capy-server'));
fs.chmodSync(path.join(tmp, 'capy-server'), 0o755);
const harness = spawn(path.join(tmp, 'capy-server'), ['30'], { stdio: 'ignore' });
try {
fs.mkdirSync(path.join(tmp, 'sessions'));
fs.writeFileSync(path.join(tmp, 'sessions', String(harness.pid)), '');
run('gstack-skill-end', ['--skill', 'qa', '--outcome', 'success', '--session-id', `${harness.pid}-1700000000-1`, '--tel-start', '1700000000']);
const [e] = await eventually(1);
expect(e.harness_session_id).toBeNull();
expect(fs.readFileSync(path.join(tmp, 'sessions', String(harness.pid)), 'utf-8')).toBe('');
} finally { harness.kill('SIGKILL'); }
});
test('a recycled pid with a stale marker gets a new session id, never the old one', async () => {
fs.appendFileSync(path.join(tmp, 'config.yaml'), 'update_check: false\n');
fs.mkdirSync(path.join(tmp, 'sessions'));
const marker = path.join(tmp, 'sessions', '4343');
fs.writeFileSync(marker, 'aaaaaaaaaaaaaaaa');
const threeHoursAgo = new Date(Date.now() - 3 * 3600_000);
fs.utimesSync(marker, threeHoursAgo, threeHoursAgo);
run('gstack-skill-start', ['--skill', 'qa', '--parent-pid', '4343']);
expect(fs.readFileSync(marker, 'utf-8')).toBe('');
run('gstack-skill-end', ['--skill', 'qa', '--outcome', 'success', '--session-id', '4343-1700000000-1', '--tel-start', '1700000000'], { CLAUDECODE: '1' });
const [e] = await eventually(1);
expect(e.harness_session_id).toMatch(/^[0-9a-f]{16}$/);
expect(e.harness_session_id).not.toBe('aaaaaaaaaaaaaaaa');
}, 30_000);
test('an unknown start time sends a null duration, not 0', async () => {
const out = run('gstack-skill-end', ['--skill', 'qa', '--outcome', 'success', '--session-id', '99-1']);
expect(out).toContain('duration_s=unknown');
const [e] = await eventually(1);
expect(e.duration_s).toBeNull();
});
});
describe.skipIf(process.platform === 'win32')('gstack-telemetry-sync batch size', () => {
test('a backlog of full-size events is sent in bodies under ingest\'s 50,000-byte limit', () => {
fs.writeFileSync(path.join(tmp, 'bin', 'curl'),
'#!/bin/sh\nfor a in "$@"; do case "$a" in @*) wc -c < "${a#@}" >> "$SIZES" ;; esac; done\nprintf 500\n', { mode: 0o755 });
const event = JSON.stringify({ v: 1, ts: '2026-11-19T21:00:00Z', event_type: 'skill_run', skill: 'plan-design-review', session_id: '12345-1763600000-abc',
gstack_version: '1.102.0.0', os: 'darwin', arch: 'arm64', duration_s: 1234, outcome: 'success', error_class: null, error_message: null, failed_step: null,
used_browse: false, sessions: 1, installation_id: null, source: 'live', security_url_domain: null, security_payload_hash: null, security_confidence: null,
security_layer: null, security_verdict: null, host: 'claude_code', model_family: 'claude_opus', invoked: 'user', exit_reason: 'completed', checks: 'passed',
routing: 'both', harness_session_id: '0123456789abcdef', from_version: null, upgrade_error: null, _repo_slug: 'owner/repo', _branch: 'main' });
fs.mkdirSync(path.join(tmp, 'analytics'), { recursive: true });
fs.writeFileSync(path.join(tmp, 'analytics', 'skill-usage.jsonl'), `${event}\n`.repeat(120));
const sizes = path.join(tmp, 'sizes');
run('gstack-telemetry-sync', [], { SIZES: sizes });
const sent = fs.readFileSync(sizes, 'utf-8').trim().split('\n').map(Number);
expect(sent).toHaveLength(1);
expect(sent[0]).toBeGreaterThan(40_000);
expect(sent[0]).toBeLessThanOrEqual(45_000);
});
test('multi-byte events are measured in bytes, so the body stays under the limit', () => {
fs.writeFileSync(path.join(tmp, 'bin', 'curl'),
'#!/bin/sh\nfor a in "$@"; do case "$a" in @*) wc -c < "${a#@}" >> "$SIZES" ;; esac; done\nprintf 500\n', { mode: 0o755 });
const event = JSON.stringify({ v: 1, ts: '2026-11-19T21:00:00Z', event_type: 'skill_run', skill: 'qa', session_id: '12345-1763600000-abc',
outcome: 'error', error_message: '\u20ac'.repeat(300), source: 'live', host: 'claude_code' });
fs.mkdirSync(path.join(tmp, 'analytics'), { recursive: true });
fs.writeFileSync(path.join(tmp, 'analytics', 'skill-usage.jsonl'), `${event}\n`.repeat(100));
const sizes = path.join(tmp, 'sizes');
run('gstack-telemetry-sync', [], { SIZES: sizes, LC_ALL: process.platform === 'darwin' ? 'en_US.UTF-8' : 'C.UTF-8' });
const sent = fs.readFileSync(sizes, 'utf-8').trim().split('\n').map(Number);
expect(sent).toHaveLength(1);
expect(sent[0]).toBeGreaterThan(40_000);
expect(sent[0]).toBeLessThanOrEqual(45_000);
});
});
+52
View File
@@ -94,3 +94,55 @@ describe('migrations keep the public key and signed-in users out', () => {
expect([...created].filter(p => /^anon_(insert|update)/.test(p))).toEqual([]);
});
});
describe('the next telemetry fields stay in step across client, ingest and schema', () => {
const LOG = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-telemetry-log'), 'utf-8');
const sql = fs.readFileSync(path.join(MIGRATIONS, '009_next_telemetry_fields.sql'), 'utf-8');
const ingestEnums = (field: string) => {
const m = INGEST.match(new RegExp(`${field}: \\[([^\\]]*)\\]`));
return [...(m?.[1] ?? '').matchAll(/"([a-z_]+)"/g)].map(x => x[1]).sort();
};
const fields: Array<[string, string]> = [['host', 'HOST_FIELD'], ['model_family', 'MODEL_FIELD'], ['invoked', 'INVOKED_FIELD'],
['exit_reason', 'EXIT_FIELD'], ['checks', 'CHECKS_FIELD'], ['routing', 'ROUTING_FIELD'], ['upgrade_error', 'UPGRADE_ERR_FIELD']];
test('each fixed list is the same in the client, the ingest and the CHECK (plus unknown)', () => {
for (const [col, v] of fields) {
const client = (LOG.match(new RegExp(`${v}="\\$\\(enum_field "\\$[A-Z_]+" ([^)]*)\\)"`))?.[1] ?? '').split(' ').filter(Boolean);
const check = [...(sql.match(new RegExp(`ADD COLUMN IF NOT EXISTS ${col} TEXT CHECK \\(${col} IN \\(([^)]*)\\)`))?.[1] ?? '').matchAll(/'([a-z_]+)'/g)].map(x => x[1]);
expect(client.length, col).toBeGreaterThan(2);
expect(ingestEnums(col), col).toEqual([...client, 'unknown'].sort());
expect(check.sort(), col).toEqual([...client, 'unknown'].sort());
}
});
test('every column the ingest writes is added by a migration', () => {
const cols = new Set<string>();
for (const f of fs.readdirSync(MIGRATIONS).filter(x => x.endsWith('.sql'))) {
const text = fs.readFileSync(path.join(MIGRATIONS, f), 'utf-8');
const table = text.match(/CREATE TABLE telemetry_events \(([\s\S]*?)\n\);/)?.[1] ?? '';
for (const m of table.matchAll(/^\s+([a-z_]+)\s+[A-Z]/gm)) cols.add(m[1]);
for (const m of text.matchAll(/ADD COLUMN IF NOT EXISTS ([a-z_]+)/g)) cols.add(m[1]);
}
const rows = INGEST.match(/rows\.push\(\{([\s\S]*?)\n\s*\}\);/)?.[1] ?? '';
const written = [...rows.matchAll(/^\s+([a-z_]+):/gm)].map(m => m[1])
.concat(Object.keys({ outcome: 1, outcome_raw: 1, harness_session_id: 1, from_version: 1 }))
.concat(fields.map(([c]) => c))
.concat(['security_url_domain', 'security_payload_hash', 'security_confidence', 'security_layer', 'security_verdict']);
expect(written.length).toBeGreaterThan(25);
expect(written.filter(c => !cols.has(c))).toEqual([]);
});
test('009 gives the backend role what the pulse needs and nobody else anything', () => {
expect(sql).toContain('GRANT SELECT ON telemetry_live_events TO service_role;');
expect(sql).toContain('GRANT SELECT, INSERT, UPDATE ON community_pulse_cache TO service_role;');
expect(sql).not.toMatch(/GRANT [^;]* TO [^;]*\b(anon|authenticated|PUBLIC)\b/i);
});
test('007 drops the live-only user_id policies before the columns, and treats backup_version 0 as empty', () => {
const s007 = fs.readFileSync(path.join(MIGRATIONS, '007_drop_unused_installation_columns.sql'), 'utf-8');
const drops = ['auth_read_own', 'auth_update_own', 'auth_write_own'].map(p => s007.indexOf(`DROP POLICY IF EXISTS ${p} ON public.installations`));
expect(drops.every(i => i > 0 && i < s007.indexOf('DROP COLUMN'))).toBe(true);
expect(s007).toContain("CASE WHEN col = 'backup_version' THEN ' AND backup_version <> 0' ELSE '' END");
});
});