diff --git a/TODOS.md b/TODOS.md index f7a7dae2..9c569c0d 100644 --- a/TODOS.md +++ b/TODOS.md @@ -303,17 +303,103 @@ **Priority:** P2 **Depends on:** Transcript sync (Phase 3) -### Weekly AI usage digest +### ~~Weekly AI usage digest~~ ✓ Shipped in Phase 4 -**What:** Supabase edge function that runs weekly, aggregates session_transcripts + eval_runs, sends team summary to Slack/email. +Implemented as `supabase/functions/weekly-digest/index.ts`. pg_cron Monday 9am UTC, aggregates 7-day team data, sends Slack summary. -**Why:** Passive team visibility without running commands. "Your team ran 47 sessions this week. Top tools: Edit(156), Bash(89). Sarah shipped 3 PRs via /ship." +## Team Dashboard -**Context:** Design doc Phase 4 item. Requires Supabase edge functions + Slack/email integration. Transcript data from Phase 3 is the primary input alongside eval_runs. +### Regression alert: include failing test names + dashboard link + +**What:** Slack alert message should list the specific tests that regressed and include a direct URL to the dashboard Evals tab. + +**Why:** Current alert says "pass rate dropped 89% → 82%" but doesn't say which tests. The person paged has to open the dashboard and hunt. Including test names and a direct link saves 2 minutes of triage. + +**Context:** `all_results` array in eval_runs has per-test data. `formatSlackMessage()` in regression-alert/index.ts is the change point. Dashboard URL can be derived from SUPABASE_URL. + +**Effort:** S +**Priority:** P2 +**Depends on:** Phase 4 (shipped) + +### Projected monthly cost annotation on dashboard + +**What:** Add "Projected monthly: ~$X" annotation to the cost chart on the dashboard. + +**Why:** Everyone wants the monthly number for budgeting. One line of math (last 4 weeks average × 4.33), huge value for finance conversations. + +**Context:** `renderVBarChart` or `renderCosts` in dashboard/ui.ts. Data is already fetched. + +**Effort:** XS +**Priority:** P3 + +### Ship notification to Slack + +**What:** Post a Slack message when someone ships: "alice shipped v0.4.2 → repo-slug (PR #45)". Reuses existing Slack webhook from team_settings. + +**Why:** Real-time team shipping awareness. Currently only regression alerts go to Slack — positive events (ships) should too. + +**Context:** Either add to the sync push path in ship/SKILL.md.tmpl or create a new edge function triggered on ship_logs INSERT (same pattern as regression-alert). + +**Effort:** S +**Priority:** P2 +**Depends on:** Phase 4 (shipped) + +### Dynamic favicon based on team pass rate + +**What:** Dashboard favicon changes color (green/yellow/red dot) based on current overall eval pass rate. Visible from the browser tab bar without switching to the dashboard tab. + +**Why:** Zero-click observability. At a glance from your tab bar, you know if the team is healthy. + +**Context:** Canvas → data URL favicon, update on each fetchAll() refresh in dashboard/ui.ts. Green >80%, yellow 50-80%, red <50%. + +**Effort:** XS +**Priority:** P3 + +### Server-side aggregation / materialized views + +**What:** Replace client-side data fetching (6 parallel REST calls per refresh) with server-side pre-aggregated views or Supabase materialized views. + +**Why:** Current approach pulls up to 100 rows per table per refresh. With 5+ users and 60s refresh, this puts pressure on Supabase request limits. Materialized views would return pre-computed summaries in a single call. + +**Context:** Could use Supabase pg_cron to refresh materialized views every 5 minutes. Dashboard would fetch one view instead of 6 tables. **Effort:** L -**Priority:** P2 -**Depends on:** Transcript sync (Phase 3), Supabase edge functions +**Priority:** P3 +**Depends on:** Phase 4 (shipped) + +### Real-time SSE streaming on dashboard + +**What:** Server-Sent Events stream from a Supabase edge function that pushes updates when new data arrives (eval_runs INSERT, ship_logs INSERT, heartbeats). + +**Why:** Dashboard currently polls every 60s. SSE would make it truly real-time — see an eval complete the moment it finishes. + +**Context:** Supabase Realtime can be used client-side, or a custom SSE edge function can listen to Postgres NOTIFY. Year 2 roadmap item. + +**Effort:** L +**Priority:** P3 + +### GitHub Check Run integration + +**What:** When an eval run is pushed, create a GitHub Check Run on the corresponding commit/PR showing pass rate, regressions, and cost. + +**Why:** Eval results become visible directly in the PR review workflow. Regressions can block merge. + +**Context:** Requires GitHub App installation or personal access token. Uses GitHub REST API `POST /repos/{owner}/{repo}/check-runs`. Year 2 roadmap item. + +**Effort:** L +**Priority:** P3 +**Depends on:** Phase 4 (shipped) + +### ship_logs index on (team_id, created_at) + +**What:** Add composite index `idx_ship_logs_team_date ON ship_logs(team_id, created_at DESC)`. + +**Why:** Weekly digest queries `ship_logs WHERE team_id = ? AND created_at >= ?`. Without this index, it table-scans. Low priority because ship_logs volume is small in Year 1, but needed before scale. + +**Context:** Add to a new migration 008 or append to 007. + +**Effort:** XS +**Priority:** P3 ## Infrastructure