From 683b16076a0a6c3524216a8f05c2bf71fab1eb88 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 29 Aug 2026 06:02:15 +0000 Subject: [PATCH] docs(TODOS): close the absorbed test-infra items, file the overhaul follow-ups MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closed with receipts: the periodic coverage contract (implemented as full weekly coverage + exclusions), the eval-harness observability P1 (verified already landed: heartbeat, incremental _partial persistence, live stderr + eval-watch), and the sidebar trio (already deleted — tombstones remain). Filed: matrix deletion after parity, the required-check maintainer decision, browse /tmp-namespace hardening, PTY boot-readiness waits, the single typed test registry, bun-native LPT swap, runBin/free-runner migrations, eval-list partial exclusion, phantom key cleanup, duration-weighted slicing. Co-Authored-By: Claude Fable 5 --- TODOS.md | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 94 insertions(+), 3 deletions(-) diff --git a/TODOS.md b/TODOS.md index e364578fd..66e602ddf 100644 --- a/TODOS.md +++ b/TODOS.md @@ -353,7 +353,16 @@ touchfiles and re-offer pending ones on the next interactive run. false) permanently misses the artifacts-rename migration unless they paste the manual command. **Effort:** M. **Priority:** P2. -### P2: periodic tier — three documented-red tests need structural repair +### P2: periodic tier — TWO documented-red tests need structural repair (was three) + +**2026-08-29 update (test-infra overhaul):** (1) the sidebar E2E trio is +ALREADY DELETED — no file in the tree POSTs to /sidebar-command or +/sidebar-chat; only tombstone tests remain (browse/test/sidebar-tabs.test.ts +asserts the endpoints STAY deleted), so part (1) closes as already-done. +(2) skill-e2e-ship-idempotency and (3) skill-e2e-brain-privacy-gate are now +EXCLUDED from the weekly lane with tracking +(test/helpers/periodic-exclude-data.ts) — removing their entries re-activates +them; the structural investigations below are the re-entry condition. **What:** (1) The sidebar E2E trio (navigate, url-accuracy, css-interaction) POSTs to /sidebar-command and /sidebar-chat — endpoints removed on every tree @@ -416,6 +425,64 @@ references — include it in this fix's coverage list. ## Test infrastructure +### 2026-08-29 test-infra overhaul — follow-ups (filed at implementation) + +The overhaul landed: green-means-green fixes (make-pdf gates in the required +lane, zero-test eval jobs killed, 4 orphaned paid files activated + orphan +tripwire, touchfiles self-registration + warn→fail), the serial +tree-mutating shard dissolved (main() guard + --out-dir all hosts), +duration-packed free shards, the sharded paid runner as the CI engine +(planner/slices/fail-closed report, parity phase), the weekly all-periodic +coverage contract + gate census, eval-budget timeout tiers, and the +coverage fill. Remaining, in rough priority order: + +- **P1 — Delete the legacy evals.yml matrix after parity.** The sliced lane + runs alongside the 18-row matrix (`needs: evals`, so provider concurrency + never doubles). After 1-2 PR cycles of parity (compare executed-test sets: + intersection strict + the 8 KNOWN_MATRIX_GAPS files as expected additions; + stochastic outcomes informational), delete the matrix as a PURE-DELETION + commit (one revert restores it), drop the `needs: evals` edge, rewrite + test/evals-workflow-matrix.test.ts into a runner-wiring pin, and retire + KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET wholesale. Effort S. +- **P1 — Maintainer decision: make `slices-report` a required check** once + post-migration flake data exists (the Codex outside-voice's "green means + green is not delivered while paid stays advisory" point — correct, and + deliberately a branch-protection decision, not repo YAML). Effort S. +- **P2 — browse daemon /tmp-namespace hardening.** Every file-path transport + to the daemon (eval , load-html --from-file, pdf output, upload, + cookie-import) assumes client and daemon share one /tmp view; a sandboxed + shell reusing an out-of-namespace daemon gets "File not found" on files it + just wrote (root-caused live, reproduced with unshare). Minimal fix: the + CLI reads a local `eval ` itself and sends the code as `js` ( + semantics-preserving; keep the daemon path for remote callers), plus a + namespace hint appended to read-commands.ts:313's error. Effort S. +- **P2 — PTY boot-readiness wait.** The PTY tests' Bun.sleep(8000) preludes + and invokeAndObserve's 6s boot_grace_ms are blind waits; a real readiness + waitFor needs empirical CLI 2.1.x ready-marker probing in a working + terminal environment (this sandbox's PTY probe wedged). Effort S, needs a + dev machine. +- **P2 — single typed test registry.** Paid globs, tiers, touchfiles keys, + and exclusions are still separate literal authorities synced by tripwires; + derive them from one registry and the drift class dies structurally + (outside-voice recommendation; the tripwires are the interim). Effort M. +- **P2 — swap the custom LPT packer for bun-native `--timings`/`--shard`** + at the next Bun unpin (native LPT scheduling ships ≥1.3.14; the packer is + deliberately small and swappable — see the successor note in + scripts/test-free-shards.ts). Effort S. +- **P3 — runBin migration remainder** (~31 of 36 local run() duplicates; + helper + first 3 migrated). Mechanical batches. Effort S. +- **P3 — migrate the free runner onto runShardChild** (the shared lifecycle + helper the paid runner now uses; designed for it). Effort S. +- **P3 — eval-list should exclude _partial runs** (pinned as current + behavior in test/eval-cli-family.test.ts with an improvement note). + Effort S. +- **P3 — codex-e2e-plan-format's testIfSelected names have no map keys** + (run-all only today) + 15 E2E / 2 judge PHANTOM touchfiles keys select + tests that exist nowhere — add keys or delete, one sweep. Effort S. +- **P3 — duration-weighted slice assignment** if parity data shows slice + walls diverging >1.5x (round-robin today; eval-store durations exist). + Effort S. + ### P2: /context-save worktree-identity hardening (the #2052 residual) **What:** Persist a stable worktree identity (path hash or worktree name) into @@ -460,7 +527,19 @@ Trigger condition documented in `lib/gbrain-sources.ts` at the drift log line. **Effort:** M (human ~1d, CC ~45min). **Depends on:** drift-log evidence from the wave's `ensureSourceRegistered` logging. -### P2: Periodic CI matrix covers 9 of ~66 e2e files — decide the coverage contract +### ✅ DONE (2026-08-29): Periodic CI coverage contract — implemented as option (a) + +**Resolved by the test-infra overhaul:** evals-periodic.yml re-platformed onto +scripts/test-paid-shards.ts — ALL periodic-tier files run weekly (EVALS_ALL, +planner manifest → 6 slices → fail-closed report) minus the reasoned +exclusions in test/helpers/periodic-exclude-data.ts (reason + tracking per +entry, policy-pinned). A weekly EVALS_ALL gate census rides the same cron. +The silent-rot class is dead: a test that runs nowhere is now either planned, +diff-skipped, excluded-with-reason, or a failed report. Original filing kept +below for the receipts. + +#### Original filing (closed) +Periodic CI matrix covers 9 of ~66 e2e files — decide the coverage contract **Priority:** P2 @@ -498,7 +577,19 @@ in `.github/workflows/evals*.yml`. Receipts from the autoplan incident: `~/.gstack/projects/garrytan-gstack/e2e-runs/2026-07-10-0154/` (0-turn "Unknown command" transcripts). -### Eval harness: live progress + incremental result persistence (kill the silent hour) +### ✅ DONE (verified 2026-08-29): Eval harness live progress + incremental persistence + +**Verified landed** (the v1.66-era harness work delivered all three asks): +(1) heartbeat — session-runner writes ~/.gstack-dev/e2e-live.json atomically +per tool call (+ progress.log + per-test ndjson); (2) incremental persistence +— EvalCollector writes _partial-e2e.json after every addTest, dual-signal +isPartialEval keeps partials out of baselines; (3) live signal — per-tool +stderr progress lines flush unbuffered, and scripts/eval-watch.ts dashboards +the heartbeat. The 2026-08 overhaul added per-shard full-stream spool logs +(path printed at START) on top. Original filing kept below for receipts. + +#### Original filing (closed) +Eval harness: live progress + incremental result persistence (kill the silent hour) **Priority:** P1