docs(TODOS): close the absorbed test-infra items, file the overhaul follow-ups

Closed with receipts: the periodic coverage contract (implemented as
full weekly coverage + exclusions), the eval-harness observability P1
(verified already landed: heartbeat, incremental _partial persistence,
live stderr + eval-watch), and the sidebar trio (already deleted —
tombstones remain). Filed: matrix deletion after parity, the
required-check maintainer decision, browse /tmp-namespace hardening,
PTY boot-readiness waits, the single typed test registry, bun-native
LPT swap, runBin/free-runner migrations, eval-list partial exclusion,
phantom key cleanup, duration-weighted slicing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry TanandClaude Fable 5 committed 2026-08-29 06:02:15 +00:00
1 parent b54d5eefb4
commit 683b16076a
1 file changed
+94 -3
+94 -3
View File
@@ -353,7 +353,16 @@ touchfiles and re-offer pending ones on the next interactive run.
false) permanently misses the artifacts-rename migration unless they paste the
manual command. **Effort:** M. **Priority:** P2.
### P2: periodic tier — three documented-red tests need structural repair
### P2: periodic tier — TWO documented-red tests need structural repair (was three)
**2026-08-29 update (test-infra overhaul):** (1) the sidebar E2E trio is
ALREADY DELETED — no file in the tree POSTs to /sidebar-command or
/sidebar-chat; only tombstone tests remain (browse/test/sidebar-tabs.test.ts
asserts the endpoints STAY deleted), so part (1) closes as already-done.
(2) skill-e2e-ship-idempotency and (3) skill-e2e-brain-privacy-gate are now
EXCLUDED from the weekly lane with tracking
(test/helpers/periodic-exclude-data.ts) — removing their entries re-activates
them; the structural investigations below are the re-entry condition.
**What:** (1) The sidebar E2E trio (navigate, url-accuracy, css-interaction)
POSTs to /sidebar-command and /sidebar-chat — endpoints removed on every tree
@@ -416,6 +425,64 @@ references — include it in this fix's coverage list.
## Test infrastructure
### 2026-08-29 test-infra overhaul — follow-ups (filed at implementation)
The overhaul landed: green-means-green fixes (make-pdf gates in the required
lane, zero-test eval jobs killed, 4 orphaned paid files activated + orphan
tripwire, touchfiles self-registration + warn→fail), the serial
tree-mutating shard dissolved (main() guard + --out-dir all hosts),
duration-packed free shards, the sharded paid runner as the CI engine
(planner/slices/fail-closed report, parity phase), the weekly all-periodic
coverage contract + gate census, eval-budget timeout tiers, and the
coverage fill. Remaining, in rough priority order:
- **P1 — Delete the legacy evals.yml matrix after parity.** The sliced lane
runs alongside the 18-row matrix (`needs: evals`, so provider concurrency
never doubles). After 1-2 PR cycles of parity (compare executed-test sets:
intersection strict + the 8 KNOWN_MATRIX_GAPS files as expected additions;
stochastic outcomes informational), delete the matrix as a PURE-DELETION
commit (one revert restores it), drop the `needs: evals` edge, rewrite
test/evals-workflow-matrix.test.ts into a runner-wiring pin, and retire
KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET wholesale. Effort S.
- **P1 — Maintainer decision: make `slices-report` a required check** once
post-migration flake data exists (the Codex outside-voice's "green means
green is not delivered while paid stays advisory" point — correct, and
deliberately a branch-protection decision, not repo YAML). Effort S.
- **P2 — browse daemon /tmp-namespace hardening.** Every file-path transport
to the daemon (eval <file>, load-html --from-file, pdf output, upload,
cookie-import) assumes client and daemon share one /tmp view; a sandboxed
shell reusing an out-of-namespace daemon gets "File not found" on files it
just wrote (root-caused live, reproduced with unshare). Minimal fix: the
CLI reads a local `eval <file>` itself and sends the code as `js` (
semantics-preserving; keep the daemon path for remote callers), plus a
namespace hint appended to read-commands.ts:313's error. Effort S.
- **P2 — PTY boot-readiness wait.** The PTY tests' Bun.sleep(8000) preludes
and invokeAndObserve's 6s boot_grace_ms are blind waits; a real readiness
waitFor needs empirical CLI 2.1.x ready-marker probing in a working
terminal environment (this sandbox's PTY probe wedged). Effort S, needs a
dev machine.
- **P2 — single typed test registry.** Paid globs, tiers, touchfiles keys,
and exclusions are still separate literal authorities synced by tripwires;
derive them from one registry and the drift class dies structurally
(outside-voice recommendation; the tripwires are the interim). Effort M.
- **P2 — swap the custom LPT packer for bun-native `--timings`/`--shard`**
at the next Bun unpin (native LPT scheduling ships ≥1.3.14; the packer is
deliberately small and swappable — see the successor note in
scripts/test-free-shards.ts). Effort S.
- **P3 — runBin migration remainder** (~31 of 36 local run() duplicates;
helper + first 3 migrated). Mechanical batches. Effort S.
- **P3 — migrate the free runner onto runShardChild** (the shared lifecycle
helper the paid runner now uses; designed for it). Effort S.
- **P3 — eval-list should exclude _partial runs** (pinned as current
behavior in test/eval-cli-family.test.ts with an improvement note).
Effort S.
- **P3 — codex-e2e-plan-format's testIfSelected names have no map keys**
(run-all only today) + 15 E2E / 2 judge PHANTOM touchfiles keys select
tests that exist nowhere — add keys or delete, one sweep. Effort S.
- **P3 — duration-weighted slice assignment** if parity data shows slice
walls diverging >1.5x (round-robin today; eval-store durations exist).
Effort S.
### P2: /context-save worktree-identity hardening (the #2052 residual)
**What:** Persist a stable worktree identity (path hash or worktree name) into
@@ -460,7 +527,19 @@ Trigger condition documented in `lib/gbrain-sources.ts` at the drift log line.
**Effort:** M (human ~1d, CC ~45min). **Depends on:** drift-log evidence from
the wave's `ensureSourceRegistered` logging.
### P2: Periodic CI matrix covers 9 of ~66 e2e files — decide the coverage contract
### ✅ DONE (2026-08-29): Periodic CI coverage contract — implemented as option (a)
**Resolved by the test-infra overhaul:** evals-periodic.yml re-platformed onto
scripts/test-paid-shards.ts — ALL periodic-tier files run weekly (EVALS_ALL,
planner manifest → 6 slices → fail-closed report) minus the reasoned
exclusions in test/helpers/periodic-exclude-data.ts (reason + tracking per
entry, policy-pinned). A weekly EVALS_ALL gate census rides the same cron.
The silent-rot class is dead: a test that runs nowhere is now either planned,
diff-skipped, excluded-with-reason, or a failed report. Original filing kept
below for the receipts.
#### Original filing (closed)
Periodic CI matrix covers 9 of ~66 e2e files — decide the coverage contract
**Priority:** P2
@@ -498,7 +577,19 @@ in `.github/workflows/evals*.yml`. Receipts from the autoplan incident:
`~/.gstack/projects/garrytan-gstack/e2e-runs/2026-07-10-0154/` (0-turn "Unknown command"
transcripts).
### Eval harness: live progress + incremental result persistence (kill the silent hour)
### ✅ DONE (verified 2026-08-29): Eval harness live progress + incremental persistence
**Verified landed** (the v1.66-era harness work delivered all three asks):
(1) heartbeat — session-runner writes ~/.gstack-dev/e2e-live.json atomically
per tool call (+ progress.log + per-test ndjson); (2) incremental persistence
— EvalCollector writes _partial-e2e.json after every addTest, dual-signal
isPartialEval keeps partials out of baselines; (3) live signal — per-tool
stderr progress lines flush unbuffered, and scripts/eval-watch.ts dashboards
the heartbeat. The 2026-08 overhaul added per-shard full-stream spool logs
(path printed at START) on top. Original filing kept below for receipts.
#### Original filing (closed)
Eval harness: live progress + incremental result persistence (kill the silent hour)
**Priority:** P1